Model optimization method and related device

Abstract

The embodiment of the present invention provides a model optimization method and related devices, which belong to the field of artificial intelligence technology. The method includes: setting a hierarchical cache area, the hierarchical cache area is used to cache data during the operation of a large model; performing task decomposition on the large model to obtain multiple subtasks corresponding to the large model; setting a data pre-fetching mechanism, storing the initial operation data required during the operation of the subtask in the hierarchical cache area according to the data pre-fetching mechanism; obtaining the initial operation data from the hierarchical cache area to perform operation calculations on the subtasks, and obtaining the intermediate operation data corresponding to the subtasks; merging all the intermediate operation data to obtain the target operation data corresponding to the large model. This method reduces the storage and transportation of duplicate data and reduces the transmission time by loading the required data in advance, which can effectively solve the shortcomings of large models in terms of video memory occupancy, computing delays and resource utilization.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model optimization method and related devices. Background Art

[0002] The Transformer model has made significant breakthroughs in the development of deep learning, especially in natural language processing tasks. This is mainly due to its unique self-attention mechanism, which can effectively capture the long-distance dependencies between words in a sentence. Therefore, Transformer has achieved remarkable results in fields such as machine translation, text generation, and question-answering systems.

[0003] However, as the model size and data volume continue to expand, large Transformer models face a series of challenges. First, high video memory usage and large computing latency have become major problems. The Transformer model consists of multiple layers of encoders and decoders, each of which contains a self-attention mechanism and a feedforward neural network. Since the calculation of each layer depends on the output of the previous layer, the calculation process of the entire model has obvious serial characteristics. Therefore, when processing long sequence data, the Transformer model needs to store and calculate a large number of weight matrices and intermediate results, which places high demands on video memory and computing resources. To solve these problems, the current common method is to increase computing units to speed up the training and inference process of the model. In addition, frequent data transmission and cache management also bring video memory pressure and data overhead, further exacerbating computing latency.

[0004] In addition, in terms of parallel computing, the parallel computing technology of multiple graphics processing units (GPUs) has been widely used. Through data parallelism and model parallelism, the training and reasoning of large models can be accelerated. However, due to the inter-layer dependencies in the calculation process of the Transformer large model, traditional parallel computing technology is difficult to give full play to its advantages.

[0005] Therefore, there is an urgent need for a computational strategy to optimize the Transformer large model to cope with inter-layer dependencies and improve the performance of the Transformer large model. Summary of the invention

[0006] The main purpose of the embodiments of the present invention is to provide a model optimization method and related devices, aiming to solve the problems of high video memory occupancy, large calculation delay, low resource utilization and unreasonable cache management of large models in the related technology.

[0007] In a first aspect, an embodiment of the present invention provides a model optimization method, comprising:

[0008] Setting a hierarchical cache area, wherein the hierarchical cache area is used to cache data during the operation of the large model;

[0009] Decomposing the large model into tasks to obtain a plurality of subtasks corresponding to the large model;

[0010] Setting a data pre-fetching mechanism, storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data pre-fetching mechanism;

[0011] Obtain the initial operation data from the hierarchical cache area to perform operation calculation on the subtask to obtain intermediate operation data corresponding to the subtask;

[0012] All the intermediate operation data are merged to obtain the target operation data corresponding to the large model.

[0013] In a second aspect, an embodiment of the present invention provides a model optimization device, comprising:

[0014] A cache partitioning module is used to set a hierarchical cache area, and the hierarchical cache area is used to cache data during the operation of the large model;

[0015] A task decomposition module, used to decompose the large model into multiple subtasks corresponding to the large model;

[0016] A data storage module, used for setting a data pre-fetching mechanism, storing the initial running data required in the running process of the subtask in the hierarchical cache area according to the data pre-fetching mechanism;

[0017] A task calculation module, used to obtain the initial operation data from the hierarchical cache area, perform operation calculation on the subtask, and obtain intermediate operation data corresponding to the subtask;

[0018] The data merging module is used to merge all the intermediate operation data to obtain the target operation data corresponding to the large model.

[0019] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the model optimization methods provided in the specification of the present invention are implemented.

[0020] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any model optimization method provided in the specification of the present invention.

[0021] The embodiment of the present invention provides a model optimization method and related devices, which include: setting a hierarchical cache area, which is used to cache data during the operation of a large model; decomposing the large model to obtain multiple subtasks corresponding to the large model; setting a data pre-fetching mechanism, storing the initial operation data required during the operation of the subtask in the hierarchical cache area according to the data pre-fetching mechanism; obtaining the initial operation data from the hierarchical cache area to perform operation calculations on the subtasks to obtain the intermediate operation data corresponding to the subtasks; merging all the intermediate operation data to obtain the target operation data corresponding to the large model. The method introduces a hierarchical cache area, reasonably manages and allocates cache resources, reduces the redundant storage of intermediate results and weight matrices, thereby reducing the video memory occupancy, and ensures that high-frequency access data is stored preferentially through hierarchical management, thereby improving the video memory utilization efficiency. The present application uses a data pre-fetching mechanism and parallel computing collaborative technology to load the required data in advance, reduce the waiting time of inter-layer data transmission, realize delay hiding, and perform multiple subtask calculations in parallel through multi-threading, thereby improving the response speed and real-time performance of the large model and maximizing the utilization efficiency of computing resources. The problem of high video memory occupancy, large calculation delay, low resource utilization and unreasonable cache management of large models in the related art is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of a flow chart of a model optimization method provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of the module structure of a model optimization device provided by an embodiment of the present invention;

[0025] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] The embodiment of the present invention provides a model optimization method and related devices. The model optimization method can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0030] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] The Transformer Large Model is a major breakthrough in the rapid development of deep learning, especially in natural language processing tasks. Its unique self-attention mechanism can effectively capture the long-distance dependencies between words in a sentence, and has achieved remarkable results in machine translation, text generation, question-answering systems and other fields.

[0032] However, as the model size and data volume grow, it faces problems such as high video memory usage and large computational delay. The large transformer model consists of multiple layers of encoders and decoders, each of which includes a self-attention mechanism and a feedforward neural network. Since the calculation of each layer depends on the output of the previous layer, the calculation process of the entire model shows obvious serial characteristics. In the process of processing long sequence data, the model needs to store and calculate a large number of weight matrices and intermediate results. Currently, it is mostly accelerated by adding computing units. In addition, frequent data transmission and cache management during calculations all present problems such as video memory pressure, data overhead and latency.

[0033] There are endless optimization studies on the above problems. In terms of parallel computing, multi-GPU parallel computing technology has been widely used to accelerate model training and reasoning processes through data parallelism and model parallelism. However, due to the inter-layer dependencies in the calculation of large transformer models, traditional parallel computing technology is difficult to fully exert its advantages. At the same time, in the research of cache optimization strategies and prefetching mechanisms, due to the complex computational dependencies of deep learning models, the optimization effects of some common existing technologies are limited. The current application of large transformer models has the following disadvantages:

[0034] High video memory usage: When processing long sequence data, the large transformer model needs to store a large number of intermediate results and weight matrices, which results in very high video memory usage. Even on a high-performance GPU, insufficient video memory may occur, limiting the application scope of the model.

[0035] Large computational delay: The self-attention mechanism and feedforward network calculations of traditional transformer large models are executed sequentially, and the calculation results of each layer depend on the output of the previous layer. This serial dependency leads to significant computational delays in the inference process, affecting the real-time performance and response speed of the model.

[0036] Low resource utilization: The waiting time during data transmission and computing is long, resulting in low computing resource utilization. Especially in a multi-GPU environment, the advantages of parallel computing cannot be fully utilized, resulting in a waste of computing resources and increased energy consumption.

[0037] Unreasonable cache management: Existing cache management strategies are mostly simple cache replacement algorithms that fail to optimize the model's computational characteristics, resulting in low cache hit rates. High-frequency data cannot be effectively utilized, increasing data transmission overhead and latency.

[0038] Based on the above analysis, there is an urgent need for a computing strategy to optimize the Transformer large model to solve the problems of high video memory occupancy, large computing latency, low resource utilization, and unreasonable cache management of large models.

[0039] Please refer to Figure 1 , Figure 1 A flow chart of a model optimization method provided in an embodiment of the present invention.

[0040] like Figure 1 As shown, the model optimization method includes steps S101 to S105.

[0041] Step S101: Set a hierarchical cache area, where the hierarchical cache area is used to cache data during the operation of the large model.

[0042] Exemplarily, a hierarchical cache is used to cache data during the operation of a large model. The hierarchical cache is usually composed of multiple levels, each with different speeds and capacities. For example: The first-level cache is usually the fastest and smallest cache, directly close to the computing core. The second-level cache is larger and slightly slower, and is usually a cache between the processor and the main memory. The third-level cache (L3 Cache) is larger and slower, and is usually a cache shared by multiple processing cores. Therefore, for the hierarchical cache of a large model, a similar hierarchical structure can be designed according to specific needs to ensure that different data is cached in different levels.

[0043] In some embodiments, the hierarchical cache area includes a global cache area, an intermediate cache area, and a local cache area. The setting of the hierarchical cache area includes: determining data cache requirements, the data cache requirements include high-frequency access cache requirements, intermediate data access cache requirements, and upcoming access cache requirements; determining the global cache area according to the high-frequency access cache requirements, the global cache area is used to store the weight matrix and high-frequency access data corresponding to the large model; determining the intermediate cache area according to the intermediate data access cache requirements, the intermediate cache area is used to store the intermediate results of the current calculation layer; determining the local cache area according to the upcoming access cache requirements, the local cache area is used to store data that is about to be accessed in the current step of the calculation process.

[0044] Exemplarily, the hierarchical cache area includes a global cache area, an intermediate cache area, and a local cache area. Data cache requirements are obtained by performing data analysis on a large model, including high-frequency access cache requirements, intermediate data access cache requirements, and upcoming access cache requirements.

[0045] For example, by analyzing the large model, the high-frequency access cache requirements are obtained. The storage data corresponding to the high-frequency access cache requirements is critical to the performance of the entire model and needs to be accessed efficiently and quickly. For example, based on the high-frequency access requirements, analyze and decide which weight matrices and data should be placed in the global cache. In addition, it is necessary to ensure that the capacity of the global cache is large enough to accommodate all high-frequency data and the speed is sufficient to meet the requirements of fast access.

[0046] Exemplarily, the global cache is used to store the weight matrix and other high-frequency access data of the large model. The role of the global cache is to reduce the overhead of repeatedly loading the weight matrix for each calculation, significantly reducing the pressure on the video memory. Assume that the transformer large model has N layers, and each layer has a weight matrix and intermediate results. The global cache stores the weight matrices of all layers. These weight matrices are loaded once when the large model is initialized and remain unchanged throughout the calculation process. This method can avoid reloading the weight matrix when calculating each layer.

[0047] Exemplarily, by analyzing the large model, the intermediate data access cache requirements are obtained, and the intermediate cache area corresponding to the intermediate data access cache requirements stores the intermediate results in the calculation process, so as to quickly transfer data between the calculation layers. For example, according to the intermediate data access requirements, the key intermediate data generated in different calculation layers are identified, and then the intermediate cache area is designed to meet the storage requirements of these intermediate data, and ensure that the cache strategy can effectively manage these data.

[0048] Exemplarily, the intermediate buffer is used to store the intermediate results of the current calculation layer, and the temporary results after each layer of calculation are retained so that they can be directly accessed in subsequent calculations, reducing repeated calculations when calculating intermediate results. After each layer of calculation is completed, the intermediate results will be stored in the intermediate buffer. When you need to access these intermediate results, you can read them directly from the intermediate buffer without recalculating. For example, after the calculation of the nth layer is completed, its output results will be stored in the intermediate buffer for use in the calculation of the n+1th layer.

[0049] Exemplarily, by analyzing the large model, the cache requirements to be accessed are obtained, and then the data to be accessed during the calculation process is stored through the cache requirements to be accessed, which is usually used for data preparation of the current calculation step. For example, by predicting the data to be accessed, the capacity and management strategy of the local cache area are set, so that the local cache area is designed to quickly access the data required for the current step, reducing the data access delay during the calculation process.

[0050] Exemplarily, the local cache is used to store data that is about to be accessed, ensuring that the data has been loaded into the local cache when the calculation requires it. The local cache can load the data required for the next calculation in advance in order to reduce the waiting time for data transmission during the calculation process. The required data is loaded into the local cache in advance so that the calculation can be performed continuously. For example, when calculating the nth layer, the input data required for the n+1th layer can be loaded into the local cache in advance. When the calculation is completed, the data is already in the local cache and can be used directly, reducing the waiting time for data transmission.

[0051] Specifically, based on the high-frequency access cache requirements, designing a global cache area to store large model weight matrices and high-frequency data can ensure that frequent data requests can be responded to quickly. Based on the intermediate data access cache requirements, designing an intermediate cache area to store intermediate calculation results can ensure fast access to intermediate results during the calculation process. Based on the upcoming access cache requirements, designing a local cache area to store data that will be accessed in the current calculation step can ensure that the data in the calculation step is fully prepared and timely. Therefore, based on the above cache strategy, it can effectively support the computing needs of large models and improve computing efficiency and performance.

[0052] Step S102: Decompose the large model into tasks to obtain multiple subtasks corresponding to the large model.

[0053] Exemplarily, the various components of the large model are obtained, including the input layer, hidden layer, output layer and the connections between them, and the functions and effects of each layer in the large model are analyzed to facilitate the subsequent sub-task division.

[0054] For example, the large model is decomposed into manageable components, such as layers, neurons, parameters (such as weights and biases), etc. The method of task decomposition is determined according to the model type (such as convolutional network CNN, recurrent network RNN, fully connected network FCN, etc.). It can also be decomposed based on data flow or computational flow, taking into account memory usage, communication overhead, parallel capability, etc. In this way, tasks are decomposed according to the model structure and the connection relationship between model structures to obtain multiple subtasks corresponding to the large model.

[0055] In some embodiments, the large model includes a self-attention mechanism and a feedforward neural network, and the task decomposition of the large model to obtain multiple subtasks corresponding to the large model includes: obtaining a self-attention head corresponding to the self-attention mechanism, and obtaining multiple first tasks corresponding to the self-attention mechanism based on the self-attention head; task decomposing the feedforward neural network to obtain multiple second tasks corresponding to the feedforward neural network; and obtaining multiple subtasks corresponding to the large model based on multiple first tasks and multiple second tasks.

[0056] Exemplarily, the large model includes a self-attention mechanism and a feedforward neural network. The self-attention mechanism is usually composed of multiple self-attention heads, and each self-attention head independently calculates its attention score and weighted sum. Then the self-attention layer in the large model is identified and all self-attention heads are extracted. Each self-attention head will be regarded as an independent subtask, which is mainly responsible for calculating its specific attention weight and weighted result. Thus, multiple first tasks are obtained according to the self-attention head.

[0057] For example, the large model includes a self-attention mechanism and a feedforward neural network, and then the computational task of the large model is decomposed into multiple subtasks. For example, in the self-attention mechanism, the computational task of each head is decomposed into independent subtasks. In the self-attention mechanism, each head independently calculates its attention score and output. The computational task of each head is decomposed into independent subtasks so that these tasks can be executed in parallel. For example, if the self-attention mechanism has 8 heads, the computational task can be decomposed into 8 independent subtasks, each of which calculates the attention score and output of a head.

[0058] For example, understand the structure of a feedforward neural network. A feedforward network is usually composed of multiple fully connected layers, each of which is processed by an activation function. Determine the number of layers of the feedforward neural network and the computational operations of each layer. Then define the computational task of each layer of the feedforward neural network as a "second task".

[0059] For example, depending on the number of layers in a feedforward neural network, each layer is defined as an independent computing task. For example, if the feedforward neural network has 4 layers, there will be 4 "second tasks".

[0060] Exemplarily, each subtask consists of a computation task of a self-attention head and a computation task of a feedforward neural network. Thus, the tasks of the self-attention head and the feedforward neural network are combined, and each combination represents a subtask. These subtasks may include the self-attention computation and the forward propagation step of the feedforward network.

[0061] In addition, since the self-attention mechanism and the feedforward network are usually processed serially, it is necessary to ensure the dependencies of the subtasks. For example, the self-attention calculation is usually completed before the feedforward network calculation, and then define parallel and serial execution strategies for each subtask to ensure that computing resources are effectively utilized and that the dependencies between subtasks are satisfied.

[0062] Specifically, decomposing the self-attention mechanism and feedforward neural network into multiple tasks, and then obtaining subtasks of the large model based on these tasks, can significantly improve computing efficiency, optimize resource utilization, improve the scalability and maintainability of the model, and improve training and reasoning performance. In addition, this task decomposition also helps to enhance the robustness of the model and ensure the stability and reliability of the system when processing complex tasks.

[0063] Step S103: Setting a data pre-fetching mechanism, storing the initial running data required during the running of the subtask into the hierarchical cache area according to the data pre-fetching mechanism.

[0064] For example, implementing a data pre-fetch mechanism and storing the initial running data required during the subtask running process in a hierarchical cache involves the entire process from demand analysis to design, implementation, and testing. Through an effective data pre-fetch mechanism and hierarchical cache, data access speed can be significantly improved, system performance can be optimized, and system responsiveness can be enhanced.

[0065] In some embodiments, the data prefetch mechanism is set up to include a global prefetch mechanism, a hierarchical prefetch mechanism, and a local prefetch mechanism. The data prefetch mechanism is set up to include: obtaining first access data corresponding to the high-frequency access cache requirement according to the global prefetch mechanism, and storing the first access data in the global cache area; obtaining second access data corresponding to the intermediate data access cache requirement according to the hierarchical prefetch mechanism, and storing the second access data in the intermediate cache area; obtaining third access data corresponding to the upcoming access cache requirement according to the local prefetch mechanism, and storing the third access data in the local cache area.

[0066] Exemplarily, setting the data prefetch mechanism includes a global prefetch mechanism, a hierarchical prefetch mechanism and a local prefetch mechanism.

[0067] Exemplarily, the global prefetch mechanism is used to obtain high-frequency access data, usually data frequently accessed in the entire system. A global prefetch algorithm is designed, such as based on access frequency statistics, global cache prediction, etc., to determine which data needs to be prefetched.

[0068] Exemplarily, the hierarchical prefetch mechanism is used to obtain access requirements for intermediate data that are accessed in a specific hierarchy or module. A hierarchical prefetch algorithm is designed, for example, based on a hierarchical access model or a modular prefetch model to determine the data that needs to be prefetched.

[0069] Exemplarily, the local prefetch mechanism is used to obtain data to be accessed, usually data in a local area. A local prefetch algorithm is designed, such as based on data access prediction, local cache strategy, etc., to determine the data to be prefetched.

[0070] Exemplarily, first access data corresponding to high-frequency access cache requirements is obtained according to the global pre-fetch mechanism, and a request is sent to the data source. The first access data is loaded from the data source. The data is stored in the global cache area according to the global cache strategy.

[0071] Exemplarily, the second access data corresponding to the intermediate data access cache requirement is obtained according to the hierarchical prefetch mechanism, and a request is issued to the data source, thereby loading the second access data from the data source, thereby storing the data in the intermediate cache area according to the hierarchical cache strategy.

[0072] Exemplarily, the third access data corresponding to the upcoming access cache demand is obtained according to the local prefetch mechanism, and a request is sent to the data source to load the third access data from the data source, and then the data is stored in the local cache area according to the local cache strategy.

[0073] Specifically, the global pre-fetch mechanism, hierarchical pre-fetch mechanism and local pre-fetch mechanism are effectively implemented, and the corresponding data is stored in the global cache, intermediate cache and local cache. This will significantly improve the data access speed, optimize system performance, and enhance the overall system responsiveness.

[0074] In some embodiments, storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data prefetch mechanism includes: when the data prefetch mechanism is the global prefetch mechanism, determining the weight matrix and input data required for calculation of each layer in the subtask according to the global prefetch mechanism; and loading the weight matrix and input data required for calculation of the next layer in the subtask into the global cache area.

[0075] Exemplarily, when the data prefetch mechanism is a global prefetch mechanism, it is necessary to determine the weight matrix and input data required for calculation of each layer according to the global prefetch mechanism, and load these data into the global cache.

[0076] Exemplarily, determine the computational flow of the subtask, in particular the weight matrix and input data required for each layer of computation, and then analyze the data access pattern in the global scope to identify which data will be frequently used or will be used in the computation process. Thus, according to the computational requirements of the subtask, identify the weight matrix and input data required for each layer of computation. Then, according to the global prefetching strategy, trigger the prefetching of the data required for the next layer of computation. Usually, this is done before the computation of the current layer begins or during the computation. Transfer the weight matrix and input data from a storage source (such as main memory, disk, or remote storage) to a global cache.

[0077] Specifically, through the above steps, the global pre-fetch mechanism can be effectively implemented to determine the weight matrix and input data required for each layer of calculation, and load these data into the global cache. Such an implementation will significantly improve computing efficiency and system performance and optimize the data access process.

[0078] In some embodiments, determining the weight matrix and input data required for calculation of each layer in the subtask according to the global prefetch mechanism includes: determining the global dependency corresponding to the subtask according to the global prefetch mechanism; analyzing the calculation graph of the large model using the global prefetch strategy according to the global dependency; determining the weight matrix and the input data required for calculation of each layer in the subtask according to the calculation graph.

[0079] Exemplarily, the global prefetching mechanism may involve a global view of the entire large model, including the global dependencies between each subtask, that is, the dependencies between subtasks or between computational stages within a subtask. The subtasks and their data dependencies are identified and recorded, and then the global dependencies are represented using graph theory methods (such as directed graphs and topological sorting). The computational graph is manually or automatically constructed based on the model structure and computational flow combined with the global dependencies, and it is ensured that the computational graph covers all computational nodes and data transmission relationships.

[0080] Exemplarily, the calculation graph is analyzed according to the global pre-fetching strategy, a data pre-fetching plan is formulated, and the data nodes and order to be pre-fetched are determined, thereby identifying the weight matrix and input data required for each layer of calculation.

[0081] For example, the global prefetch mechanism prefetches the data that may be needed for the next layer in advance based on the global dependencies of the large model calculation. The global prefetch strategy analyzes the calculation graph of the entire large model, predicts the data required for each layer of calculation, and loads this data before the calculation begins. When the large model is initialized, the calculation graph of the entire large model is analyzed to determine the weight matrix and input data required for each layer of calculation. During the large model calculation process, according to the global prefetch strategy, the data required for the next layer of calculation is loaded into the global cache in advance. For example, when calculating the first layer, the weight matrix and input data required for the second layer are loaded into the global cache in advance.

[0082] Specifically, the global dependencies of subtasks are determined, the computation graph is analyzed using a global prefetching strategy, and the weight matrix and input data required for each layer of computation are determined. This will help to efficiently manage computing resources, optimize the model computation process, and improve the performance of the overall system.

[0083] In some embodiments, the initial running data required during the running of the subtask is stored in the hierarchical cache area according to the data prefetch mechanism, including: when the data prefetch mechanism is the hierarchical prefetch mechanism, the hierarchical prefetch mechanism predicts the first target data required for the next layer corresponding to the current layer based on the calculation result of the current layer before each layer is calculated, and then loads the first target data into the intermediate cache area.

[0084] For example, when the data prefetching mechanism is a hierarchical prefetching mechanism, the data flow and calculation process of each layer in the subtask are clarified. The calculation result of each layer will affect the input data of the next layer, so a mechanism needs to be designed to predict and load the data required by the next layer.

[0085] Exemplarily, after the current layer is calculated, the next layer calculates the required input data. These data can be intermediate results, feature maps, matrices, etc., and then based on the calculation results of the current layer, a data prediction mechanism is designed to predict the first target data required for the next layer. For example, static analysis is performed according to predefined rules and data dependencies to predict the first target data required for the next layer. Or the calculation results of the current layer are dynamically evaluated during the calculation process to predict the data requirements of the next layer to obtain the first target data.

[0086] Exemplarily, an intermediate buffer is configured for storing the predicted first target data. The intermediate buffer should have sufficient capacity to store the data to be used in the next layer to avoid frequent memory read and write operations.

[0087] Specifically, effectively pre-fetching the first target data required for the next layer of calculation can reduce data access delay and improve the computing efficiency of the system.

[0088] In some embodiments, the hierarchical prefetching mechanism predicts the first target data required for the next layer corresponding to the current layer based on the calculation result of the current layer before each layer is calculated, including: determining the connection relationship between the current layer and the next layer corresponding to the current layer based on the subtask; performing data processing on the calculation result of the current layer based on the connection relationship to obtain the first target data required for the next layer.

[0089] For example, clarify the subtasks of the current task and their computational requirements. This can include the specific functions of each layer, the relationship between input and output data. And sort out the dependencies between each layer and the next layer. Usually, this involves how the computational results of the current layer affect the input data of the next layer. Thus, a graph or mapping is constructed to describe how the output of each layer is used as the input of the next layer. This can be a graph structure in which each node represents a computational layer and each edge represents the connection relationship between layers. In addition, data dependencies need to be clarified, such as whether the output of the current layer needs to be split, transposed, or converted to adapt to the input format of the next layer.

[0090] Exemplarily, the output of the current layer is calculated. For example, in a neural network, this may be a feature map or a matrix, and the calculation results of the current layer are saved for subsequent processing. Then, the calculation results of the current layer are subjected to necessary data conversion according to the connection relationship between the layers. This may include data rearrangement, data fusion, data extraction, and data transformation to meet the requirements of the next layer. Thus, the processed data is prepared as the input of the next layer. Ensure that the data format and content meet the requirements of the next layer, and obtain the first target data.

[0091] For example, before each layer of calculation, the first target data that may be needed by the next layer is predicted based on the calculation results of the current layer, and loaded into the intermediate cache. After each layer of calculation is completed, the hierarchical prefetch mechanism prepares the data required for the next layer in advance to ensure that the data is already in the intermediate cache when the next layer is calculated. After each layer of calculation is completed, the output of the current layer is analyzed to determine the first target data required for the next layer. For example, after the calculation of the nth layer is completed, the data required for the n+1th layer is immediately analyzed and loaded into the intermediate cache in advance. In this way, when the n+1th layer starts to calculate, the required data is already in the intermediate cache and can be used directly.

[0092] Specifically, by determining the hierarchical connection relationship according to the subtasks and performing data processing on the calculation results of the current layer, the computing efficiency, performance and stability of large models can be significantly improved, while optimizing resource management and system design. These steps help to build an efficient, reliable and flexible computing system to provide better support for complex computing tasks.

[0093] In some embodiments, storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data prefetch mechanism includes: when the data prefetch mechanism is the local prefetch mechanism, using the local prefetch mechanism to obtain the second target data to be accessed in advance in each calculation step of the subtask, and loading the second target data into the local cache area.

[0094] Exemplarily, the local prefetch mechanism aims to predict the data to be accessed and load it in advance to reduce data access latency. The key is to predict the data required for the next step based on the data access pattern in the calculation step and load it into the local cache in advance.

[0095] Exemplarily, the data access pattern in the calculation step is obtained, such as continuous access, random access or repeated access, to determine the pre-fetch strategy, and then the data required for the next step is predicted according to the data access pattern of the current step.

[0096] Exemplarily, at the beginning of each calculation step, the calculation task of the current step is executed and the data access pattern is recorded, and then a prediction model or algorithm is used to analyze the data access situation of the current step to predict the second target data required for the next step, and then based on the prediction result, the second target data to be accessed is loaded into the local cache area in advance.

[0097] Specifically, by implementing a local prefetch mechanism, computing efficiency and system performance can be significantly improved. This mechanism reduces data access latency and improves the overall system response speed and computing throughput by predicting data requirements, loading data into local caches in advance, and using this data efficiently in the calculation steps.

[0098] In some embodiments, the use of the local prefetch mechanism to obtain the second target data to be accessed in advance in each computing step of the subtask includes: using the local prefetch mechanism to perform data flow analysis between each computing step of the subtask to obtain the data flow relationship between the computing steps; and obtaining the second target data to be accessed in advance based on the data flow relationship.

[0099] For example, the overall task is broken down into several subtasks, each of which contains multiple computational steps. The input and output data of each computational step is determined, and how the data is transferred between computational steps is understood. Then a data flow graph or dependency graph is constructed to describe the data flow relationship between computational steps.

[0100] Exemplarily, the data flow relationship and the current calculation step are used to determine the second target data to be accessed in the next calculation step. Thus, when the current calculation step starts, the data to be accessed is loaded into the local cache in advance. For example, while the current step is being executed, the second target data required for the next step is pre-fetched asynchronously. Alternatively, the second target data may be loaded into the local cache after the current step ends but before the next step starts.

[0101] For example, in each calculation step, the data to be accessed is loaded into the local cache in advance. The local prefetch mechanism loads the data to be used in advance at each calculation step to ensure that the data is already in the local cache when needed. In each calculation step, the data required for the next step is loaded in advance according to the progress of the current calculation. For example, in the current calculation step, the input data required for the next calculation step is loaded into the local cache in advance. In this way, after the current calculation step is completed, the data required for the next step is already in the local cache, reducing the waiting time for data transmission.

[0102] Specifically, by using the local prefetch mechanism to perform data flow analysis on each computing step of the subtask, the data flow relationship between the computing steps is obtained, and the second target data to be accessed is obtained in advance based on these relationships, which can significantly improve computing efficiency, system performance and stability, optimize resource usage, and improve user experience. This mechanism not only makes the data processing process smoother, but also provides more flexibility and scalability for system design and maintenance.

[0103] Step S104: Obtain the initial operation data from the hierarchical cache area, perform operation calculation on the subtask, and obtain intermediate operation data corresponding to the subtask.

[0104] Exemplarily, a hierarchical structure of layered cache areas is determined, and the cache size and type of each layer are configured.

[0105] Select appropriate cache strategies for each cache layer to optimize data access speed. Then load the initial operation data into the appropriate cache layer to ensure that the computing task can quickly access the data. According to the needs of the computing subtask, the corresponding initial operation data is obtained from the appropriate cache area of ​​the layered cache area for operation calculation to obtain the corresponding intermediate operation data of the subtask.

[0106] Exemplarily, by obtaining initial operation data from the hierarchical cache area and performing operation calculations on the subtasks, cache resources can be effectively utilized to improve calculation efficiency, thereby quickly obtaining intermediate operation data corresponding to the subtasks.

[0107] In some embodiments, the obtaining of the initial running data from the hierarchical cache area to perform running calculations on the subtasks to obtain intermediate running data corresponding to the subtasks includes: using a multi-threaded library to assign different computing resources to the subtasks to obtain target threads corresponding to the subtasks; and performing parallel execution calculations on the subtasks using the initial running data according to the target threads to obtain the intermediate running data corresponding to each of the subtasks.

[0108] For example, the entire large model is split into multiple subtasks, each of which can be processed independently and has its own input data. Ensure that the calculation logic of each subtask is independent and does not cause data competition problems.

[0109] Exemplarily, the size of the thread pool (number of threads) is determined. Usually, the size of the thread pool can be set according to the number of processor cores in the system, and the commonly used size is several times the number of processor cores. Submit the subtask to the thread pool. Each subtask needs to pass its required input data for use in calculation. Pass the initial running data to the thread in the thread pool together with the subtask. The thread pool then assigns the subtask to the idle thread. Each thread independently processes the assigned subtask, and the runtime resources (such as CPU time, memory) are scheduled and managed by the operating system and thread library. The target thread corresponding to each subtask is obtained.

[0110] Exemplarily, the target thread performs parallel operations on the subtasks using the initial running data to obtain the intermediate running data corresponding to each subtask. In addition, to ensure that any target thread safety issues are handled during the calculation process. If the subtasks need to share data, use appropriate synchronization mechanisms (such as locks, conditional variables) to prevent data competition and consistency issues.

[0111] Exemplarily, through multithreading technology, the decomposed subtasks are executed in parallel to reduce the waiting time in the calculation process. For example, in a feedforward neural network, the calculation tasks of each layer are assigned to different threads for parallel execution. Using multithreading technology, the decomposed subtasks are assigned to different computing resources for parallel execution, and each subtask is assigned to a different thread or graphics processing unit (GPU) core using a multithreading library, so that these tasks can be calculated simultaneously, reducing the overall calculation time.

[0112] Specifically, using a multithreaded library to assign subtasks to different computing resources and perform parallel computing can significantly improve computing efficiency and system performance.

[0113] Step S105: merge all the intermediate operation data to obtain the target operation data corresponding to the large model.

[0114] For example, in the process of parallel computing, the intermediate running data generated by each subtask needs to be effectively collected and stored. This data may be saved in different threads or stored in a distributed system. Ensuring that all intermediate running data is correctly collected is a prerequisite for successful merging.

[0115] For example, before merging, the intermediate operation data may need to be preprocessed to ensure that its format and structure are consistent so that the subsequent merging operation can proceed smoothly. Then, according to the type of the intermediate operation data and the requirements of the merging, an appropriate merging strategy is selected. For example, the intermediate operation data is spliced ​​according to a specific rule or structure, and then the merged data is further calculated as needed to generate the final target operation data.

[0116] In some embodiments, merging all of the intermediate operating data to obtain the target operating data corresponding to the large model includes: determining the degree of influence of the subtask on the large model, and determining the degree of difficulty corresponding to the subtask; determining the target weight corresponding to the subtask based on the degree of influence and the degree of difficulty; weighting all of the intermediate operating data and the target weight to obtain the target operating data corresponding to the large model.

[0117] For example, the degree of influence describes the contribution of each subtask to the final large model result. Common definition methods include: analyzing the contribution of the intermediate operation data generated by each subtask to the target operation data of the large model, or combining the knowledge of domain experts to determine the degree of influence of the subtask according to the actual business and model complexity. Thus, the impact of each subtask on the output of the large model can be analyzed in the large model through model interpretation tools or feature importance analysis.

[0118] Exemplarily, the difficulty level describes the resources and time required to complete each subtask. Common definition methods include: analyzing the computational complexity of each subtask, including time complexity and space complexity. Or evaluating the computational resources required for each subtask, such as CPU, memory, etc., or considering the difficulty of implementing the subtask, such as algorithm complexity, development difficulty, etc., thereby evaluating the difficulty of the subtask based on historical data or runtime performance indicators, or evaluating the difficulty of the task based on the experience of developers or engineers.

[0119] Exemplarily, the impact and difficulty are combined to calculate the target weight of each subtask. For example, weights are assigned to the impact and difficulty, and then the target weight is calculated by linear combination. For example, target weight = α × impact + β × difficulty, where α and β are weight coefficients.

[0120] It should be noted that the present application can also adjust the calculation method of the target weight according to specific needs and actual conditions.

[0121] Exemplarily, the intermediate operation data generated by each subtask is obtained, and then the intermediate operation data is weighted by using the target weight to achieve data merging, thereby generating the final large model target operation data.

[0122] Specifically, by evaluating and weighting the impact and difficulty of subtasks, the overall performance of the model can be significantly improved. Focusing on optimizing subtasks that have a great impact on the results and arranging resources reasonably can effectively improve the accuracy and stability of the model. And by weighting the intermediate operation data, more accurate and reliable target operation data can be generated, improving the quality of data processing and the credibility of the results.

[0123] This application introduces hierarchical cache optimization technology to reasonably manage and allocate cache resources, reduce redundant storage of intermediate results and weight matrices, and thus reduce video memory occupancy. Through hierarchical management of global cache, intermediate cache and local cache, it is ensured that high-frequency access data is stored first and the utilization efficiency of video memory is improved. In addition, through data pre-fetching mechanism and parallel computing collaborative technology, the required data is loaded in advance, the waiting time of inter-layer data transmission is reduced, and delay hiding is achieved. Through multi-threaded parallel execution of self-attention and feedforward network calculations, the response speed and real-time performance of the model are improved. Then, through task decomposition and parallel computing technology, the computing tasks are assigned to multiple computing units for parallel execution to maximize the utilization efficiency of computing resources. Combined with cache optimization strategy, the utilization rate of computing resources is further improved and the overall energy consumption is reduced. Finally, a hierarchical cache management strategy is designed to dynamically adjust the data allocation and replacement strategy in the cache according to the computing characteristics of the model to improve the cache hit rate. Through the collaborative work of global cache, intermediate cache and local cache, the data transmission overhead is reduced and the reasoning efficiency of the model is improved.

[0124] See also Figure 2 , Figure 2 A model optimization device 200 is provided in an embodiment of the present application, and the model optimization device 200 includes a cache partition module 201, a task decomposition module 202, a data storage module 203, a task calculation module 204, and a data merging module 205, wherein the cache partition module 201 is used to set a hierarchical cache area, and the hierarchical cache area is used to cache data during the operation of a large model; the task decomposition module 202 is used to decompose the large model into tasks to obtain multiple subtasks corresponding to the large model; the data storage module 203 is used to set a data prefetching mechanism, and according to the data prefetching mechanism, the initial operation data required during the operation of the subtask is stored in the hierarchical cache area; the task calculation module 204 is used to obtain the initial operation data from the hierarchical cache area to perform operation calculation on the subtask, and obtain the intermediate operation data corresponding to the subtask; the data merging module 205 is used to merge all the intermediate operation data to obtain the target operation data corresponding to the large model.

[0125] In some implementations, the hierarchical cache area includes a global cache area, an intermediate cache area, and a local cache area. In the process of setting the hierarchical cache area, the cache partitioning module 201 performs:

[0126] Determine data cache requirements, wherein the data cache requirements include high-frequency access cache requirements, intermediate data access cache requirements, and upcoming access cache requirements;

[0127] Determine the global cache area according to the high-frequency access cache requirement, the global cache area is used to store the weight matrix and high-frequency access data corresponding to the large model;

[0128] Determine the intermediate cache area according to the intermediate data access cache requirement, wherein the intermediate cache area is used to store the intermediate results of the current computing layer;

[0129] The local cache area is determined according to the cache demand to be accessed, and the local cache area is used to store data to be accessed in the current step of the calculation process.

[0130] In some implementations, the data pre-fetching mechanism includes a global pre-fetching mechanism, a hierarchical pre-fetching mechanism, and a local pre-fetching mechanism. In the process of setting the data pre-fetching mechanism, the data storage module 203 executes:

[0131] Obtaining first access data corresponding to the high-frequency access cache demand according to the global prefetch mechanism, and storing the first access data in the global cache area;

[0132] Obtaining second access data corresponding to the intermediate data access cache requirement according to the hierarchical prefetch mechanism, and storing the second access data in the intermediate cache area;

[0133] The third access data corresponding to the upcoming access cache requirement is obtained according to the local pre-fetch mechanism, and the third access data is stored in the local cache area.

[0134] In some implementations, the data storage module 203 performs, during the process of storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data pre-fetching mechanism:

[0135] When the data pre-fetching mechanism is the global pre-fetching mechanism, the weight matrix and input data required for calculation of each layer in the subtask are determined according to the global pre-fetching mechanism;

[0136] According to the method, the weight matrix and the input data required for the next layer calculation in the subtask are loaded into the global buffer.

[0137] In some implementations, the data storage module 203 performs, during the process of determining the weight matrix and input data required for calculation of each layer in the subtask according to the global pre-fetching mechanism:

[0138] Determine the global dependency relationship corresponding to the subtask according to the global prefetch mechanism;

[0139] Analyzing the computation graph of the large model using a global prefetching strategy according to the global dependency;

[0140] The weight matrix and the input data required for calculation of each layer in the subtask are determined according to the calculation graph.

[0141] In some implementations, the data storage module 203 performs, during the process of storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data pre-fetching mechanism:

[0142] When the data prefetching mechanism is the hierarchical prefetching mechanism, the hierarchical prefetching mechanism predicts the first target data required by the next layer corresponding to the current layer based on the calculation result of the current layer before each layer is calculated, and then loads the first target data into the intermediate cache area.

[0143] In some implementations, the data storage module 203 performs the following steps in the process of predicting the first target data required for the next layer corresponding to the current layer according to the calculation result of the current layer before calculating each layer according to the hierarchical prefetching mechanism:

[0144] Determine, according to the subtask, a connection relationship between the current layer and the next layer corresponding to the current layer;

[0145] Data processing is performed on the calculation result of the current layer according to the connection relationship to obtain the first target data required by the next layer.

[0146] In some implementations, the data storage module 203 performs, during the process of storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data pre-fetching mechanism:

[0147] When the data prefetch mechanism is the local prefetch mechanism, the local prefetch mechanism is used to obtain the second target data to be accessed in advance in each calculation step of the subtask, and the second target data is loaded into the local cache area.

[0148] In some implementations, the data storage module 203 performs the following steps in the process of obtaining the second target data to be accessed in advance in each computing step of the subtask by using the local prefetch mechanism:

[0149] Using the local prefetch mechanism, data flow analysis is performed between each computing step of the subtask to obtain the data flow relationship between the computing steps;

[0150] The second target data to be accessed is obtained in advance according to the data flow relationship.

[0151] In some implementations, the task calculation module 204 performs, during the process of obtaining the initial operation data from the hierarchical cache area to perform operation calculation on the subtask and obtaining the intermediate operation data corresponding to the subtask:

[0152] Using a multithreading library to allocate different computing resources to the subtasks, and obtaining target threads corresponding to the subtasks;

[0153] The target thread executes the subtasks in parallel using the initial operation data to obtain the intermediate operation data corresponding to each subtask.

[0154] In some implementations, the data merging module 205 performs, during the process of merging all the intermediate operation data to obtain the target operation data corresponding to the large model:

[0155] Determine the degree of influence of the subtask on the large model, and determine the degree of difficulty corresponding to the subtask;

[0156] Determine the target weight corresponding to the subtask according to the impact degree and the difficulty degree;

[0157] All the intermediate operating data and the target weights are weighted to obtain the target operating data corresponding to the large model.

[0158] In some implementations, the large model includes a self-attention mechanism and a feedforward neural network, and the task decomposition module 202 performs, during the process of decomposing the large model to obtain a plurality of subtasks corresponding to the large model:

[0159] Obtaining a self-attention head corresponding to the self-attention mechanism, and obtaining a plurality of first tasks corresponding to the self-attention mechanism according to the self-attention head;

[0160] Decomposing the feedforward neural network into tasks to obtain a plurality of second tasks corresponding to the feedforward neural network;

[0161] A plurality of the subtasks corresponding to the large model are obtained according to a plurality of the first tasks and a plurality of the second tasks.

[0162] In some implementations, the model optimization apparatus 200 may be applied to a terminal device.

[0163] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the model optimization device 200 described above can refer to the corresponding process in the aforementioned model optimization method embodiment, and will not be repeated here.

[0164] See also Figure 3 , Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0165] like Figure 3 As shown, the terminal device 300 includes a processor 301 and a memory 302 , and the processor 301 and the memory 302 are connected via a bus 303 , such as an I2C (Inter-integrated Circuit) bus.

[0166] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0167] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0168] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0169] The processor is used to run a computer program stored in the memory, and implement any one of the model optimization methods provided by the embodiments of the present invention when executing the computer program.

[0170] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0171] Setting a hierarchical cache area, wherein the hierarchical cache area is used to cache data during the operation of the large model;

[0172] Decomposing the large model into tasks to obtain a plurality of subtasks corresponding to the large model;

[0173] Setting a data pre-fetching mechanism, storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data pre-fetching mechanism;

[0174] Obtain the initial operation data from the hierarchical cache area to perform operation calculation on the subtask to obtain intermediate operation data corresponding to the subtask;

[0175] All the intermediate operation data are merged to obtain the target operation data corresponding to the large model.

[0176] In some implementations, the hierarchical cache area includes a global cache area, an intermediate cache area, and a local cache area. In the process of setting the hierarchical cache area, the processor 301 executes:

[0177] Determine data cache requirements, wherein the data cache requirements include high-frequency access cache requirements, intermediate data access cache requirements, and upcoming access cache requirements;

[0178] Determine the global cache area according to the high-frequency access cache requirement, the global cache area is used to store the weight matrix and high-frequency access data corresponding to the large model;

[0179] Determine the intermediate cache area according to the intermediate data access cache requirement, wherein the intermediate cache area is used to store the intermediate results of the current computing layer;

[0180] The local cache area is determined according to the cache demand to be accessed, and the local cache area is used to store data to be accessed in the current step of the calculation process.

[0181] In some implementations, the data prefetching mechanism includes a global prefetching mechanism, a hierarchical prefetching mechanism, and a local prefetching mechanism. In the process of setting the data prefetching mechanism, the processor 301 executes:

[0182] Obtaining first access data corresponding to the high-frequency access cache demand according to the global prefetch mechanism, and storing the first access data in the global cache area;

[0183] Obtaining second access data corresponding to the intermediate data access cache requirement according to the hierarchical prefetch mechanism, and storing the second access data in the intermediate cache area;

[0184] The third access data corresponding to the upcoming access cache requirement is obtained according to the local pre-fetch mechanism, and the third access data is stored in the local cache area.

[0185] In some implementations, during the process of storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data pre-fetching mechanism, the processor 301 executes:

[0186] When the data pre-fetching mechanism is the global pre-fetching mechanism, the weight matrix and input data required for calculation of each layer in the subtask are determined according to the global pre-fetching mechanism;

[0187] According to the method, the weight matrix and the input data required for the next layer calculation in the subtask are loaded into the global buffer.

[0188] In some implementations, the processor 301, during the process of determining the weight matrix and input data required for calculation of each layer in the subtask according to the global prefetch mechanism, performs:

[0189] Determine the global dependency relationship corresponding to the subtask according to the global prefetch mechanism;

[0190] Analyzing the computation graph of the large model using a global prefetching strategy according to the global dependency;

[0191] The weight matrix and the input data required for calculation of each layer in the subtask are determined according to the calculation graph.

[0192] In some implementations, during the process of storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data pre-fetching mechanism, the processor 301 executes:

[0193] When the data prefetching mechanism is the hierarchical prefetching mechanism, the hierarchical prefetching mechanism predicts the first target data required by the next layer corresponding to the current layer based on the calculation result of the current layer before each layer is calculated, and then loads the first target data into the intermediate cache area.

[0194] In some implementations, the processor 301 performs, in the process of predicting the first target data required by the next layer corresponding to the current layer according to the calculation result of the current layer before calculating each layer according to the hierarchical prefetching mechanism, the following steps:

[0195] Determine, according to the subtask, a connection relationship between the current layer and the next layer corresponding to the current layer;

[0196] Data processing is performed on the calculation result of the current layer according to the connection relationship to obtain the first target data required by the next layer.

[0197] In some implementations, during the process of storing the initial running data required during the running of the subtask in the hierarchical cache area according to the data pre-fetching mechanism, the processor 301 executes:

[0198] When the data prefetch mechanism is the local prefetch mechanism, the local prefetch mechanism is used to obtain the second target data to be accessed in advance in each calculation step of the subtask, and the second target data is loaded into the local cache area.

[0199] In some implementations, the processor 301, in the process of using the local prefetch mechanism to obtain the second target data to be accessed in advance in each computing step of the subtask, executes:

[0200] Using the local prefetch mechanism, data flow analysis is performed between each computing step of the subtask to obtain the data flow relationship between the computing steps;

[0201] The second target data to be accessed is obtained in advance according to the data flow relationship.

[0202] In some implementations, the processor 301, in the process of obtaining the initial operation data from the hierarchical cache area to perform operation calculation on the subtask and obtaining the intermediate operation data corresponding to the subtask, executes:

[0203] Using a multithreading library to allocate different computing resources to the subtasks, and obtaining target threads corresponding to the subtasks;

[0204] The target thread executes the subtasks in parallel using the initial operation data to obtain the intermediate operation data corresponding to each subtask.

[0205] In some implementations, the processor 301, during the process of merging all the intermediate operation data to obtain the target operation data corresponding to the large model, executes:

[0206] Determine the degree of influence of the subtask on the large model, and determine the degree of difficulty corresponding to the subtask;

[0207] Determine the target weight corresponding to the subtask according to the impact degree and the difficulty degree;

[0208] All the intermediate operating data and the target weights are weighted to obtain the target operating data corresponding to the large model.

[0209] In some implementations, the large model includes a self-attention mechanism and a feedforward neural network, and the processor 301, in the process of performing task decomposition on the large model to obtain a plurality of subtasks corresponding to the large model, executes:

[0210] Obtaining a self-attention head corresponding to the self-attention mechanism, and obtaining a plurality of first tasks corresponding to the self-attention mechanism according to the self-attention head;

[0211] Decomposing the feedforward neural network into tasks to obtain a plurality of second tasks corresponding to the feedforward neural network;

[0212] A plurality of the subtasks corresponding to the large model are obtained according to a plurality of the first tasks and a plurality of the second tasks.

[0213] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned model optimization method embodiment, and will not be repeated here.

[0214] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any model optimization method provided in the description of the embodiment of the present invention.

[0215] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0216] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0217] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0218] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A model optimization method, characterized in that: The method comprises: Set a hierarchical cache area, which is used to cache data during the operation of the large model; wherein the hierarchical cache area includes a global cache area, an intermediate cache area, and a local cache area, wherein the global cache area is used to store the weight matrix and high-frequency access data corresponding to the large model, the intermediate cache area is used to store the intermediate results of the current calculation layer, and the local cache area is used to store the data to be accessed in the current step during the calculation process; Decomposing the large model into tasks to obtain a plurality of subtasks corresponding to the large model; wherein the manner of decomposing the large model into tasks includes one of the following: decomposing into manageable components, decomposing based on model type, decomposing based on data flow or computation flow; A data prefetching mechanism is set, and the initial running data required in the running process of the subtask is stored in the hierarchical cache area according to the data prefetching mechanism. The data prefetching mechanism includes a global prefetching mechanism, a hierarchical prefetching mechanism and a local prefetching mechanism; wherein the global prefetching mechanism involves a global view of the entire large model, including the global dependency relationship between each subtask, and the global dependency relationship is the dependency relationship between subtasks or between the calculation stages within the subtask; the local prefetching mechanism is used to predict the data required for the next step based on the data access pattern in the calculation step, and load it into the local cache area in advance; Obtain the initial operation data from the hierarchical cache area to perform operation calculation on the subtask to obtain intermediate operation data corresponding to the subtask; All the intermediate operation data are merged to obtain the target operation data corresponding to the large model.

2. The method according to claim 1, characterized in that The step of setting a hierarchical cache area includes: Determine data cache requirements, wherein the data cache requirements include high-frequency access cache requirements, intermediate data access cache requirements, and upcoming access cache requirements; Determining the global cache area according to the high-frequency access cache requirement; Determining the intermediate cache area according to the intermediate data access cache requirement; The local cache area is determined according to the upcoming cache access requirement.

3. The method according to claim 2, characterized in that The data pre-fetching mechanism is set, including: Obtaining first access data corresponding to the high-frequency access cache demand according to the global prefetch mechanism, and storing the first access data in the global cache area; Obtaining second access data corresponding to the intermediate data access cache requirement according to the hierarchical prefetch mechanism, and storing the second access data in the intermediate cache area; The third access data corresponding to the upcoming access cache requirement is obtained according to the local pre-fetch mechanism, and the third access data is stored in the local cache area.

4. The method according to claim 3, characterized in that The storing the initial running data required in the running process of the subtask into the hierarchical cache area according to the data pre-fetching mechanism includes: When the data pre-fetching mechanism is the global pre-fetching mechanism, the weight matrix and input data required for calculation of each layer in the subtask are determined according to the global pre-fetching mechanism; According to the method, the weight matrix and the input data required for the next layer calculation in the subtask are loaded into the global buffer.

5. The method according to claim 4, characterized in that The determining, according to the global prefetch mechanism, the weight matrix and input data required for calculation of each layer in the subtask comprises: Determine the global dependency relationship corresponding to the subtask according to the global prefetch mechanism; Analyzing the computation graph of the large model using a global prefetching strategy according to the global dependency; The weight matrix and the input data required for calculation of each layer in the subtask are determined according to the calculation graph.

6. The method according to claim 3, characterized in that: The storing the initial running data required in the running process of the subtask into the hierarchical cache area according to the data pre-fetching mechanism includes: When the data prefetching mechanism is the hierarchical prefetching mechanism, the hierarchical prefetching mechanism predicts the first target data required by the next layer corresponding to the current layer based on the calculation result of the current layer before each layer is calculated, and then loads the first target data into the intermediate cache area.

7. The method according to claim 6, characterized in that The method of predicting the first target data required by the next layer corresponding to the current layer according to the calculation result of the current layer before calculating each layer according to the hierarchical prefetching mechanism includes: Determine, according to the subtask, a connection relationship between the current layer and the next layer corresponding to the current layer; Data processing is performed on the calculation result of the current layer according to the connection relationship to obtain the first target data required by the next layer.

8. The method according to claim 3, characterized in that The storing the initial running data required in the running process of the subtask into the hierarchical cache area according to the data pre-fetching mechanism includes: When the data prefetch mechanism is the local prefetch mechanism, the local prefetch mechanism is used to obtain the second target data to be accessed in advance in each calculation step of the subtask, and the second target data is loaded into the local cache area.

9. The method according to claim 8, characterized in that The using the local prefetch mechanism to obtain the second target data to be accessed in advance in each calculation step of the subtask includes: Using the local prefetch mechanism, data flow analysis is performed between each computing step of the subtask to obtain the data flow relationship between the computing steps; The second target data to be accessed is obtained in advance according to the data flow relationship.

10. The method according to claim 1, characterized in that The obtaining the initial operation data from the hierarchical buffer area, performing operation calculation on the subtask, and obtaining the intermediate operation data corresponding to the subtask includes: Using a multithreading library to allocate different computing resources to the subtasks, and obtaining target threads corresponding to the subtasks; The target thread executes the subtasks in parallel using the initial operation data to obtain the intermediate operation data corresponding to each subtask.

11. The method according to claim 1, characterized in that: The step of merging all the intermediate operation data to obtain the target operation data corresponding to the large model includes: Determine the degree of influence of the subtask on the large model, and determine the degree of difficulty corresponding to the subtask; Determine the target weight corresponding to the subtask according to the impact degree and the difficulty degree; All the intermediate operating data and the target weights are weighted to obtain the target operating data corresponding to the large model.

12. The method according to any one of claims 1 to 11, characterized in that The large model includes a self-attention mechanism and a feedforward neural network, and the task decomposition of the large model to obtain multiple subtasks corresponding to the large model includes: Obtaining a self-attention head corresponding to the self-attention mechanism, and obtaining a plurality of first tasks corresponding to the self-attention mechanism according to the self-attention head; Decomposing the feedforward neural network into tasks to obtain a plurality of second tasks corresponding to the feedforward neural network; A plurality of the subtasks corresponding to the large model are obtained according to a plurality of the first tasks and a plurality of the second tasks.

13. A model optimization device, characterized in that: include: A cache partitioning module is used to set a hierarchical cache area, which is used to cache data during the operation of the large model; wherein the hierarchical cache area includes a global cache area, an intermediate cache area, and a local cache area, wherein the global cache area is used to store the weight matrix and high-frequency access data corresponding to the large model, the intermediate cache area is used to store the intermediate results of the current calculation layer, and the local cache area is used to store the data to be accessed in the current step during the calculation process; A task decomposition module, used to decompose the large model into tasks to obtain a plurality of subtasks corresponding to the large model; wherein the manner of decomposing the large model into tasks includes one of the following: decomposing into manageable components, decomposing based on model type, decomposing based on data flow or computation flow; A data storage module, used to set a data pre-fetch mechanism, according to which the initial running data required in the running process of the subtask is stored in the hierarchical cache area, the data pre-fetch mechanism includes a global pre-fetch mechanism, a hierarchical pre-fetch mechanism and a local pre-fetch mechanism; wherein the global pre-fetch mechanism involves a global view of the entire large model, including the global dependency relationship between each subtask, the global dependency relationship is the dependency relationship between subtasks or between the calculation stages within the subtask; the local pre-fetch mechanism is used to predict the data required for the next step based on the data access pattern in the calculation step, and load it into the local cache area in advance; A task calculation module, used to obtain the initial operation data from the hierarchical cache area, perform operation calculation on the subtask, and obtain intermediate operation data corresponding to the subtask; The data merging module is used to merge all the intermediate operation data to obtain the target operation data corresponding to the large model.

14. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the model optimization method according to any one of claims 1 to 12 when executing the computer program.

15. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the model optimization method according to any one of claims 1 to 12.

Images