Multi-core large model calculation method and device, equipment and storage medium

By using high-bandwidth memory (HBM) directly to read and write to storage in a multi-core large-scale computing system, the problem of low hardware resource utilization in traditional methods is solved, and more efficient hardware resource utilization and performance improvement is achieved.

CN119065847APending Publication Date: 2024-12-03STORAGEX TECHNOLOGY INC
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411172163.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The traditional multi-core large-model computing method focuses on the superposition of computing resources, resulting in low utilization of hardware computing resources and wasted hardware costs.

Method used

High bandwidth memory (HBM) is used to open memory to the entire system, eliminating data transfer between memory and memory. Each computing core can directly read and write access to the storage, making full use of hardware computing resources.

Benefits of technology

It improves the utilization rate of hardware computing resources, maximizes the utilization of storage bandwidth and computing resources, reduces hardware costs, and improves the performance of large-scale model applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065847A_ABST
    Figure CN119065847A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-core large model calculation method, device and equipment and a storage medium, and is applied to the field of artificial intelligence, and the method comprises the steps that a central processing unit obtains a large model weight file and model text data; the central processing unit caches the large model weight file into a high-bandwidth memory unit; the central processing unit converts the model text data into embedded vector data and writes the embedded vector data into the core computing unit; the core computing unit reads the large model weight file in the high-bandwidth memory unit, and processes the embedded vector data according to weight data in the large model weight file and the number of middle layers of the large model network structure to generate attention parameters; and the core calculation unit processes the attention parameters according to an output layer structure of the large model network structure and outputs a large model calculation result. The method has the technical effects that the utilization rate of hardware computing power resources is improved, and the hardware cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multi-core large model computing method, device, equipment and storage medium. Background Art

[0002] With the rise of new technologies such as AI big models and intelligent driving, users have an increasing demand for high-bandwidth memory. Due to the increase in data volume, AI servers need to process a large amount of data in a short period of time, which makes AI servers have higher requirements for bandwidth. HBM (High Bandwidth Memory) can be the basic configuration of AI servers. HBM has ultra-high bandwidth capabilities. The application of HBM can significantly improve the performance of large model applications for computing and storage. In addition, HBM can be used in the automotive field. There are many cameras in the car, and the amount of data that all cameras in the car system need to process and transmit is extremely large. The use of HBM technology can increase bandwidth to improve data processing and data transmission capabilities.

[0003] In the process of large model data processing, it is particularly important to make full use of HBM's ultra-high bandwidth capabilities to optimize the hardware's computing resources. The traditional calculation method of multi-core large models focuses on the superposition of computing resources, which makes computing resources very redundant. However, this method will lead to low utilization of hardware computing resources, thereby wasting hardware costs. Summary of the invention

[0004] In order to help solve the problem that traditional multi-core large model calculation methods focus on the superposition of computing resources, which makes computing resources have a large surplus, resulting in low utilization of hardware computing resources and waste of hardware costs, the present application provides a multi-core large model calculation method, device, equipment and storage medium.

[0005] In a first aspect, the present application provides a multi-core large model computing method, which adopts the following technical solution: the method is applied to a multi-core large model computing system, the multi-core large model computing system includes a central processing unit, a high-bandwidth memory unit and a core computing unit, and the method includes:

[0006] The central processing unit obtains a large model weight file and model text data;

[0007] The central processing unit caches the large model weight file in the high bandwidth memory unit;

[0008] The central processing unit converts the model text data into embedded vector data, and writes the embedded vector data into the core computing unit;

[0009] The core computing unit reads the large model weight file in the high bandwidth memory unit, processes the embedded vector data according to the weight data in the large model weight file and the number of layers of the middle layer of the large model network structure, and generates attention parameters;

[0010] The core computing unit processes the attention parameters according to the output layer structure of the large model network structure and outputs the large model calculation results.

[0011] In a specific implementation scheme, the large model includes a vocabulary, and the central processing unit converts the model text data into embedded vector data including:

[0012] The central processing unit segments the model text data according to the vocabulary to generate a plurality of word units;

[0013] The central processing unit searches for positions of a number of the word elements in the word library;

[0014] The central processing unit generates the embedded vector data according to a plurality of the word units and positions corresponding to the word units.

[0015] In a specific implementation scheme, the core computing unit reads the large model weight file in the high bandwidth memory unit, processes the embedded vector data according to the weight data in the large model weight file and the number of layers of the middle layer of the large model network structure, and generates the attention parameter including:

[0016] The core computing unit identifies the number of intermediate layers of the large model network structure;

[0017] The core computing unit reads the weight data of the current middle layer in the large model weight file from the high bandwidth memory unit;

[0018] The core computing unit processes the embedded vector data using the basic operators in the large model according to the weight data of the current intermediate layer read to generate the attention parameters of the current layer;

[0019] The core computing unit performs cyclic calculations according to the number of intermediate layers of the identified large model network structure until the attention parameters of the last intermediate layer are calculated and output.

[0020] In a specific implementation scheme, the basic operator includes a matrix multiplication operator, and the core computing unit processes the embedded vector data using the basic operator in the large model according to the weight data of the current intermediate layer read, and generates the attention parameters of the current layer including:

[0021] The core computing unit inputs the weight data of the current intermediate layer and the embedded vector data into the matrix multiplication operator to calculate the attention parameters of the current layer.

[0022] In a specific implementation scheme, the attention parameter of the current layer includes a current layer key vector, a current layer numerical vector and a current layer query vector. After the core computing unit inputs the weight data of the current intermediate layer and the embedding vector data into the matrix multiplication operator to calculate the attention parameter of the current layer, it also includes:

[0023] The core computing unit writes the current layer key vector and the current layer value vector into the high bandwidth memory unit.

[0024] In a specific implementation scheme, the central processing unit is a CPU.

[0025] In a specific implementation scheme, the core computing unit is a GPU and / or an FPGA.

[0026] In a second aspect, the present application provides a multi-core large model computing device, which adopts the following technical solution: the device is applied to a multi-core large model computing system, the multi-core large model computing system includes a central processing unit, a high-bandwidth memory unit and a core computing unit, and the device includes:

[0027] The central processing unit is used to obtain a large model weight file and model text data;

[0028] The central processing unit is also used to cache the large model weight file in the high bandwidth memory unit;

[0029] The central processing unit is also used to convert the model text data into embedded vector data, and write the embedded vector data into the core computing unit;

[0030] The core computing unit is used to read the large model weight file in the high-bandwidth memory unit, process the embedded vector data according to the weight data in the large model weight file and the number of layers of the middle layer of the large model network structure, and generate attention parameters;

[0031] The core computing unit is also used to process the attention parameters according to the output layer structure of the large model network structure and output the large model calculation results.

[0032] In a third aspect, the present application provides a computer device, which adopts the following technical solution: it includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute any of the multi-core large model calculation methods mentioned above.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned multi-core large model calculation methods.

[0034] In summary, this application has the following beneficial technical effects:

[0035] Traditional multi-core computing architectures often need to move data from memory to the cache of their own computing cores before performing corresponding calculations or other processing. In the present application, high-bandwidth memory (HBM) is used to open the memory to the entire system, eliminating the need to move data from one memory to another. HBM (High Bandwidth Memory) is the data cache center of the entire solution. Each computing core can directly read and write to the storage. Compared with ordinary memory, the ultra-high bandwidth HBM can meet the speed requirements of high-computing power cores for data acquisition, so that the performance of the entire architecture can be maximized and the utilization rate of hardware computing power resources can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a multi-core large model calculation method in an embodiment of the present application;

[0037] Figure 2 It is a schematic diagram of the network structure of the LlaMa2 large model in the embodiment of the present application;

[0038] Figure 3 is a schematic diagram of converting model text data into embedded vector data in an embodiment of the present application;

[0039] Figure 4 is a schematic diagram of QKV data generation in an embodiment of the present application;

[0040] Figure 5 It is a flowchart of generating QKV data according to the LlaMa2 large model network structure in an embodiment of the present application;

[0041] Figure 6 It is a complete flow chart of the multi-core large model calculation method in the embodiment of the present application;

[0042] Figure 7 It is a schematic diagram of the framework of a multi-core large model computing system in an embodiment of the present application;

[0043] Figure 8 It is a schematic diagram used to embody a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following combination Figure 1-Figure 8 This application is described in further detail.

[0045] An embodiment of the present application discloses a multi-core large model computing method, which is applied to a multi-core large model computing system. The method implements data reading and writing based on high-bandwidth memory (HBM), thereby making full use of the computing resources of the hardware and maximizing the storage bandwidth and computing resources within the entire architecture.

[0046] With the rise of new technologies such as AI big models and intelligent driving, users have an increasing demand for high-bandwidth memory. Due to the increase in data volume, AI servers need to process a large amount of data in a short period of time, which makes AI servers have higher requirements for bandwidth. HBM (High Bandwidth Memory) can be the basic configuration of AI servers. HBM has ultra-high bandwidth capabilities. The application of HBM can significantly improve the performance of large model applications for computing and storage. In addition, HBM can be used in the automotive field. There are many cameras in the car, and the amount of data that all cameras in the car system need to process and transmit is extremely large. The use of HBM technology can increase bandwidth to improve data processing and data transmission capabilities.

[0047] HBM is a high-performance DRAM (Dynamic Random Access Memory) based on a 3D stacking process with ultra-high bandwidth capabilities; the HBM stack is compactly and quickly connected through an intermediate layer to achieve a larger number of IOs. Secondly, HBM has readjusted the power efficiency of memory, making the bandwidth per watt more than 3 times higher than GDDR5 (Graphics Double Data Rate, version 5), which means that HBM power consumption is reduced by more than 3 times. In addition, HBM also has certain advantages in saving product space. HBM saves 94% of the surface area compared to GDDR5, which significantly improves the performance of large model applications for computing and storage.

[0048] In the process of large model data processing, it is particularly important to make full use of the ultra-high bandwidth capability of HBM to optimize the computing power resources of the hardware; the traditional multi-core large model calculation method focuses on the superposition of computing resources, which makes the computing resources very redundant, but this method will lead to low utilization of hardware computing resources, resulting in a waste of hardware costs. In order to help improve the utilization of hardware resources and maximize the role of storage bandwidth and computing resources, this application provides a multi-core large model calculation method.

[0049] Reference Figure 1 , the method comprises the following steps:

[0050] S10, the central processing unit obtains the large model weight file and model text data.

[0051] Specifically, the multi-core large model computing system includes a central processing unit, a high-bandwidth memory unit and a core computing unit; the central processing unit can also be understood as a central processing unit, and the high-bandwidth memory unit can be understood as a high-bandwidth memory, that is, HBM. The central processing unit obtains a large model weight file and model text data. Among them, the large model weight file is an extremely important data file for the large model, which contains a file of the connection weights between each neuron or layer in the model; that is, the large model weight file stores the weight parameters learned by model training. These weights determine how the model generates outputs based on input data. Using the trained large model weight file, the large model can be directly used to process a large amount of data for reasoning or prediction. The weight file is the key to the model's ability to accurately predict. In an embodiment of the present application, the CPU core is the core of the entire multi-core large model calculation. The acquisition of the previous files or data requires the CPU to process. In addition, the planning of the weight data cache address of the subsequent model and the issuance of the model's calculation instructions also require the CPU to participate in the control, so that the entire system can support network models of different sizes and types.

[0052] It should be noted that in the embodiments of the present application, the central processing unit, also known as the central processing unit, can directly adopt a CPU core; the CPU has versatility and flexibility, can perform various tasks, is good at serial calculations, that is, executing tasks in a specified order, and has good logical control, that is, good computing versatility, which can meet various complex computing needs.

[0053] S20, the central processing unit caches the large model weight file in the high-bandwidth memory unit.

[0054] Specifically, after the central processing unit (CPU) obtains the large model weight file, it caches the model weight file in the high-bandwidth memory unit, that is, in the HBM. In the subsequent large model inference process, the large model weight data can be directly read from the HBM and used for calculation.

[0055] S30, the central processing unit converts the model text data into embedded vector data, and writes the embedded vector data into the core computing unit.

[0056] Specifically, the model text data originally obtained by the central processing unit CPU is text information and cannot be used directly for calculation. Therefore, the model text data needs to be converted into embedded vector data (Embedding data) before the embedded vector data can be written into the core computing unit for calculation.

[0057] S40, the core computing unit reads the large model weight file in the high-bandwidth memory unit, processes the embedded vector data according to the weight data in the large model weight file and the number of middle layers of the large model network structure, and generates attention parameters.

[0058] Specifically, the core computing unit is responsible for the main computing tasks of the entire large model. After receiving the embedded vector data, the core computing unit reads the weight data in the large model weight file from the HBM, and processes the received embedded vector data according to the weight data in the read large model weight file and the number of layers in the middle layer of the large model network structure to obtain the attention parameter. Among them, the large model network structure generally includes an input layer, an intermediate layer, and an output layer. The intermediate layer generally includes multiple layers. For example, taking the LlaMa2 large model as an example, the LlaMa2 large model is a large language model evolved from the transform network model. Figure 2 The LlaMa2 large model includes input layer, middle layer and output layer. The middle layer includes 32 layers. Each middle layer can be expanded into an attention mechanism (Attention) kernel feed forward network (Feed Forward) processing; it should be noted that the weight data may be different during the processing of each middle layer. Each middle layer has corresponding weight data. Therefore, the calculation of each middle layer needs to read the weight data of the corresponding layer from HBM to ensure the accuracy of the model calculation. The calculation results of all middle layers after calculation are completed also need to be returned to the central processing unit CPU for processing. In the calculation process of the large model, the generation of the next set of Embedding data can only be obtained based on the calculation results of the previous set.

[0059] Among them, the core computing unit can be implemented by GPU and / or FPGA. In the embodiment of the present application, multi-core refers to the central processing unit CPU combined with the core computing unit to realize the calculation of the large model; for different application scenarios, the core computing unit can use different computing cores, such as GPU, FPGA, etc., and one of the computing cores can be used, or two computing cores can be used at the same time. The specific selection can be based on the large model used in the actual application and the amount of data calculation. GPU and FPGA each have certain advantages. For different application scenarios and different stages, the computing core can use GPU. The parallelism of GPU is more advantageous than FPGA. For example, in the model pre-filled stage, the GPU core can be used to process more Embedding data in parallel; and in the generated stage, more FPGA computing cores can be configured for data calculation. The specific user can configure it according to the actual situation.

[0060] S50, the core computing unit processes the attention parameters according to the output layer structure of the large model network structure and outputs the calculation results of the large model.

[0061] Specifically, the attention parameters are obtained after the calculation of the attention mechanism of the middle layer and the feedforward network. The attention parameters are calculated by the output layer of the large model network structure to output the final large model calculation results. Taking the LlaMa2 large model as an example, after the output of the 32-layer middle layer, the output layer needs to be calculated by RMSNorm, matrix multiplication, Softmax and other operators to output the final large model calculation results.

[0062] It should be noted that after the CPU calculates the embedded vector data (Embedding data), the CPU writes the embedded vector data (Embedding data) into the core computing unit. The core computing unit will perform corresponding calculations based on the network structure of the specific large model used, such as several intermediate layers and output layers, and output the final large model calculation results. The weight data required for each layer can be directly read and used from the HBM. In addition, the calculation results of the intermediate layer of the large model include attention parameters, as well as some other parameters or calculation results, the main of which is the attention parameter. The output attention parameter represents the importance distribution of each element in the sequence when considering other elements. It is a feature vector that integrates global information and is a more important parameter in the large model.

[0063] Traditional multi-core computing architectures often need to move data from memory to the cache of their own computing cores before performing corresponding calculations or other processing. In the present application, high-bandwidth memory (HBM) is used to open the memory to the entire system, eliminating the need to move data from one memory to another. HBM (High Bandwidth Memory) is the data cache center of the entire solution. Each computing core can directly read and write to the storage. Compared with ordinary memory, the ultra-high bandwidth HBM can meet the speed requirements of high-computing power cores for data acquisition, so that the performance of the entire architecture can be maximized and the utilization rate of hardware computing power resources can be improved.

[0064] In one embodiment, the large model includes a vocabulary, referring to Figure 3 , the step of the central processing unit converting the model text data into embedded vector data can be specifically performed as follows:

[0065] First, the central processing unit, i.e., the central processing unit CPU, segments the model text data according to the vocabulary to generate a number of word units; then, the central processing unit searches for the positions of the word units in the vocabulary; and the central processing unit CPU generates embedding vector data according to the positions of the word units and the corresponding word units. Figure 3For example, the text information is "I have an apple". The CPU divides the text information into several individual words according to the vocabulary of the large model. After that, the CPU finds the position of each word in the vocabulary and combines the word with the corresponding position to generate embedded vector data. Figure 3 The calculation vector X can be understood as embedded vector data, which is vector data that can participate in the calculation.

[0066] Considering that the original model text data contains text information, the text information cannot be directly written into the calculation core for calculation. Therefore, the original model text data is processed to obtain vector data that can be calculated, and the vector data that can be calculated is written into the calculation core for calculation, and the results required by the user are output.

[0067] In one embodiment, the core computing unit reads the large model weight file in the high bandwidth memory unit, processes the embedded vector data according to the weight data in the large model weight file and the number of layers of the middle layer of the large model network structure, and the step of generating the attention parameter can be specifically performed as follows:

[0068] First, the core computing unit identifies the number of layers in the middle layer of the large model network structure, and reads the weight data of the current middle layer in the large model weight file from the high-bandwidth memory unit. Specifically, the core computing unit, such as FPGA, receives the Embedding data and related configuration instruction data processed by the CPU core, parses the received instruction information one by one, and executes them in sequence according to the parsed instruction information, thereby completing the calculation task of the entire model. When calculating the attention parameters, it is necessary to obtain the address information of the relevant weight data from the instruction, and then start the weight data reading task to read the corresponding weight data in the large model weight file from the HBM.

[0069] Afterwards, the core computing unit processes the embedded vector data based on the weight data of the current intermediate layer read, using the basic operators in the large model to generate the attention parameters of the current layer; for details, refer to Figure 4 , the core computing unit inputs the weight data and embedding vector data of the current middle layer into the matrix multiplication operator to calculate the attention parameters of the current layer; wherein the attention parameters include the query vector Q, the key vector K and the numerical vector V, the core computing unit reads the weights Wq, Wk and Wv corresponding to the query vector Q, the key vector K and the numerical vector V of the current middle layer from the HBM, inputs the embedding vector data and the weights Wq, Wk and Wv that can be used for calculation obtained after the CPU core processing into the matrix multiplication operator, and calculates the QKV data, which is also the required attention parameters. Finally, the core computing unit performs a loop calculation according to the number of layers of the middle layer of the identified large model network structure until the attention parameters of the last middle layer are calculated and output.

[0070] In the present application scheme, the calculation process is carried out in a core computing unit such as FPGA, and the required weight data is read through HBM. Based on the read and write bandwidth of the memory (HBM), the hardware computing power resources can be fully utilized, thereby improving the utilization rate of the hardware computing power resources and reducing resource waste.

[0071] In one embodiment, considering that the same data may need to be accessed multiple times during the inference process of a large model, after the core computing unit inputs the weight data and the embedding vector data of the current intermediate layer into the matrix multiplication operator and calculates the attention parameters of the current layer, the following steps may be performed:

[0072] The core computing unit writes the current layer key vector and the current layer numerical vector into the high-bandwidth memory unit, that is, the KV data obtained during the intermediate layer calculation process is also cached in the HBM to realize KV cache (KV Cache) data sharing.

[0073] In this application, considering that the same data may need to be accessed multiple times during the large model reasoning process, the KV data is also cached in the HBM. When the data needs to be accessed, it can be read directly from the HBM, which can avoid unnecessary repeated calculations as much as possible, thereby accelerating the model reasoning and calculation process. A reasonable KV Cache strategy can reduce video memory overhead and computing costs while maintaining the model's expressiveness.

[0074] It should be noted that the role of the core computing unit is to obtain the final large model calculation result based on the large model network structure. There may be differences between the network structures of different models. The attention parameter query vector Q, key vector K and numerical vector V, that is, the main parameter data for the middle layer calculation of the large model when QKV data, are mainly calculated by the matrix multiplication operator to obtain the QKV data, but the generation of QKV data may be calculated through other operators to obtain the final result. In the actual application process, specific analysis and calculation can be performed according to the network structure of the specific model used. For example, in the embodiment of the present application, the LlaMa2 large model is used as an example for explanation, referring to Figure 2 The calculation of the middle layer QKV data includes attention mechanism and feed forward network processing. The attention mechanism processing includes RMSNorm, matrix multiplication, RoPE, Softmax and other operator calculations; refer to Figure 5, the weight data of each layer of the large model needs to be planned by the CPU before the calculation starts, according to the size of the LlaMa2 large model (the size can be divided into 7B, 13B and 70B), and the address of each layer of weight data in HBM memory is planned, and then the weight data of each layer is written to the corresponding address of HBM as initialization data in turn; the embedding data of the input layer is sent to the computing core. After the computing core receives the start calculation signal, the computing core starts to calculate the large model. After the computing core ROPE processing, the KV data is written back to HBM for KV data sharing; the address space location and size of the KV Cache are pre-allocated by the CPU core mentioned above; QKT, Softmax, Linear and other attention mechanisms (Attention) processing, as well as the feedforward network (Feed Forward) processing after the attention mechanism, the calculation results can be cached in the internal RAM of the computing core (FPGA) for CPU reading; the calculation results can also be directly written back to the cache inside the CPU for CPU query and reading, and the calculation results can be processed differently according to the needs of the application.

[0075] Reference Figure 6 , which is the complete process of multi-core large model calculation. Taking FPGA as an example, the CPU obtains the large model weight data and model text data Tokenizer. The CPU caches the large model weight file in HBM to complete the initialization of the model weight data. The CPU processes the model text data Tokenizer to generate embedded vector data (Embedding data) that can be used for calculation. FPGA processes the input Embedding data according to the network structure of the large model and reads the corresponding weight data of each layer from HBM to generate QKV data. The KV data is cached in HBM to realize KV Cache data sharing. In addition, the calculation core calculates the QKV data according to the operator contained in the output layer structure in the large model network structure to obtain the final large model calculation result.

[0076] Reference Figure 7, is a schematic diagram of the system architecture corresponding to the multi-core large model method. The storage resources of the entire solution use HBM (High Bandwidth Memory), and with various computing cores with different characteristics, the memory (HBM) read and write bandwidth resources are fully utilized to maximize the computing resource utilization. In addition, in traditional network communications, data usually needs to be copied multiple times between the operating systems of the sender and the receiver through the protocol stack, while RDMA bypasses the operating system and directly transfers data from one memory to another, thereby avoiding these copy operations and improving data transmission efficiency. Therefore, RDMA is used in the solution to carry the weight data and calculation results required by the large model. Taking the LlaMa2 7B model as an example, the model weight data volume is relatively large, and RDMA has a natural advantage in the transportation of large data, reducing the delay of data transportation in the entire architecture, thereby improving the overall computing performance of the entire solution.

[0077] Figure 1 FIG. 1 is a flow chart of a multi-core large model calculation method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0078] Based on the above method, the present application also discloses a multi-core large model computing device. The device includes:

[0079] The central processing unit is used to obtain large model weight files and model text data;

[0080] The CPU is also used to cache large model weight files in the high-bandwidth memory unit;

[0081] The central processing unit is also used to convert the model text data into embedded vector data and write the embedded vector data into the core computing unit;

[0082] The core computing unit is used to read the large model weight file in the high-bandwidth memory unit, process the embedded vector data according to the weight data in the large model weight file and the number of layers in the middle layer of the large model network structure, and generate attention parameters;

[0083] The core computing unit is also used to process the attention parameters according to the output layer structure of the large model network structure and output the calculation results of the large model.

[0084] In one embodiment, the central processing unit segments the model text data according to the vocabulary to generate a number of word units; the central processing unit searches for positions of the number of word units in the vocabulary; and the central processing unit generates embedded vector data according to the number of word units and positions corresponding to the word units.

[0085] In one embodiment, the core computing unit identifies the number of intermediate layers of the large model network structure; the core computing unit reads the weight data of the current intermediate layer in the large model weight file from the high-bandwidth memory unit; the core computing unit processes the embedded vector data using the basic operators in the large model according to the read weight data of the current intermediate layer to generate the attention parameters of the current layer; the core computing unit performs a loop calculation according to the number of intermediate layers of the identified large model network structure until the attention parameters of the last intermediate layer are calculated and output.

[0086] In one embodiment, the core computing unit inputs the weight data and embedded vector data of the current intermediate layer into the matrix multiplication operator to calculate the attention parameters of the current layer.

[0087] In one embodiment, the core computing unit writes the current layer key vector and the current layer value vector to the high bandwidth memory unit.

[0088] In one embodiment, the central processing unit is a CPU.

[0089] In one embodiment, the core computing unit is a GPU and / or an FPGA.

[0090] The multi-core large model computing device provided in the embodiment of the present application can be applied to the multi-core large model computing method provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principles and technical effects are similar and will not be repeated here.

[0091] It should be noted that: when the multi-core large model computing device provided in the embodiment of the present application performs multi-core large model computing, only the division of the above-mentioned functional modules / functional units is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the multi-core large model computing device is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the multi-core large model computing method provided in the above method embodiment and the implementation method of the multi-core large model computing device provided in this embodiment belong to the same concept. The specific implementation process of the multi-core large model computing device provided in this embodiment is detailed in the above method embodiment, which will not be repeated here.

[0092] The embodiment of the present application also discloses a computer device.

[0093] Specifically, if Figure 8 As shown, the computer device can be a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or otherwise. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0094] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0095] The embodiment of the present application also discloses a computer-readable storage medium.

[0096] Specifically, a computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art will understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the process of the implementation of the above-mentioned methods. Among them, the storage medium may be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memory.

[0097] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A multi-core large model calculation method, Features: The method is applied to a multi-core large model computing system, the multi-core large model computing system includes a central processing unit, a high bandwidth memory unit and a core computing unit, and the method includes: The central processing unit obtains a large model weight file and model text data; The central processing unit caches the large model weight file in the high bandwidth memory unit; The central processing unit converts the model text data into embedded vector data, and writes the embedded vector data into the core computing unit; The core computing unit reads the large model weight file in the high bandwidth memory unit, processes the embedded vector data according to the weight data in the large model weight file and the number of layers of the middle layer of the large model network structure, and generates attention parameters; The core computing unit processes the attention parameters according to the output layer structure of the large model network structure and outputs the large model calculation results.

2. The method according to claim 1, Features: The large model includes a vocabulary, and the central processing unit converts the model text data into embedded vector data including: The central processing unit segments the model text data according to the vocabulary to generate a plurality of word units; The central processing unit searches for positions of a number of the word elements in the word library; The central processing unit generates the embedded vector data according to a plurality of the word units and positions corresponding to the word units.

3. The method according to claim 1, Features: The core computing unit reads the large model weight file in the high bandwidth memory unit, processes the embedded vector data according to the weight data in the large model weight file and the number of layers of the middle layer of the large model network structure, and generates the attention parameter, including: The core computing unit identifies the number of intermediate layers of the large model network structure; The core computing unit reads the weight data of the current middle layer in the large model weight file from the high bandwidth memory unit; The core computing unit processes the embedded vector data using the basic operators in the large model according to the weight data of the current intermediate layer read to generate the attention parameters of the current layer; The core computing unit performs cyclic calculations according to the number of intermediate layers of the identified large model network structure until the attention parameters of the last intermediate layer are calculated and output.

4. The method according to claim 3, Features: The basic operator includes a matrix multiplication operator. The core computing unit processes the embedded vector data using the basic operator in the large model according to the weight data of the current intermediate layer read, and generates the attention parameters of the current layer including: The core computing unit inputs the weight data of the current intermediate layer and the embedded vector data into the matrix multiplication operator to calculate the attention parameters of the current layer.

5. The method according to claim 1, Features: The attention parameters of the current layer include a current layer key vector, a current layer numerical vector and a current layer query vector. After the core computing unit inputs the weight data of the current intermediate layer and the embedding vector data into the matrix multiplication operator to calculate the attention parameters of the current layer, it also includes: The core computing unit writes the current layer key vector and the current layer value vector into the high bandwidth memory unit.

6. The method according to claim 1, Features: The central processing unit is a CPU.

7. The method according to claim 1, Features: The core computing unit is a GPU and / or FPGA.

8. A multi-core large model computing device, Features: The device is applied to a multi-core large model computing system, the multi-core large model computing system includes a central processing unit, a high bandwidth memory unit and a core computing unit, and the device includes: The central processing unit is used to obtain a large model weight file and model text data; The central processing unit is also used to cache the large model weight file in the high bandwidth memory unit; The central processing unit is also used to convert the model text data into embedded vector data, and write the embedded vector data into the core computing unit; The core computing unit is used to read the large model weight file in the high-bandwidth memory unit, process the embedded vector data according to the weight data in the large model weight file and the number of layers of the middle layer of the large model network structure, and generate attention parameters; The core computing unit is also used to process the attention parameters according to the output layer structure of the large model network structure and output the large model calculation results.

9. A computer device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Method and device for generating reasoning result, storage medium and electronic equipment

    CN120930810A

  • Inference apparatus, inference system, and inference method

    WO2026174755A1