Method, device and equipment for reasoning engineering of large language model, medium and program product

Abstract

The invention provides a method, device and equipment for reasoning engineering of a large language model, a medium and a program product, and relates to the technical field of artificial intelligence. A target matrix operation operator for executing a first target task is determined in a plurality of preset matrix operation operators, the first target task is a matrix operation task in a reasoning project of a large language model, and a matrix participating in the first target task is a matrix to be operated by the matrix operation task; and executing the first target task by using the target matrix operator to obtain a first task result which is used for forming output information of the reasoning project. According to the method and the device, the proper matrix operator can be selected according to the actual operation requirement of the reasoning project, so that the reasoning efficiency of related hardware on the large language model is improved, and the reasoning resources of the hardware are effectively utilized.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, medium and program product for reasoning engineering of a large language model. Background Art

[0002] In the field of artificial intelligence (AI), with the development of large language models (LLM), the functions provided by AI have been further improved in scenarios such as human-computer dialogue, machine translation, and assisted programming.

[0003] At present, the amount of data involved in the reasoning of large language models is huge, which places high demands on the use of reasoning resources (such as memory resources and computing resources). For some hardware manufacturers, although the theoretical performance of the reasoning resources of the hardware (such as CPU, GPU, etc.) they provide can meet the reasoning requirements of LLM, due to the lack of corresponding optimization mechanisms, these hardware have low reasoning efficiency for large language models, resulting in the actual performance of the reasoning resources being significantly lower than the theoretical performance. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method, apparatus, device, medium and program product for reasoning engineering of large language models, which are used to solve the technical problem of low reasoning efficiency of hardware for large language models in related technologies.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for reasoning engineering of a large language model, the method comprising:

[0006] Determining a target matrix operation operator to perform the first target task from among a plurality of pre-set matrix operation operators according to the dimension of the matrix involved in the first target task, wherein the first target task is a matrix operation task in the reasoning project of the large language model, and the matrix involved in the first target task is a matrix to be operated by the matrix operation task;

[0007] The first target task is executed using the target matrix operation operator to obtain a first task result, and the first task result is used to form output information of the reasoning project.

[0008] In a second aspect, an embodiment of the present disclosure provides a device for reasoning engineering of a large language model, the device comprising:

[0009] An operator determination module, used to determine a target matrix operation operator to perform the first target task from among a plurality of pre-set matrix operation operators according to the dimension of the matrix participating in the first target task, wherein the first target task is a matrix operation task in the reasoning project of the large language model, and the matrix participating in the first target task is a matrix to be operated by the matrix operation task;

[0010] An execution module is used to use the target matrix operation operator to execute the first target task to obtain a first task result, and the first task result is used to form output information of the reasoning project.

[0011] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method described in the embodiment of the present disclosure when executed by the processor.

[0012] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the embodiment of the present disclosure is implemented.

[0013] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer instructions, which, when executed by a processor, implement the method described in the embodiment of the present disclosure.

[0014] In the present disclosure, according to the dimension of the matrix to be operated by the matrix operation task, the target matrix operation operator to perform the matrix operation task is determined from multiple pre-set matrix operation operators, and the appropriate matrix operation operator can be selected according to the characteristics of the operation object matrix of different matrix operation tasks in the large language model inference project, thereby improving the reasoning efficiency of the underlying hardware related to the matrix operation operator for the large language model, thereby effectively utilizing the hardware reasoning resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of a method for reasoning engineering of a large language model provided by an embodiment of the present disclosure;

[0016] Figure 2 is a schematic diagram provided by an embodiment of the present disclosure for displaying the efficiency of optimized hardware processing;

[0017] Figure 3 It is a structural schematic diagram of a device for reasoning engineering of a large language model provided by an embodiment of the present disclosure;

[0018] Figure 4 It is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0020] For some hardware manufacturers, although the theoretical performance of the inference resources of the hardware (such as CPU, GPU, etc.) they provide can meet the inference requirements of LLM, due to the lack of corresponding optimization mechanisms, these hardware have low inference efficiency for large language models, resulting in the actual performance of the inference resources being significantly lower than the theoretical performance.

[0021] For example, in related technologies, for the reasoning engineering of large language models, some hardware manufacturers will directly use Python language machine learning platforms such as HuggingFace, and do not optimize the hardware's reasoning resources. In the reasoning engineering, the hardware may spend a lot of time performing operations such as data movement and memory arrangement, which wastes the hardware's reasoning resources to a certain extent. Another part of the hardware manufacturers may transplant the reasoning engine developed for a specific hardware manufacturer to optimize the reasoning resources, but due to the different hardware architectures of different manufacturers, the simple reuse of other manufacturers' reasoning engines in actual applications may cause many problems due to incompatibility, and may even cause negative optimization.

[0022] In view of the above situation, the embodiments of the present disclosure provide a method for reasoning engineering of a large language model, so that even if hardware with different hardware architectures is used to implement reasoning engineering, it is possible to select appropriate matrix operation operators according to the actual computing requirements of the reasoning engineering, so as to improve the reasoning efficiency of the underlying hardware related to the matrix operation operators for the large language model, thereby effectively utilizing the reasoning resources of the hardware.

[0023] like Figure 1 As shown, the method for reasoning engineering of a large language model according to an embodiment of the present disclosure includes the following steps: step 101, determining a target matrix operation operator to perform the first target task from a plurality of pre-set matrix operation operators according to the dimensions of the matrices involved in the first target task; and step 102, performing the first target task using the target matrix operation operator to obtain a first task result.

[0024] First, in step 101, according to the dimension of the matrix participating in the first target task, a target matrix operation operator to perform the first target task is determined from a plurality of pre-set matrix operation operators. The first target task is a matrix operation task in the reasoning project of the large language model, and the matrix participating in the first target task is a matrix to be operated by the matrix operation task.

[0025] In the present disclosure, the reasoning engineering of the large language model can be used to implement any AI task supported by the large language model, such as human-computer dialogue tasks, translation tasks, etc.

[0026] In the present disclosure, an operator can be understood as a specific operation logic for implementing an operation task. By scheduling and combining different operators, various operation tasks in the reasoning engineering of a large language model can be implemented. Since the reasoning engineering of a large language model (such as the attention mechanism and feedforward neural network for implementing the reasoning engineering) involves a series of matrix operations, it can be considered that the reasoning engineering of a large language model includes a series of matrix operation tasks. In the following, unless otherwise specified, an operator generally represents a matrix operation operator. For the same matrix operation task (such as matrix multiplication), different operation logics, i.e., operators, can obtain the same operation result, but different operators are used, the operation process is also different, and the utilization efficiency of the hardware reasoning resources will also be different. Therefore, by selecting appropriate operators for the matrix operation tasks in the reasoning engineering of a large language model, the reasoning efficiency can be improved and the reasoning resources of the hardware can be optimized.

[0027] In some embodiments, the first target task may be, for example, a matrix multiplication task in the reasoning project of a large language model, and the matrices involved in the first target task are two matrices to be multiplied. For the matrix multiplication task, the matrix operation operator may be, for example, a general matrix multiplication (GEMM) operator, a flat matrix multiplication (Flat GEMM) operator, or any other operator capable of implementing a matrix multiplication operation.

[0028] In some embodiments, each matrix operation operator among the pre-set plurality of matrix operation operators is suitable for operating matrices of a fixed dimension interval, and different matrix operation operators are suitable for operating matrices of different dimension intervals.

[0029] Taking a two-dimensional matrix as an example, a dimension interval may be {[1, 500], [1, 500]}, and the matrix size included in the dimension interval is X×Y, where the value range of X is 1 to 500, and the value range of Y is also 1 to 500. If the dimension of the matrix is ​​300×300, then the matrix is ​​considered to be within the dimension interval. It should be understood that the dimension interval may also include dimension intervals of one-dimensional matrices, three-dimensional matrices, four-dimensional matrices, etc.

[0030] The dimension interval that each matrix operation operator is suitable for operation can be understood as that among the multiple matrix operation operators, the matrix operation operator has the highest operation efficiency and the shortest operation time for the matrix located in the dimension interval that it is suitable for operation. Different matrix operation operators are suitable for operating matrices in different dimension intervals, which can be understood as the value range of a certain dimension in the dimension interval of the matrix that different matrix operation operators are suitable for operation does not overlap or the value range of all dimensions does not overlap. For example, still taking the example of the two-dimensional matrix of size X×Y above, the four different dimension intervals of X and Y suitable for operation by four different matrix operation operators can be, for example, {[1,500], [1,500]}, {[500,1000], [1,500]}, {[1,500], [500,1000]} and {[500,1000], [500,1000]}.

[0031] In some embodiments, for example, the target matrix operation operator to be executed for the first target task can be determined from among the pre-set multiple matrix operation operators according to the dimension of the matrix participating in the first target task in the following manner: determine the dimension interval where the dimension of the matrix participating in the first target task is located in the pre-set multiple dimension intervals, i.e., the target dimension interval, and determine the matrix operation operator suitable for operating the matrix of the target dimension interval among the pre-set multiple matrix operation operators as the target matrix operation operator to be executed for the first target task. For example, assuming that the pre-set multiple dimension intervals are the four dimension intervals in the above example, and the dimension of the matrix participating in the first target task, i.e., the matrix operation of the first target task, is to be operated is 300×600, then the dimension interval where the dimension of the matrix participating in the first target task is located can be determined to be {[1, 500], [500, 1000]}. Determine the matrix operation operator suitable for operating the matrix of the dimension interval as the target matrix operation operator to be executed for the first target task. When the matrix operation task is to operate multiple matrices (for example, matrix multiplication is to operate two matrices), different multiple dimension intervals can be set for each matrix, or the same multiple dimension intervals can be set for multiple matrices.

[0032] In some embodiments, a matrix operation operator suitable for operating a matrix of each dimension interval in a plurality of preset dimension intervals may be determined, for example, in the following manner.

[0033] Before determining the target matrix operation operator to perform the first target task from among the preset multiple matrix operation operators according to the dimensions of the matrices participating in the first target task, first, for the preset multiple dimension intervals, multiple test matrices whose dimensions are within one of the multiple dimension intervals are determined. For example, for the above four preset dimension intervals {[1,500], [1,500]}, {[500,1000], [1,500]}, {[1,500], [500,1000]} and {[500,1000], [500,1000]}, multiple test matrices whose dimensions are within each dimension interval are determined, for example, multiple test matrices whose dimensions are 300×300, 600×300, 300×600, and 600×600, respectively. Of course, the dimensions of the test matrices in each dimension interval may also be different, as long as the dimensions are within the preset multiple intervals.

[0034] Next, under the same operation conditions (e.g., the same combination of heterogeneous hardware, the same large language model operation platform / cluster, etc.), each of the pre-set multiple matrix operation operators is operated on multiple test matrices, and the operation time of each matrix operation operator for each test matrix is ​​recorded. The matrix operation operator with the shortest operation time for the test matrix of each dimensional interval among all multiple matrix operation operators is determined as the matrix operation operator suitable for operating the matrix of the dimensional interval. For example, there are four matrix operation operators A, B, C, and D, each of which represents an operation logic (which can also be understood as an operation process). For the four dimensional intervals in the previous example, assuming that the operation time of the A operator for the test matrix of the dimensional interval {[1, 500], [500, 1000]} (e.g., a test matrix with a dimension of 300×600) is the shortest among the four operators A, B, C, and D, then it is determined that the A operator is suitable for operating the matrix of the dimensional interval {[1, 500], [500, 1000]}. In this case, when it is determined that the dimension of the matrix to be operated by the matrix operation task is within the interval {[1, 500], [500, 1000]}, the A operator suitable for operating the matrix in the interval can be selected to perform the matrix operation task.

[0035] In another example, if the pre-set multiple matrix operation operators are set to be matrix operation operator A, matrix operation operator B and matrix operation operator C; the multiple test matrices of different dimensional intervals are respectively: test matrix L1 of dimensional interval I1, test matrix L2 of dimensional interval I2, test matrix L3 of dimensional interval I3, and test matrix L4 of dimensional interval I4. By respectively using the above three matrix operation operators to operate on the above four test matrices, four sets of operation time consumption information are obtained: L1: a1 (30ms), b2 (33ms), c3 (35ms); L2: a2 (25ms), b2 (30ms), c2 (35ms); L3: a3 (40ms), b3 (30ms), c3 (35ms); L4: a4 (50ms), b4 (40ms), c4 (35ms), among which a1, a2, a3, a4 respectively indicate the time consumption of the matrix operation operator A operating the test matrices L1~L4, b1, b2, b3, b4 respectively indicate the time consumption of the matrix operation operator B operating the test matrices L1~L4, and c1, c2, c3, c4 respectively indicate the time consumption of the matrix operation operator C operating the test matrices L1~L4.

[0036] According to the above four sets of operation time information, matrix operator A consumes the least time when operating test matrices L1 and L2, so the dimension interval suitable for matrix operator A is I1+I2. Similarly, the dimension interval suitable for matrix operator B is I3, and the dimension interval suitable for matrix operator C is I4.

[0037] After determining the target matrix operation operator to perform the first target task, in step 102, the first target task is performed using the target matrix operation operator to obtain the first task result. The first task result is used to form the output information of the reasoning project of the large language model. For example, when the first target task is the matrix multiplication operation of the attention mechanism used in the reasoning project of the large language model, according to the dimension of the matrix of the operation object of the matrix multiplication, the matrix multiplication operator suitable for operating the matrix in the dimension interval where the dimension is located is determined, and the matrix multiplication operator is used to perform matrix multiplication. The result obtained can be used for subsequent operations in the reasoning project, and finally the output information of the reasoning project is obtained.

[0038] For example, when the reasoning engineering of the large language model is applied to the human-computer dialogue scenario, the input for the reasoning engineering can be a question initiated by the user to the large language model, such as "What is the weather in place A today?", and the output information of the reasoning engineering can be the answer given by the large language model to the user's question, such as "It is sunny in place A today, the temperature is 25-30 degrees, and the air quality is relatively good". In the reasoning engineering, the semantic features of the question initiated by the user to the large language model are represented by a matrix, and a series of operations are performed on the matrix, and the answer to the question is output according to the operation results. Or, for example, when the reasoning engineering of the large language model is applied to the translation scenario, the input for the reasoning engineering can be the text in the first language input by the user, and the output information of the reasoning engineering can be the translation text in the second language corresponding to the text in the first language returned by the large language model. In the reasoning engineering, the semantic features of the text in the first language input by the user are represented by a matrix, and a series of operations are performed on the matrix, and the translation text in the second language is output according to the operation results.

[0039] In the present disclosure, according to the dimension of the matrix to be operated by the matrix operation task, the target matrix operation operator to perform the matrix operation task is determined from multiple pre-set matrix operation operators, and the appropriate matrix operation operator can be selected according to the characteristics of the operation object matrix of different matrix operation tasks in the large language model inference project, thereby improving the reasoning efficiency of the underlying hardware related to the matrix operation operator for the large language model, thereby effectively utilizing the hardware reasoning resources.

[0040] In some embodiments, for example, if there is a lack of sufficient reasoning resources to directly operate on the matrix participating in the first target task, or the direct operation is inefficient, or other situations are not suitable for direct operation, the matrix participating in the first target task can also be split to obtain multiple sub-matrices, and according to the dimension interval of each sub-matrix in the multiple sub-matrices, the target matrix operation operator suitable for operating each sub-matrix (the operation time for each sub-matrix is ​​the shortest) is determined from multiple matrix operation operators by the aforementioned method. The target matrix operation operators of the multiple sub-matrices are used to perform parallel operations on the multiple sub-matrices to obtain multiple sub-task results, and the task result of the first target task is obtained based on the multiple sub-task results.

[0041] For example, a 1200×1200 matrix can be split into 8 sub-matrices of 300×600, and for each sub-matrix of dimension 300×600, an A operator suitable for operating the sub-matrix is ​​determined, and 8 A operators are used to operate the 8 sub-matrices in parallel, and the operation result of the original 1200×1200 matrix is ​​obtained according to the 8 operation results. Or, for example, the operation object matrix that originally needs to be operated by the GEMM operator is split into multiple short fat matrices, and the Flat GEMM operator is used to perform parallel operations on the multiple short fat matrices.

[0042] In this way, by splitting the operation object matrix into sub-matrices and processing each sub-matrix in parallel, the parallel computing capability of the hardware itself can be fully utilized, the reasoning efficiency can be further improved, and thus the reasoning performance of the hardware can be further improved.

[0043] In some embodiments, when the first target task is a matrix operation of an attention mechanism, the activation function of the attention mechanism can also be iteratively calculated using fixed parameters. Only in the last iterative calculation of the activation function are parameters synchronized among all computing nodes to obtain the calculation result of the activation function, which is used together with the first task result to form the output information of the reasoning project.

[0044] In the related art, when performing iterative calculation of the activation function (such as softmax) of the attention mechanism, except for the first iteration, each subsequent iteration needs to synchronize the result of the previous iteration among all computing nodes. In this way, multiple synchronization operations are required, which occupies more reasoning resources and takes a long time to process. The disclosed embodiment optimizes the iterative calculation logic of the activation function of the attention mechanism, and uses fixed parameters (such as a fixed max value) to participate in each iterative calculation, so that only in the last iterative calculation of the activation function, the parameters are synchronized among all computing nodes to obtain the calculation result of the activation function, thereby reducing the consumption of reasoning resources and shortening the processing time.

[0045] In some embodiments, when the first target task is a matrix operation task including matrix multiplication in the reasoning project, and another matrix operation task in the reasoning project, i.e., the second target task, is an element-by-element matrix operation task, the target matrix operation operator of the first target task can be fused with the element-by-element matrix operation operator to be executed for the second target task to obtain a fused operation operator, and the fused operation operator is used to simultaneously execute the first target task and the second target task, and the obtained task result is used to form the output information of the reasoning project. The element-by-element matrix operation task can be, for example, an addition operation or a subtraction operation. The fusion of operators can be understood as the fusion of the operation process. For example, the operation of the first target task can be performed while the partial operation result obtained is performed for the second target task. Although the task result finally obtained is equivalent to the task result obtained by first executing the first target task and then executing the second target task, since the operation process is performed simultaneously, the reasoning resources can be more reasonably utilized, the intermediate data reading and writing overhead and the waiting time when executing the operation task are saved, and the reasoning efficiency is improved.

[0046] For example, when performing a Q×W+E operation on matrices Q, W, and E, in the related art, the operation task will be split into two operation tasks, first using a matrix multiplication operator to perform the matrix multiplication operation task of Q×W, and then using an element-by-element addition operator to perform the addition operation task of the task result of Q×W and E. According to the above method disclosed in the present invention, the matrix multiplication operator and the element-by-element addition operator can be fused, and the fused operation operator can be used to simultaneously perform the operation of the two operation tasks Q×W+E.

[0047] If a control test is conducted on hardware (such as graphics cards) from the same manufacturer, the control group is set to directly use HuggingFace for inference, and the experimental group is optimized using the disclosed method for inference engineering of large language models. When the large language models Llama2-7B and OPT-6.7B are used for testing respectively, the processing efficiency of input data with batch sizes of 1, 2, 4, and 8 and lengths of 128, 1k, 2k, and 4k tokens under the same hardware specifications (computing conditions) is as follows: Figure 2 As shown. Figure 2 In the figure, the light-colored column indicates the control group, and the dark-colored column indicates the experimental group. The treatment efficiency of the control group is used as a standard unit. The relative expression of the treatment efficiency of the experimental group is shown in Figure 2 Left scale. Figure 2 The diamond points in the middle indicate the actual processing efficiency of the hardware after the optimization by the method disclosed in the present invention under the corresponding hardware specifications. Figure 2 Right scale.

[0048] See also Figure 3 , Figure 3 The present disclosure provides an apparatus 300 for inference engineering of a large language model. Figure 3 As shown, the device 300 for reasoning engineering of a large language model includes:

[0049] An operator determination module 301 is used to determine a target matrix operation operator to perform a first target task from among a plurality of pre-set matrix operation operators according to the dimension of a matrix participating in a first target task, wherein the first target task is a matrix operation task in an inference project of a large language model, and the matrix participating in the first target task is a matrix to be operated by the matrix operation task;

[0050] The execution module 302 is used to use the target matrix operation operator to execute the first target task to obtain a first task result, and the first task result is used to form output information of the reasoning project.

[0051] In one embodiment, each matrix operation operator of the plurality of matrix operation operators is suitable for operating matrices of a fixed dimension interval, and different matrix operation operators are suitable for operating matrices of different dimension intervals;

[0052] The operator determination module 301 is specifically used for:

[0053] Determine a target dimension interval among a plurality of preset dimension intervals, wherein the target dimension interval is a dimension interval in which the dimension of the matrix participating in the first target task is located;

[0054] A matrix operation operator suitable for operating the matrix of the target dimension interval among the multiple matrix operation operators is determined as the target matrix operation operator.

[0055] In one embodiment, the apparatus 300 for reasoning engineering of a large language model further includes:

[0056] An operator testing module, used for enabling each of the plurality of matrix operation operators to operate on a plurality of test matrices under the same operation conditions, and recording the operation time consumed by each matrix operation operator for each test matrix, wherein the dimension of each of the plurality of test matrices is within one of the plurality of dimensional intervals;

[0057] The interval determination module is used to determine the matrix operation operator with the shortest operation time of the test matrix for each dimensional interval among all the multiple matrix operation operators as the matrix operation operator suitable for operating the matrix of the dimensional interval.

[0058] In one embodiment, the operator determination module 301 is specifically used for:

[0059] Splitting the matrix involved in the first target task to obtain multiple sub-matrices;

[0060] Determining, according to the dimension interval of each submatrix in the multiple submatrices, a target matrix operation operator suitable for operating each submatrix from among the multiple matrix operation operators;

[0061] The execution module 302 is specifically used for:

[0062] Using the target matrix operation operators of the multiple sub-matrices to perform parallel operations on the multiple sub-matrices to obtain multiple sub-task results;

[0063] The task result is obtained according to the multiple subtask results.

[0064] In one embodiment, the apparatus 300 for reasoning engineering of a large language model further includes:

[0065] An activation function calculation module is used to perform iterative calculation of the activation function of the attention mechanism using fixed parameters when the first target task is the matrix operation of the attention mechanism; parameters are synchronized among all computing nodes only in the last iterative calculation of the activation function to obtain the calculation result of the activation function, and the calculation result is used together with the first task result to form the output information of the reasoning project.

[0066] In one embodiment, the apparatus 300 for reasoning engineering of a large language model further includes:

[0067] An operator fusion module, used to fuse the target matrix operation operator of the first target task with the element-by-element matrix operation operator to perform the second target task to obtain a fused operation operator;

[0068] A fusion operator operation module is used to use the fusion operation operator to simultaneously execute the first target task and the second target task, and the obtained task results are used to form the output information of the reasoning project,

[0069] Among them, the first target task is a matrix operation task including matrix multiplication in the reasoning project, and the second target task is an element-by-element matrix operation task in the reasoning project.

[0070] The apparatus 300 for reasoning engineering of a large language model provided in the embodiment of the present disclosure can implement each process in the above-mentioned method embodiment for reasoning engineering of a large language model, and will not be described again here to avoid repetition.

[0071] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0072] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0073] like Figure 4As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 to a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, ROM 402 and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0074] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0075] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as a method for reasoning engineering of a large language model. For example, in some embodiments, the method for reasoning engineering of a large language model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the method for reasoning engineering of a large language model described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the method for reasoning engineering of a large language model in any other appropriate manner (eg, by means of firmware).

[0076] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0078] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0079] As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0080] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0081] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0082] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0083] The present disclosure also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0084] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0085] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for reasoning engineering of a large language model, characterized in that: The method comprises: Determining a target matrix operation operator to perform the first target task from among a plurality of pre-set matrix operation operators according to the dimension of the matrix involved in the first target task, wherein the first target task is a matrix operation task in the reasoning project of the large language model, and the matrix involved in the first target task is a matrix to be operated by the matrix operation task; The first target task is executed using the target matrix operation operator to obtain a first task result, and the first task result is used to form output information of the reasoning project.

2. The method according to claim 1, characterized in that: Each matrix operation operator of the plurality of matrix operation operators is suitable for operating matrices of a fixed dimension interval, and different matrix operation operators are suitable for operating matrices of different dimension intervals; The step of determining a target matrix operation operator to perform the first target task from among a plurality of pre-set matrix operation operators according to the dimension of the matrix participating in the first target task comprises: Determine a target dimension interval among a plurality of preset dimension intervals, wherein the target dimension interval is a dimension interval in which the dimension of the matrix participating in the first target task is located; A matrix operation operator suitable for operating the matrix of the target dimension interval among the multiple matrix operation operators is determined as the target matrix operation operator.

3. The method according to claim 2, characterized in that Before determining the target matrix operation operator to perform the first target task from among a plurality of preset matrix operation operators according to the dimension of the matrix participating in the first target task, the method further includes: Under the same operation conditions, each matrix operation operator of the multiple matrix operation operators operates on multiple test matrices, and records the operation time consumed by each matrix operation operator for each test matrix, wherein the dimension of each test matrix of the multiple test matrices is within one dimension interval of the multiple dimension intervals; The matrix operation operator whose operation time for the test matrix of each dimensional interval is the shortest among all the plurality of matrix operation operators is determined as the matrix operation operator suitable for operating the matrix of the dimensional interval.

4. The method according to claim 2, characterized in that: The method further comprises: Splitting the matrix involved in the first target task to obtain multiple sub-matrices; Determining, according to the dimension interval of each submatrix in the multiple submatrices, a target matrix operation operator suitable for operating each submatrix from among the multiple matrix operation operators; Using the target matrix operation operators of the multiple sub-matrices to perform parallel operations on the multiple sub-matrices to obtain multiple sub-task results; The task result is obtained according to the multiple subtask results.

5. The method according to claim 1, characterized in that The method further comprises: In the case where the first target task is a matrix operation of an attention mechanism, an activation function of the attention mechanism is iteratively calculated using fixed parameters; Only in the last iterative calculation of the activation function are parameters synchronized among all computing nodes to obtain the calculation result of the activation function, and the calculation result is used together with the first task result to form the output information of the reasoning project.

6. The method according to claim 1, characterized in that The method further comprises: Fusing the target matrix operation operator of the first target task with the element-by-element matrix operation operator to perform the second target task to obtain a fused operation operator; The first target task and the second target task are simultaneously executed by using a fusion operation operator, and the obtained task results are used to form the output information of the reasoning project, Among them, the first target task is a matrix operation task including matrix multiplication in the reasoning project, and the second target task is an element-by-element matrix operation task in the reasoning project.

7. A device for reasoning engineering of a large language model, characterized in that: The device comprises: An operator determination module, used to determine a target matrix operation operator to perform the first target task from among a plurality of pre-set matrix operation operators according to the dimension of the matrix participating in the first target task, wherein the first target task is a matrix operation task in the reasoning project of the large language model, and the matrix participating in the first target task is a matrix to be operated by the matrix operation task; An execution module is used to use the target matrix operation operator to execute the first target task to obtain a first task result, and the first task result is used to form output information of the reasoning project.

8. An electronic device, characterized in that: include processor; Memory; and A computer program is stored in the memory and can be run on the processor, and when the computer program is executed by the processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which implement the method according to any one of claims 1 to 6 when executed by a processor.

Images