Tensor splitting result verification method and device, computer equipment and storage medium

Abstract

The invention relates to a tensor splitting result verification method and device, computer equipment and a storage medium, and belongs to the technical field of computers, and the method comprises the steps: carrying out the quantization processing of a target weight of a target level in a deep learning model based on a preset quantization digit, and obtaining a quantization processing result; performing compression processing on the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format; in response to the detected tensor splitting result verification request, analyzing a file in a preset format to obtain an analysis result; and based on the analysis result, determining a first reasoning result of the deep learning model, and based on the first reasoning result and a second reasoning result, verifying the tensor splitting result, the second reasoning result being a reasoning result determined based on the deep learning model which is not subjected to weight tensor splitting processing. According to the method, the development efficiency of int4 quantitative model tensor parallel reasoning can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a tensor splitting result verification method, apparatus, computer equipment and storage medium. Background Art

[0002] Tensor parallelism is to split a large tensor into multiple small tensors and distribute them to multiple nodes. It can effectively solve the problem of single-node memory limitation and can handle very large models and data sets. In addition, multiple nodes can perform calculations at the same time, which can significantly improve reasoning performance. When developing and implementing tensor reasoning, weight tensor splitting is a very important step. For the underlying implementation in development, it is essential to verify the correctness of the weight tensor splitting results through demo (demonstration program). However, many underlying layers do not support the data type int4 (four-byte integer data type). Therefore, how to verify the correctness of parallel reasoning of int4 quantized weight tensors is an urgent problem to be solved. Summary of the invention

[0003] Based on this, it is necessary to provide a tensor splitting result verification method, device, computer equipment and storage medium that can verify the correctness of parallel reasoning of int4 quantized weight tensors in response to the above technical problems.

[0004] In a first aspect, a tensor splitting result verification method is provided, the method comprising:

[0005] Based on the preset quantization bit number, the target weight of the target level in the deep learning model is quantized to obtain the quantization result;

[0006] Compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format;

[0007] In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result;

[0008] Based on the analysis result, a first inference result of the deep learning model is determined, and based on the first inference result and the second inference result, the tensor splitting result is verified, wherein the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0009] Optionally, the target weight includes at least a first target weight generated based on an attention mechanism and a second target weight generated based on a multi-layer perceptron. Based on a preset number of quantization bits, the target weight of the target level in the deep learning model is quantized, and the quantization processing result includes:

[0010] Based on the weighted quantization mechanism, determine the preset number of quantization bits;

[0011] According to the preset number of quantization bits, the first target weight and the second target weight are quantized respectively to obtain a first quantization result and a second quantization result.

[0012] Optionally, compressing the quantization result includes:

[0013] Get the preset integer data type;

[0014] Based on the preset integer data type, compressing the first quantization processing result and the second quantization processing result respectively to obtain a first compression processing result and a second compression processing result;

[0015] The compression processing result is split into tensors to generate files in a preset format, including:

[0016] Based on the first target direction, performing tensor splitting processing on the first compression processing result to obtain a first tensor splitting processing result;

[0017] Based on the second target direction, performing tensor splitting processing on the first compression processing result to obtain a second tensor splitting processing result;

[0018] Based on the first target direction, performing tensor splitting processing on the second compression processing result to obtain a third tensor splitting processing result;

[0019] Based on the second target direction, performing tensor splitting processing on the second compression processing result to obtain a fourth tensor splitting processing result;

[0020] According to a preset format, saving the first tensor splitting processing result, the second tensor splitting processing result, the third tensor splitting processing result, and the fourth tensor splitting processing result to generate a file in the preset format;

[0021] The first target direction and the second target direction correspond to different tensor splitting directions.

[0022] Optionally, in response to detecting a tensor splitting result verification request, the file in the preset format is parsed, and the parsing result obtained includes:

[0023] In response to detecting a tensor splitting result verification request, sorting the weights in the file in the preset format and performing a decompression operation to obtain a first parsing result;

[0024] Perform a dequantization operation on the first parsing result to obtain a second parsing result.

[0025] Optionally, based on the parsing result, determining a first inference result of the deep learning model includes:

[0026] Based on the analysis result, a target deep learning model on which weight tensor splitting processing has been performed is generated;

[0027] Inputting the target verification data into the target deep learning model to obtain a first output result;

[0028] The first output result is defined as the first inference result.

[0029] Optionally, the method for determining the second reasoning result includes:

[0030] Inputting the target verification data into an initial deep learning model to obtain a second output result, wherein the initial deep learning model is a deep learning model that has not been subjected to weight tensor splitting processing;

[0031] The second output result is defined as the second inference result.

[0032] Optionally, verifying the tensor splitting result based on the first reasoning result and the second reasoning result includes:

[0033] determining a square value of a difference between the first reasoning result and the second reasoning result;

[0034] Determining a test accuracy of the deep learning model based on a ratio of the squared difference value to the second inference result;

[0035] In response to detecting that the test accuracy is less than or equal to a preset threshold, determining that the tensor splitting result is correct;

[0036] In response to detecting that the test accuracy is greater than a preset threshold, the tensor splitting result is determined to be erroneous, and the tensor splitting result is adjusted until the test accuracy of the degree learning model is less than or equal to the preset threshold.

[0037] In a second aspect, a tensor splitting result verification device is provided, the device comprising:

[0038] The first processing module is used to quantize the target weight of the target level in the deep learning model based on a preset quantization bit number to obtain a quantization processing result;

[0039] A second processing module is used to compress the quantization processing result and perform tensor splitting on the compression processing result to generate a file in a preset format;

[0040] A parsing module, used for parsing the file in the preset format to obtain a parsing result when a tensor splitting result verification request is detected;

[0041] A verification module is used to determine a first inference result of the deep learning model based on the analysis result, and to verify the tensor splitting result based on the first inference result and a second inference result, wherein the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0042] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0043] Based on the preset quantization bit number, the target weight of the target level in the deep learning model is quantized to obtain the quantization result;

[0044] Compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format;

[0045] In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result;

[0046] Based on the analysis result, a first inference result of the deep learning model is determined, and based on the first inference result and the second inference result, the tensor splitting result is verified, wherein the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0047] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0048] Based on the preset quantization bit number, the target weight of the target level in the deep learning model is quantized to obtain the quantization result;

[0049] Compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format;

[0050] In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result;

[0051] Based on the analysis result, a first inference result of the deep learning model is determined, and based on the first inference result and the second inference result, the tensor splitting result is verified, wherein the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0052] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0053] Based on the preset quantization bit number, the target weight of the target level in the deep learning model is quantized to obtain the quantization result;

[0054] Compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format;

[0055] In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result;

[0056] Based on the analysis result, a first inference result of the deep learning model is determined, and based on the first inference result and the second inference result, the tensor splitting result is verified, wherein the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0057] The above-mentioned tensor splitting result verification method, device, computer equipment and storage medium, the method includes: based on a preset quantization bit number, quantizing the target weight of the target level in the deep learning model to obtain a quantization processing result; compressing the quantization processing result, and performing tensor splitting on the compressed processing result to generate a file in a preset format; in response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result; based on the parsing result, determining a first inference result of the deep learning model, and verifying the tensor splitting result based on the first inference result and the second inference result, the second inference result is an inference result determined based on the deep learning model without weight tensor splitting processing. The present application automatically verifies the correctness of the int4 quantization weight tensor splitting result, and encapsulates the single-layer structure without the need to infer the complete model, thereby effectively improving the development efficiency of parallel tensor reasoning of the int4 quantization model. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 An application environment diagram of a tensor splitting result verification method in an embodiment;

[0059] Figure 2 Schematic diagram of a process of verifying a tensor splitting result in one embodiment;

[0060] Figure 3 A schematic diagram of a process flow of implementing a Self-attention tensor splitting verification solution of a tensor splitting result verification method in one embodiment;

[0061] Figure 4 A schematic diagram of an implementation flow of an MLP tensor splitting verification scheme of a tensor splitting result verification method in one embodiment;

[0062] Figure 5 is a structural block diagram of a tensor splitting result verification device in one embodiment;

[0063] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0066] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0067] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0068] The tensor splitting result verification method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the data processing platform set on the server 104 through the network, wherein the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0069] In one embodiment, Figure 2 As shown in the figure, a tensor splitting result verification method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0070] S1: Based on a preset number of quantization bits, the target weights of the target layer in the deep learning model are quantized to obtain a quantization result.

[0071] It should be noted that the deep learning model can be based on DL (Deep Learning, deep learning) framework model, such as Tensorflow, Pytorch, onnx, etc., this application for weight parsing result bin file, relying on DL framework pytorch; model quantization is to compress the original floating point model to effectively solve the single node memory limitation problem; the target weight at least includes a first target weight generated based on the attention mechanism and a second target weight generated based on the multi-layer perceptron, the attention mechanism refers to Self-attention, that is, the self-attention layer, the first target weight refers to the q weight matrix (q_weight), k weight matrix (k_weight), v weight matrix (v_weight) and c weight matrix (c_weight); multi-layer perceptron refers to MLP, the second target weight refers to gate_up_weight and down_weight; specifically, Self-attention and MLP are the core of reasoning correctness verification, among which query, key and value are the three main concepts of Self-attention, representing query, key and value respectively. For example, when searching for "deep learning", enter the four words "deep learning" into the search bar, press the search key, and the search engine will map the query query to the corresponding The relevant label key, such as "neural network", is then displayed to the searcher. The most matching result value is then displayed. Therefore, in the reasoning of Self-attention, each input is multiplied by the three weight matrices to obtain three new matrices, namely key, query, and value. The key and query matrices can be used to calculate the attention score. After obtaining the attention score matrix, the attention score matrix is ​​subjected to softmax calculation. The function of softmax is normalization, so that the sum of each item is 1. The advantage of normalization is that it highlights the largest value in the matrix and suppresses the components far below the maximum value. The normalized attention score is then multiplied by the value to achieve weighted calculation. The result of the calculation is the representation of the input after weighted summation by the attention mechanism. The whole process can split the weights without affecting the correctness of the results of each matrix element. MLP belongs to the feedforward layer. Two linear transformations are performed on the input to extract features more deeply. Gate, up, and down are linear layers in MLP. The target layer refers to the layer to which Self-attention and MLP belong in the model.Since most of the underlying layers cannot recognize the data type int4, the weights need to be quantized. The quantized structure is the single-layer Self-attention and MLP structure in the deep learning model to support truncation. It does not require reasoning about the complete model, and is fast to execute, providing the most fine-grained reference results for underlying development. ;

[0072] In some specific implementations, based on a preset number of quantization bits, the target weight of the target level in the deep learning model is quantized, and the quantization processing result includes:

[0073] Based on the weight quantization mechanism, the preset number of quantization bits is determined, where the weight quantization mechanism refers to AWQ (Activation-aware Weight Quantization). The preset number of quantization bits can be set according to actual needs. In order to effectively solve the single-node memory limitation problem, the preset number of quantization bits determined in this application is 4, that is, the 32-bit parameter quantity (i.e., weight) is quantized to 4 bits based on AWQ-int4;

[0074] According to the preset number of quantization bits, the first target weight and the second target weight are quantized respectively to obtain a first quantization result and a second quantization result.

[0075] Specifically, taking the weight q_weight as an example, assuming that the q_weight shape before quantization is (n_head*head_num, hidden_size), the q_weight after quantization becomes three related parameters (i.e., the quantization processing results), namely q_qweight, q_qzeros, and q_scales, and their shapes are:

[0076] (n_head*head_num / pack_num,hidden_size);

[0077] (n_head*head_num / pack_num,hidden_size / group_size);

[0078] (n_head*head_num,hidden_size / group_size).

[0079] The first quantization processing result and the second quantization processing result are obtained in the same way, which will not be described in detail here.

[0080] In the above implementation, by quantizing the int4 data, the universality of the method for automatically verifying the correctness of the splitting of the int4 quantization weight tensor is improved, and by quantizing the single-layer Self-attention and MLP structure in the deep learning model to support layer-by-layer verification and truncation verification, there is no need to reason about the complete model, and the execution is fast, thereby improving the development efficiency of tensor parallel reasoning of the int4 quantization model. At the same time, the quantization result can save video memory and computing resources and improve the reasoning speed of the model.

[0081] S2: compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format.

[0082] It should be noted that the int4 quantization (i.e., low-bit quantization) described in this application is to quantize 32-bit parameters into 4 bits, and the memory required can be reduced to 1 / 8 of the original. Since most devices do not support the int4 data type, by compressing and storing 8 int4 data into 1 int32 data, the compressed weight and quantization parameters are transmitted to the device. When the device bottom layer performs calculations, the int4 can be read by decompression and dequantized to obtain the real weight information. Therefore, the compression processing described in this step refers to compressing and storing 8 int4 data into 1 int32 data; the file in the preset format refers to the bin file; tensor splitting refers to dividing a large tensor into multiple small matrix multiplications along a fixed direction according to the following theoretical support, and each small matrix multiplication is assigned to an independent node for calculation, and its theoretical support includes:

[0083] (1) Computational level of model reasoning:

[0084] a. No matrix shape change is involved, and the calculation accuracy is one-dimensional (add, multiply, divide, sigmoid, etc.);

[0085] b. Matrix multiplication involving matrix shape changes;

[0086] (2) Structural level of model reasoning:

[0087] a. Both attention and MLP start with matrix multiplication Matmul0 and end with matrix multiplication Matmul1;

[0088] b. While keeping the input and output unchanged, the matrix multiplication calculation can be divided into two by splitting the weights, and the complete calculation result can be obtained by addition.

[0089] Taking into account the characteristics of calculation and structure, the weights of Matmul0 and Matmul1 are split along axis=0 and axis=1 respectively, and all calculations between the matrix multiplications Matmul0 and Matmul1 are split into two.

[0090] Furthermore, according to the characteristics of matrix multiplication, as shown in the following formula, each element in the result matrix of the matrix multiplication is the dot product of two vectors, so when vector B is divided into multiple small matrices along the direction of axis=1, it does not affect the calculation result of a single element in the result matrix:

[0091]

[0092] In some specific implementations, compressing the quantization result includes:

[0093] Get the preset integer data type, where the preset integer data type can be set according to actual needs. The preset integer data type used in this application is int32;

[0094] Based on the preset integer data type, the first quantization processing result and the second quantization processing result are compressed respectively to obtain a first compression processing result and a second compression processing result. Specifically, since many bottom layers do not support the data type int4, 8 int4 data are compressed and stored into 1 int32 data, so the compression parameter is pack_num=8, and group_size is a parameter selected in the quantization process, which is usually selected as 128;

[0095] In some specific implementations, performing tensor splitting on the compression processing result to generate a file in a preset format includes:

[0096] Based on the first target direction, performing tensor splitting processing on the first compression processing result to obtain a first tensor splitting processing result;

[0097] Based on the second target direction, performing tensor splitting processing on the first compression processing result to obtain a second tensor splitting processing result;

[0098] Based on the first target direction, performing tensor splitting processing on the second compression processing result to obtain a third tensor splitting processing result;

[0099] Based on the second target direction, performing tensor splitting processing on the second compression processing result to obtain a fourth tensor splitting processing result;

[0100] According to a preset format, saving the first tensor splitting processing result, the second tensor splitting processing result, the third tensor splitting processing result, and the fourth tensor splitting processing result to generate a file in the preset format;

[0101] The first target direction and the second target direction correspond to different tensor splitting directions.

[0102] Specifically, the first target direction refers to axis=0, and the second target direction refers to axis=1. Based on the first target direction and the second target direction, the tensor splitting method of the Self-attention int4 quantized weight includes:

[0103] If the shape of q / k / v_weight is (n_head*head_num / pack_num, hidden_size), and the shape of c_weight is (hidden_size / pack_num, n_head*head_num), then q / k / v_weight is split by axis=0 to obtain q / k / v_split, i.e., the result of the first tensor splitting process, and c_weight is split by axis=1 to obtain c_split, i.e., the result of the second tensor splitting process, wherein one-dimensional calculation has no effect on the structure, and no one-dimensional calculation is required;

[0104] The tensor splitting methods for MLP int4 quantized weights include:

[0105] Assuming that the shape of gate_up_weight is (intermediate_size*2 / pack_num, hidden_size), and the shape of down_weight is (hidden_size / pack_num, intermediate_size), gate_up_weight is split by axis=0 to obtain gate_up_split, i.e., the third tensor splitting result, and down_weight is split by axis=1 to obtain down_split, i.e., the fourth tensor splitting result;

[0106] More specifically, the following is the code for implementing the tensor splitting of the target weight (the specific weights to be split can be seen in the parameter_to_split part of the following code) through the parsing script, and saving the split results as a bin file:

[0107] parameter_to_split=[

[0108] #tuple of(name,index to split)

[0109] ('self_attn.q_proj.weight', 0),

[0110] ('self_attn.q_proj.bias', 0),

[0111] ('self_attn.k_proj.weight', 0),

[0112] ('self_attn.k_proj.bias', 0),

[0113] ('self_attn.v_proj.weight', 0),

[0114] ('self_attn.v_proj.bias', 0),

[0115] ('self_attn.o_proj.weight', 1),

[0116] ('mlp.gate_proj.weight', 0),

[0117] ('mlp.gate_proj.bias', 0),

[0118] ('mlp.up_proj.weight', 0),

[0119] ('mlp.up_proj.bias', 0),

[0120] ('mlp.down_proj.weight', 1), ]

[0122] x.tofile(os.path.join(args.weights, freq["name"]+".bin"))

[0123] Then, the bin file can be read as the data type of torch.Tensor() and reshaped into the shape that the weight should have for backup, that is, the weight after the tensor is split is reshaped into the shape that the target weight should have. The code is as follows:

[0124] input_mha=np.fromfile(input_path, dtype=np.float16,

[0125] count=input_length*max_input_length,offset=0)

[0126] In the above implementation, after the target weight is tensor split to obtain the tensor split result, the tensor split result can be distributed to the corresponding nodes for parallel computing, thereby improving the computing speed of the target level of the deep learning model, and then improving the overall computing speed of the deep learning model.

[0127] S3: In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result.

[0128] It should be noted that when a tensor splitting result verification request is detected, the bin file will be read and transferred to the device for inference.

[0129] In some specific implementations, in response to detecting a tensor splitting result verification request, parsing the file in the preset format, and obtaining the parsing result includes:

[0130] In response to detecting a tensor splitting result verification request, the weights in the file of the preset format are sorted and a decompression operation is performed to obtain a first parsing result. Specifically, the default arrangement of the compressed weights is a GPU-friendly arrangement, i.e., AWQ_order=[0, 2, 4, 6, 1, 3, 5, 7]. In actual calculation, the correct order needs to be obtained according to the arrangement, i.e., AWQ_order=[0, 4, 1, 5, 2, 6, 3, 7]. In order to improve the model reasoning performance, the order of the weights needs to be rearranged before the decompression operation is performed. The code includes:

[0131] reverse_order_tensor=

[0132] torch.arange(izeros.shape[-1],dtype=torch.int32,device=izeros.device)

[0133] reverse_order_tensor=reverse_order_tensor.view(-1,storage_bits / / bits)

[0134] reverse_order_tensor=reverse_order_tensor[:,AWQ_REVERSE_ORDER]

[0135] reverse_order_tensor=reverse_order_tensor.view(-1)

[0136] Furthermore, the weights saved as int32 are actually int4 and the decompression operation is performed. The code is:

[0137] shifts=torch.arange(0,storage_bits,q_bits,device=qmatrix.device)

[0138] iweights=torch.bitwise_right_shift(qweight[:,:,None],shifts[None,None,:]).view(qweight.shape[0],-1)

[0139] izeros=torch.bitwise_right_shift(qzeros[:,:,None],shifts[None,None,:]).view(qzeros.shape[0],-1)

[0140] iweights=iweights.to(torch.int8)&0x0F

[0141] izeros=izeros.to(torch.int8)&0x0F

[0142] The first parsing result is subjected to an inverse quantization operation to obtain a second parsing result, that is, the correct weight, wherein the inverse quantization code is as follows:

[0143] gate_up_w=(gate_up_w-gate_up_z)*gate_up_s

[0144] down_w=(down_w-down_z)*down_s

[0145] In the above implementation, when the tensor splitting result needs to be verified, the bin file is read and parsed and transmitted to the device side for deep learning reasoning to verify the correctness of the parallel reasoning of the int4 quantized weight tensor, thereby improving the execution efficiency.

[0146] S4: Based on the analysis result, determine a first inference result of the deep learning model, and verify the tensor splitting result based on the first inference result and the second inference result, where the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0147] It should be noted that the first inference result refers to the inference result determined by the deep learning model that has undergone weight tensor splitting processing, that is, the inference result obtained by performing model inference on the verification data using the model weights after quantization, compression and tensor splitting processing; the second inference result is the inference result determined based on the deep learning model that has not undergone weight tensor splitting processing, that is, the inference result obtained by performing model inference on the verification data using the original weights of the model that have not undergone quantization, compression and tensor splitting processing.

[0148] In some specific implementations, determining a first inference result of the deep learning model based on the parsing result includes:

[0149] Based on the analysis result, a target deep learning model on which weight tensor splitting processing has been performed is generated;

[0150] Inputting target verification data into the target deep learning model to obtain a first output result, wherein the target verification data refers to input data of the model;

[0151] The first output result is defined as the first inference result.

[0152] In some specific implementations, the method for determining the second reasoning result includes:

[0153] Inputting the target verification data into an initial deep learning model to obtain a second output result, wherein the initial deep learning model is a deep learning model that has not been subjected to weight tensor splitting processing;

[0154] The second output result is defined as the second inference result.

[0155] In some specific implementations, verifying the tensor splitting result based on the first reasoning result and the second reasoning result includes:

[0156] determining a square value of a difference between the first reasoning result and the second reasoning result;

[0157] Based on the ratio of the square of the difference value to the second inference result, the test accuracy of the deep learning model is determined, and the specific calculation formula of the test accuracy is:

[0158]

[0159] Among them, (f1-f2) 2 represents the square value of the difference between the first inference result f1 and the second inference result f2, Indicates the test accuracy;

[0160] In response to detecting that the test accuracy is less than or equal to a preset threshold, determining that the tensor splitting result is correct, wherein the preset threshold can be set according to actual needs, such as 0.01;

[0161] In response to detecting that the test accuracy is greater than a preset threshold, the tensor splitting result is determined to be erroneous, and the tensor splitting result is adjusted, that is, the quantization and splitting process is readjusted until the test accuracy of the degree learning model is less than or equal to the preset threshold.

[0162] Specifically, use torch.Tensor to build the reasoning implementation of Self-attention and MLP, including:

[0163] a. Construct rms_norm (root mean square layer normalization), epsilon is used as a variable, which can be passed according to actual needs (commonly used e-02 or e-03), and the code is as follows:

[0164] defrms_norm(a,n_w,eps):

[0165] input_dtype = a.dtype

[0166] a = a.to(torch.float32)

[0167] variance=a.pow(2).mean(-1,keepdim=True)

[0168] a=a*torch.rsqrt(variance+eps)

[0169] n_w=torch.Tensor(n_w,device='cpu').to(torch.float32)

[0170] n_w=torch.reshape(n_w,(hidden_size,))

[0171] return n_w*a.to(input_dtype)

[0172] b. If it is Self-attention, construct the position rotation function, calculate cosine, sinine, and PoRe. The code is as follows:

[0173] defrotate_half(x):

[0174] """Rotates halfthe hidden dims ofthe input."""

[0175] x1=x[...,:x.shape[-1] / / 2]

[0176] x2=x[...,x.shape[-1] / / 2:]

[0177] return torch.cat((-x2,x1),dim=-1)

[0178] defapply_rotary_pos_emb(q,k,cos,sin,position_ids):

[0179] cos=cos.squeeze(1).squeeze(0)#[seq_len,dim]

[0180] sin=sin.squeeze(1).squeeze(0)#[seq_len,dim]

[0181] cos=cos[position_ids].unsqueeze(1)#[bs,1,seq_len,dim]

[0182] sin=sin[position_ids].unsqueeze(1)#[bs,1,seq_len,dim]

[0183] q_embed=(q*cos)+(rotate_half(q)*sin)

[0184] k_embed=(k*cos)+(rotate_half(k)*sin)

[0185] return q_embed,k_embed

[0186] defRotaryEmbedding(q,k):

[0187] base=10000

[0188] inv_freq=1.0 / (base**(torch.arange(0,head_num,2).float().to('cpu') / head_num))

[0189] t=torch.arange(input_length,device='cpu',dtype=inv_freq.dtype)

[0190] freqs=torch.einsum("i,j->ij",t,inv_freq)

[0191] emb=torch.cat((freqs,freqs),dim=-1)

[0192] cos=emb.cos()[None,None,:,:].to(torch.float32)

[0193] sin=emb.sin()[None,None,:,:].to(torch.float32)

[0194] cos=cos[:,:,:input_length,...].to(dtype=q.dtype)

[0195] sin=sin[:,:,:input_length,...].to(dtype=q.dtype)

[0196] q,k=apply_rotary_pos_emb(q,k,cos,sin,position_ids)

[0197] return q,k

[0198] c. Construct the matrix multiplication calculation process. The Self-attention matrix multiplication implementation scheme is as follows Figure 3 As shown, the MLP matrix multiplication implementation is as follows Figure 4 shown.

[0199] d. Only the first segment of the tensor is calculated Residual_add (residual connection). The code is as follows:

[0200] ifmpi_rank==0:

[0201] res=torch.add(input_mha,res)

[0202] else:

[0203] res=res

[0204] e. Perform allreduce communication (full protocol communication) on the calculation results through MPI. The code is as follows:

[0205] res=comm.allreduce(res,op=MPI.SUM)

[0206] More specifically, Figure 3 and Figure 4 The process is further described, including: when the target verification data is processed by the above-mentioned deep learning model, the rms norm (root mean square norm) of each vector in the target verification data can be calculated first, wherein the rms norm can be used by the deep learning model to evaluate the importance or weight of the corresponding vector; then, the target weight in the target level is tensor-split through the matrix multiplication calculation module to obtain the tensor split result, wherein the execution subjects of the tensor split are Self-attention and MLP, respectively, and the tensor split result is matrix-multiplied with the target verification data to obtain the inference result; then, if the rank (Rank) of the inference result is not equal to 0 through the conditional judgment statement Rank! = 0, the residual addition operation is performed through residual_add to correct the inference result, thereby ensuring the integrity and accuracy of the inference result; after the inference result is corrected, allreduce can be performed to add the results of each sub-operation to obtain the above-mentioned first inference result.

[0207] Furthermore, the error between the first inference result and the pytorch inference result before disassembly (i.e., the second inference result) is calculated to determine its correctness. Specifically, the deep learning model before disassembly is inferred, and the input and output results of Self-attention are exported as bin files.

[0208] Read the input exported by pytorch as the demo input, record the calculation result as the first inference result, read the output result exported by pytorch as the second inference result, the second inference result as the CPU output matrix can be regarded as the expected result, and the first inference result is the inference result of the demo after splitting, so it meets the requirements. Detection, based on this, using Define the test accuracy. By comparing the test accuracy with the preset threshold, the accuracy of tensor splitting can be judged.

[0209] Furthermore, the above-mentioned first reasoning result can be saved as a preset format file. When the tensor splitting result does not meet the accuracy requirement, the preset format file is analyzed to obtain an analysis report, so as to adjust the tensor splitting operation based on the analysis report, wherein the above-mentioned preset format file can be a bin file. In the process of verifying the tensor splitting result through the above-mentioned verification, the input and output results of any link can be saved as a bin file. When a call request for the bin file is detected, the bin file can be called to display it to the user, or, when it is determined that the target weight tensor splitting result does not meet the accuracy requirement, an analysis report can be generated based on the analysis request of the bin file, wherein the weight of the tensor splitting operation that needs to be adjusted based on the analysis report is the weight tensor splitting operation in Self-attention and / or MLP, and the specific weight tensor splitting operation that needs to be adjusted can be determined according to the analysis report.

[0210] In the above implementation, the model correctness after the decompression, dequantization and tensor splitting operations on the target weight can be determined based on the comparison results between the test accuracy and the preset threshold, so as to improve the efficiency of the correctness verification of the tensor weight splitting results. At the same time, the input and output results of any link can be saved as a bin file, so as to facilitate printing out the calculation results after the tensor weight splitting, which can provide a reference result for the verification of the correctness of the calculation results in the back-end calculation execution stage.

[0211] In some embodiments, the method further comprises:

[0212] Based on the current weight splitting result, a first evaluation index and a second evaluation index for evaluating the deep learning model are obtained, wherein the first evaluation index is an index of the time dimension, i.e., the proportion of time spent, and the second evaluation index is an index of the resource usage dimension, i.e., the proportion of resources;

[0213] Based on the first evaluation index and the second evaluation index, an evaluation value is calculated and determined, wherein the calculation formula is P=L1a+L2b, wherein P represents the evaluation value, L1 and L2 both represent weight coefficients, a represents resource proportion, and b represents time proportion;

[0214] In response to detecting that the evaluation value is less than a first preset value, obtaining a loss value of the deep learning model, wherein the first preset value can be set according to actual needs;

[0215] In response to detecting that the loss value is greater than a second preset value, re-quantizing the weights and re-splitting the tensors to adjust the tensor splitting results, thereby updating the deep learning model, wherein the second preset value can be set according to actual needs;

[0216] In response to detecting that the evaluation value is greater than or equal to a first preset value and / or the loss value is less than or equal to a second preset value, the weight splitting result is defined as a correct result to update the deep learning model.

[0217] In the above implementation, the tensor splitting results are further verified based on the time dimension, resource usage dimension and loss value dimension, thereby improving the accuracy of the verification results and improving the accuracy of deep learning model reasoning.

[0218] In the above-mentioned tensor splitting result verification method, the method includes: based on a preset quantization bit number, quantizing the target weight of the target level in the deep learning model to obtain a quantization processing result; compressing the quantization processing result, and performing tensor splitting on the compressed processing result to generate a file in a preset format; in response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result; based on the parsing result, determining a first inference result of the deep learning model, and verifying the tensor splitting result based on the first inference result and the second inference result, the second inference result is an inference result determined based on the deep learning model without weight tensor splitting processing. The present application automatically verifies the correctness of the int4 quantization weight tensor splitting result, and encapsulates the single-layer structure without reasoning the complete model, thereby effectively improving the development efficiency of the int4 quantization model tensor parallel reasoning. The present application proposes an automatic verification method for Self-attention and MLP for correctness verification of the tensor parallel reasoning scheme of the int4 quantization weight under development. The method of int4 weight decompression, dequantization correctness and tensor splitting reasoning result correctness can effectively improve the development efficiency of tensor parallel reasoning of int4 quantization model, and can print out the step-by-step calculation results after tensor splitting, and can provide reference results for the verification of the correctness of the calculation results in the back-end calculation execution stage. Without performing complete model reasoning, this application automatically verifies the correctness of the dequantization and tensor splitting results of Self-attention and MLP weights, supports layer-by-layer verification, truncation verification, and supports verification of each operator, which can provide the most fine-grained reference results for underlying development. This application encapsulates single-layer reasoning and executes quickly, is convenient for truncation, has a fast startup speed, can be executed directly on the CPU, and does not rely on the underlying layer. This application has high universality and can be applied to a variety of large models such as LLAMA2, LLAMA3, Qwen, chatGLM2, gpt2, and is applicable to a variety of quantized weights such as AWQ and GPTQ.

[0219] It should be understood that although Figure 2-Figure 4The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0220] In one embodiment, Figure 5 As shown, a tensor splitting result verification device is provided, comprising: a first processing module, a second processing module, a parsing module and a verification module, wherein:

[0221] The first processing module is used to quantize the target weight of the target level in the deep learning model based on a preset quantization bit number to obtain a quantization processing result;

[0222] A second processing module is used to compress the quantization processing result and perform tensor splitting on the compression processing result to generate a file in a preset format;

[0223] A parsing module, used for parsing the file in the preset format to obtain a parsing result when a tensor splitting result verification request is detected;

[0224] A verification module is used to determine a first inference result of the deep learning model based on the analysis result, and to verify the tensor splitting result based on the first inference result and a second inference result, wherein the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0225] As a preferred implementation manner, in the embodiment of the present invention, the first processing module is specifically used for:

[0226] Based on the weighted quantization mechanism, determine the preset number of quantization bits;

[0227] According to the preset quantization bit number, the first target weight and the second target weight are quantized respectively to obtain a first quantization processing result and a second quantization processing result, wherein the target weights at least include a first target weight generated based on an attention mechanism and a second target weight generated based on a multilayer perceptron.

[0228] As a preferred implementation, in the embodiment of the present invention, the second processing module is specifically used for:

[0229] Get the preset integer data type;

[0230] Based on the preset integer data type, compressing the first quantization processing result and the second quantization processing result respectively to obtain a first compression processing result and a second compression processing result;

[0231] The compression processing result is split into tensors to generate files in a preset format, including:

[0232] Based on the first target direction, performing tensor splitting processing on the first compression processing result to obtain a first tensor splitting processing result;

[0233] Based on the second target direction, performing tensor splitting processing on the first compression processing result to obtain a second tensor splitting processing result;

[0234] Based on the first target direction, performing tensor splitting processing on the second compression processing result to obtain a third tensor splitting processing result;

[0235] Based on the second target direction, performing tensor splitting processing on the second compression processing result to obtain a fourth tensor splitting processing result;

[0236] According to a preset format, saving the first tensor splitting processing result, the second tensor splitting processing result, the third tensor splitting processing result, and the fourth tensor splitting processing result to generate a file in the preset format;

[0237] The first target direction and the second target direction correspond to different tensor splitting directions.

[0238] As a preferred implementation, in the embodiment of the present invention, the parsing module is specifically used for:

[0239] In response to detecting a tensor splitting result verification request, sorting the weights in the file in the preset format and performing a decompression operation to obtain a first parsing result;

[0240] Perform a dequantization operation on the first parsing result to obtain a second parsing result.

[0241] As a preferred implementation, in the embodiment of the present invention, the verification module is specifically used for:

[0242] Based on the analysis result, a target deep learning model on which weight tensor splitting processing has been performed is generated;

[0243] Inputting the target verification data into the target deep learning model to obtain a first output result;

[0244] The first output result is defined as the first inference result.

[0245] As a preferred implementation, in the embodiment of the present invention, the verification module is further used for:

[0246] Inputting the target verification data into an initial deep learning model to obtain a second output result, wherein the initial deep learning model is a deep learning model that has not been subjected to weight tensor splitting processing;

[0247] The second output result is defined as the second inference result.

[0248] As a preferred implementation, in the embodiment of the present invention, the verification module is further used for:

[0249] determining a square value of a difference between the first reasoning result and the second reasoning result;

[0250] Determining a test accuracy of the deep learning model based on a ratio of the squared difference value to the second inference result;

[0251] In response to detecting that the test accuracy is less than or equal to a preset threshold, determining that the tensor splitting result is correct;

[0252] In response to detecting that the test accuracy is greater than a preset threshold, the tensor splitting result is determined to be erroneous, and the tensor splitting result is adjusted until the test accuracy of the degree learning model is less than or equal to the preset threshold.

[0253] For the specific limitations of the tensor splitting result verification device, please refer to the limitations of the tensor splitting result verification method above, which will not be repeated here. Each module in the above-mentioned tensor splitting result verification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0254] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a tensor splitting result verification method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0255] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0256] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0257] S1: Based on the preset quantization bit number, the target weight of the target level in the deep learning model is quantized to obtain the quantization result;

[0258] S2: compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format;

[0259] S3: In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result;

[0260] S4: Based on the analysis result, determine a first inference result of the deep learning model, and verify the tensor splitting result based on the first inference result and the second inference result, where the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0261] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0262] Based on the weighted quantization mechanism, determine the preset number of quantization bits;

[0263] According to the preset quantization bit number, the first target weight and the second target weight are quantized respectively to obtain a first quantization processing result and a second quantization processing result, wherein the target weights at least include a first target weight generated based on an attention mechanism and a second target weight generated based on a multilayer perceptron.

[0264] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0265] Get the preset integer data type;

[0266] Based on the preset integer data type, compressing the first quantization processing result and the second quantization processing result respectively to obtain a first compression processing result and a second compression processing result;

[0267] The compression processing result is split into tensors to generate files in a preset format, including:

[0268] Based on the first target direction, performing tensor splitting processing on the first compression processing result to obtain a first tensor splitting processing result;

[0269] Based on the second target direction, performing tensor splitting processing on the first compression processing result to obtain a second tensor splitting processing result;

[0270] Based on the first target direction, performing tensor splitting processing on the second compression processing result to obtain a third tensor splitting processing result;

[0271] Based on the second target direction, performing tensor splitting processing on the second compression processing result to obtain a fourth tensor splitting processing result;

[0272] According to a preset format, saving the first tensor splitting processing result, the second tensor splitting processing result, the third tensor splitting processing result, and the fourth tensor splitting processing result to generate a file in the preset format;

[0273] The first target direction and the second target direction correspond to different tensor splitting directions.

[0274] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0275] In response to detecting a tensor splitting result verification request, sorting the weights in the file in the preset format and performing a decompression operation to obtain a first parsing result;

[0276] Perform a dequantization operation on the first parsing result to obtain a second parsing result.

[0277] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0278] Based on the analysis result, a target deep learning model on which weight tensor splitting processing has been performed is generated;

[0279] Inputting the target verification data into the target deep learning model to obtain a first output result;

[0280] The first output result is defined as the first inference result.

[0281] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0282] Inputting the target verification data into an initial deep learning model to obtain a second output result, wherein the initial deep learning model is a deep learning model that has not been subjected to weight tensor splitting processing;

[0283] The second output result is defined as the second inference result.

[0284] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0285] determining a square value of a difference between the first reasoning result and the second reasoning result;

[0286] Determining a test accuracy of the deep learning model based on a ratio of the squared difference value to the second inference result;

[0287] In response to detecting that the test accuracy is less than or equal to a preset threshold, determining that the tensor splitting result is correct;

[0288] In response to detecting that the test accuracy is greater than a preset threshold, the tensor splitting result is determined to be erroneous, and the tensor splitting result is adjusted until the test accuracy of the degree learning model is less than or equal to the preset threshold.

[0289] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0290] S1: Based on the preset quantization bit number, the target weight of the target level in the deep learning model is quantized to obtain the quantization result;

[0291] S2: compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format;

[0292] S3: In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result;

[0293] S4: Based on the analysis result, determine a first inference result of the deep learning model, and verify the tensor splitting result based on the first inference result and the second inference result, where the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0294] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0295] Based on the weighted quantization mechanism, determine the preset number of quantization bits;

[0296] According to the preset quantization bit number, the first target weight and the second target weight are quantized respectively to obtain a first quantization processing result and a second quantization processing result, wherein the target weights at least include a first target weight generated based on an attention mechanism and a second target weight generated based on a multilayer perceptron.

[0297] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0298] Get the preset integer data type;

[0299] Based on the preset integer data type, compressing the first quantization processing result and the second quantization processing result respectively to obtain a first compression processing result and a second compression processing result;

[0300] The compression processing result is split into tensors to generate files in a preset format, including:

[0301] Based on the first target direction, performing tensor splitting processing on the first compression processing result to obtain a first tensor splitting processing result;

[0302] Based on the second target direction, performing tensor splitting processing on the first compression processing result to obtain a second tensor splitting processing result;

[0303] Based on the first target direction, performing tensor splitting processing on the second compression processing result to obtain a third tensor splitting processing result;

[0304] Based on the second target direction, performing tensor splitting processing on the second compression processing result to obtain a fourth tensor splitting processing result;

[0305] According to a preset format, saving the first tensor splitting processing result, the second tensor splitting processing result, the third tensor splitting processing result, and the fourth tensor splitting processing result to generate a file in the preset format;

[0306] The first target direction and the second target direction correspond to different tensor splitting directions.

[0307] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0308] In response to detecting a tensor splitting result verification request, sorting the weights in the file in the preset format and performing a decompression operation to obtain a first parsing result;

[0309] Perform a dequantization operation on the first parsing result to obtain a second parsing result.

[0310] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0311] Based on the analysis result, a target deep learning model on which weight tensor splitting processing has been performed is generated;

[0312] Inputting the target verification data into the target deep learning model to obtain a first output result;

[0313] The first output result is defined as the first inference result.

[0314] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0315] Inputting the target verification data into an initial deep learning model to obtain a second output result, wherein the initial deep learning model is a deep learning model that has not been subjected to weight tensor splitting processing;

[0316] The second output result is defined as the second inference result.

[0317] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0318] determining a square value of a difference between the first reasoning result and the second reasoning result;

[0319] Determining a test accuracy of the deep learning model based on a ratio of the squared difference value to the second inference result;

[0320] In response to detecting that the test accuracy is less than or equal to a preset threshold, determining that the tensor splitting result is correct;

[0321] In response to detecting that the test accuracy is greater than a preset threshold, the tensor splitting result is determined to be erroneous, and the tensor splitting result is adjusted until the test accuracy of the degree learning model is less than or equal to the preset threshold.

[0322] In one embodiment, a computer program product is provided, the computer program product comprising a computer program, the computer program when executed by a processor implements the following steps:

[0323] S1: Based on the preset quantization bit number, the target weight of the target level in the deep learning model is quantized to obtain the quantization result;

[0324] S2: compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format;

[0325] S3: In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result;

[0326] S4: Based on the analysis result, determine a first inference result of the deep learning model, and verify the tensor splitting result based on the first inference result and the second inference result, where the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

[0327] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0328] Based on the weighted quantization mechanism, determine the preset number of quantization bits;

[0329] According to the preset quantization bit number, the first target weight and the second target weight are quantized respectively to obtain a first quantization processing result and a second quantization processing result, wherein the target weights at least include a first target weight generated based on an attention mechanism and a second target weight generated based on a multilayer perceptron.

[0330] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0331] Get the preset integer data type;

[0332] Based on the preset integer data type, compressing the first quantization processing result and the second quantization processing result respectively to obtain a first compression processing result and a second compression processing result;

[0333] The compression processing result is split into tensors to generate files in a preset format, including:

[0334] Based on the first target direction, performing tensor splitting processing on the first compression processing result to obtain a first tensor splitting processing result;

[0335] Based on the second target direction, performing tensor splitting processing on the first compression processing result to obtain a second tensor splitting processing result;

[0336] Based on the first target direction, performing tensor splitting processing on the second compression processing result to obtain a third tensor splitting processing result;

[0337] Based on the second target direction, performing tensor splitting processing on the second compression processing result to obtain a fourth tensor splitting processing result;

[0338] According to a preset format, saving the first tensor splitting processing result, the second tensor splitting processing result, the third tensor splitting processing result, and the fourth tensor splitting processing result to generate a file in the preset format;

[0339] The first target direction and the second target direction correspond to different tensor splitting directions.

[0340] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0341] In response to detecting a tensor splitting result verification request, sorting the weights in the file in the preset format and performing a decompression operation to obtain a first parsing result;

[0342] Perform a dequantization operation on the first parsing result to obtain a second parsing result.

[0343] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0344] Based on the analysis result, a target deep learning model on which weight tensor splitting processing has been performed is generated;

[0345] Inputting the target verification data into the target deep learning model to obtain a first output result;

[0346] The first output result is defined as the first inference result.

[0347] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0348] Inputting the target verification data into an initial deep learning model to obtain a second output result, wherein the initial deep learning model is a deep learning model that has not been subjected to weight tensor splitting processing;

[0349] The second output result is defined as the second inference result.

[0350] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0351] determining a square value of a difference between the first reasoning result and the second reasoning result;

[0352] Determining a test accuracy of the deep learning model based on a ratio of the squared difference value to the second inference result;

[0353] In response to detecting that the test accuracy is less than or equal to a preset threshold, determining that the tensor splitting result is correct;

[0354] In response to detecting that the test accuracy is greater than a preset threshold, the tensor splitting result is determined to be erroneous, and the tensor splitting result is adjusted until the test accuracy of the degree learning model is less than or equal to the preset threshold.

[0355] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0356] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0357] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A tensor splitting result verification method, characterized in that: The method comprises: Based on the preset quantization bit number, the target weight of the target level in the deep learning model is quantized to obtain the quantization result; Compressing the quantization processing result, and performing tensor splitting on the compression processing result to generate a file in a preset format; In response to detecting a tensor splitting result verification request, parsing the file in the preset format to obtain a parsing result; Based on the analysis result, a first inference result of the deep learning model is determined, and based on the first inference result and the second inference result, the tensor splitting result is verified, wherein the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

2. The tensor splitting result verification method according to claim 1, characterized in that: The target weights at least include a first target weight generated based on an attention mechanism and a second target weight generated based on a multi-layer perceptron. Based on a preset number of quantization bits, the target weights of the target level in the deep learning model are quantized, and the quantization results obtained include: Based on the weighted quantization mechanism, determine the preset number of quantization bits; According to the preset number of quantization bits, the first target weight and the second target weight are quantized respectively to obtain a first quantization result and a second quantization result.

3. The tensor splitting result verification method according to claim 2, characterized in that: The compressing process of the quantization process result includes: Get the preset integer data type; Based on the preset integer data type, compressing the first quantization processing result and the second quantization processing result respectively to obtain a first compression processing result and a second compression processing result; The compression processing result is split into tensors to generate files in a preset format, including: Based on the first target direction, performing tensor splitting processing on the first compression processing result to obtain a first tensor splitting processing result; Based on the second target direction, performing tensor splitting processing on the first compression processing result to obtain a second tensor splitting processing result; Based on the first target direction, performing tensor splitting processing on the second compression processing result to obtain a third tensor splitting processing result; Based on the second target direction, performing tensor splitting processing on the second compression processing result to obtain a fourth tensor splitting processing result; According to a preset format, saving the first tensor splitting processing result, the second tensor splitting processing result, the third tensor splitting processing result, and the fourth tensor splitting processing result to generate a file in the preset format; The first target direction and the second target direction correspond to different tensor splitting directions.

4. The tensor splitting result verification method according to claim 1, characterized in that: In response to detecting a tensor splitting result verification request, the file in the preset format is parsed, and the parsing result obtained includes: In response to detecting a tensor splitting result verification request, sorting the weights in the file in the preset format and performing a decompression operation to obtain a first parsing result; Perform a dequantization operation on the first parsing result to obtain a second parsing result.

5. The tensor splitting result verification method according to claim 1, characterized in that: Based on the analysis result, determining a first inference result of the deep learning model includes: Based on the analysis result, a target deep learning model on which weight tensor splitting processing has been performed is generated; Inputting the target verification data into the target deep learning model to obtain a first output result; The first output result is defined as the first inference result.

6. The tensor splitting result verification method according to claim 5, characterized in that: The method for determining the second reasoning result includes: Inputting the target verification data into an initial deep learning model to obtain a second output result, wherein the initial deep learning model is a deep learning model that has not been subjected to weight tensor splitting processing; The second output result is defined as the second inference result.

7. The tensor splitting result verification method according to claim 6, characterized in that: Based on the first reasoning result and the second reasoning result, verifying the tensor splitting result includes: determining a square value of a difference between the first reasoning result and the second reasoning result; Determining a test accuracy of the deep learning model based on a ratio of the squared difference value to the second inference result; In response to detecting that the test accuracy is less than or equal to a preset threshold, determining that the tensor splitting result is correct; In response to detecting that the test accuracy is greater than a preset threshold, the tensor splitting result is determined to be erroneous, and the tensor splitting result is adjusted until the test accuracy of the degree learning model is less than or equal to the preset threshold.

8. A tensor splitting result verification device, characterized in that: The device comprises: The first processing module is used to quantize the target weight of the target level in the deep learning model based on a preset quantization bit number to obtain a quantization processing result; A second processing module is used to compress the quantization processing result and perform tensor splitting on the compression processing result to generate a file in a preset format; A parsing module, used for parsing the file in the preset format to obtain a parsing result when a tensor splitting result verification request is detected; A verification module is used to determine a first inference result of the deep learning model based on the analysis result, and to verify the tensor splitting result based on the first inference result and a second inference result, wherein the second inference result is an inference result determined based on the deep learning model without performing weight tensor splitting processing.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images