Model processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a model processing method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps of performing low-dimensional conversion processing on input data, a target layer and an activation function of a next layer of the target layer based on a diffusion accounting method to obtain a converted feature vector; based on a preset interlayer similarity algorithm and the converted feature vector, constructing a similarity matrix between the target layer and the next layer of the target layer; and determining similar layers in the similarity matrix, fusing the similar layers according to preset fusion parameters to obtain fused similar layers, and obtaining a compressed large language model based on the fused similar layers. By adopting the method, the model scale can be effectively compressed while the performance of the large language model is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a model processing method, apparatus, computer equipment, readable storage medium and program product. Background Art

[0002] In the field of natural language processing, especially in the application of large language models (LLMs), large language models based on deep learning can understand complex language semantics and generate coherent and relevant text content through pre-training on large-scale text data. Large language models often require huge computing resources and storage space during deployment. In traditional technologies, model compression methods usually use parameter pruning and quantization methods, which leads to the performance degradation of large language models. Summary of the invention

[0003] Based on this, it is necessary to provide a model processing method, device, computer equipment, readable storage medium and program product to address the above technical problems, which can effectively compress the model size while ensuring the performance of the large language model.

[0004] In a first aspect, the present application provides a model processing method, comprising:

[0005] Based on the diffusion kernel algorithm, low-dimensional transformation processing is performed on the input data, the target layer and the activation function of the next layer of the target layer to obtain a transformed feature vector, and the model includes the target layer and the next layer of the target layer;

[0006] Based on a preset inter-layer similarity algorithm and the converted feature vector, construct a similarity matrix between the target layer and a layer below the target layer;

[0007] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer, and a compressed large language model is obtained based on each of the fused similarity layers.

[0008] In one embodiment, the low-dimensional transformation based on the diffusion kernel algorithm and the activation function of the input data, the target layer and the next layer of the target layer to obtain the transformed feature vector includes:

[0009] Constructing a kernel matrix based on input data, activation functions of a target layer and a layer below the target layer, and kernel bandwidth parameters, and constructing a degree matrix based on elements of the kernel matrix;

[0010] Determining the difference between the degree matrix and the kernel matrix as a Laplacian matrix;

[0011] Determine an eigenvector corresponding to a target eigenvalue of the Laplace matrix;

[0012] The product of the eigenvector and the Laplacian matrix is ​​determined as the transformed eigenvector.

[0013] In one embodiment, constructing a similarity matrix between the target layer and a layer next to the target layer based on a preset inter-layer similarity algorithm and the converted feature vector includes:

[0014] Calculating a joint probability distribution function of the target layer and a layer below the target layer, and calculating an edge probability distribution function of the target layer and an edge probability distribution function of a layer below the target layer;

[0015] Based on the joint probability distribution function, the marginal probability distribution function of the target layer and the marginal probability distribution function of the next layer of the target layer, a similarity matrix of the target layer and the next layer of the target layer is constructed.

[0016] In one embodiment, the method further comprises:

[0017] The sum of the multi-head attention value, the feedforward value and the layer normalization value of the activation function of the previous layer of the target layer is determined as the activation function of the target layer.

[0018] In one embodiment, the method further comprises:

[0019] Calculating performance parameters of the compressed large language model;

[0020] If the performance parameter does not meet the preset performance condition, continue to execute the step of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the converted feature vector until the performance parameter meets the preset performance condition.

[0021] In one embodiment, the method further comprises:

[0022] Constructing an initial inference tree of the large language model based on a preset inference tree inference algorithm, wherein the initial inference tree includes at least one target node;

[0023] For each of the target nodes, calculate the product value between the confidences of the nodes included in the path from the root node to the target node, and determine the product value as the node value function of the target node; determine a modified node value function of the target node based on the node value function, the depth value of the target node, and a hyperparameter for adjusting the depth effect; and rearrange the path from the root node to the target node based on the modified node value function to obtain a rearranged path;

[0024] A target inference tree is obtained based on the rearranged paths corresponding to each of the target nodes.

[0025] In a second aspect, the present application also provides a model processing device, comprising:

[0026] A processing module, for performing low-dimensional transformation processing on the input data, the target layer and the activation function of the next layer of the target layer based on a diffusion kernel algorithm to obtain a transformed feature vector, wherein the model includes the target layer and the next layer of the target layer;

[0027] A construction module, used for constructing a similarity matrix between the target layer and a layer below the target layer based on a preset inter-layer similarity algorithm and the converted feature vector;

[0028] The fusion module is used to determine the similarity layer in the similarity matrix, and fuse the similarity layer according to the preset fusion parameters to obtain the fused similarity layer, and obtain the compressed model based on each of the fused similarity layers.

[0029] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0030] Based on the diffusion kernel algorithm, low-dimensional transformation processing is performed on the input data, the target layer and the activation function of the next layer of the target layer to obtain a transformed feature vector, and the model includes the target layer and the next layer of the target layer;

[0031] Based on a preset inter-layer similarity algorithm and the converted feature vector, construct a similarity matrix between the target layer and a layer below the target layer;

[0032] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer, and a compressed large language model is obtained based on each of the fused similarity layers.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0034] Based on the diffusion kernel algorithm, low-dimensional transformation processing is performed on the input data, the target layer and the activation function of the next layer of the target layer to obtain a transformed feature vector, and the model includes the target layer and the next layer of the target layer;

[0035] Based on a preset inter-layer similarity algorithm and the converted feature vector, construct a similarity matrix between the target layer and a layer below the target layer;

[0036] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer, and a compressed large language model is obtained based on each of the fused similarity layers.

[0037] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0038] Based on the diffusion kernel algorithm, low-dimensional transformation processing is performed on the input data, the target layer and the activation function of the next layer of the target layer to obtain a transformed feature vector, and the model includes the target layer and the next layer of the target layer;

[0039] Based on a preset inter-layer similarity algorithm and the converted feature vector, construct a similarity matrix between the target layer and a layer below the target layer;

[0040] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer, and a compressed large language model is obtained based on each of the fused similarity layers.

[0041] The above-mentioned model processing method, device, computer equipment, readable storage medium and program product calculate the converted feature vector through the diffusion kernel algorithm, calculate the similarity matrix of the target layer and the layer below the target layer according to the converted feature vector and the preset inter-layer similarity algorithm, fuse the similar layers based on the similarity matrix, obtain the compressed model based on the fused similar layers, extract the model information through the streaming embedding technology, realize the inter-layer fusion and merging, and effectively compress the model. It can reduce the size of the model while ensuring the performance of the model, and at the same time make better use of the redundant information in the model to realize the efficient compression of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a flow chart of a model processing method in one embodiment;

[0044] Figure 2 A schematic diagram of a flow chart of a model processing method in one embodiment;

[0045] Figure 3 A schematic diagram of a flow chart of a model processing method in one embodiment;

[0046] Figure 4 A schematic diagram of a flow chart of a model processing method in one embodiment;

[0047] Figure 5 is a structural block diagram of a model processing device in an embodiment;

[0048] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] In an exemplary embodiment, Figure 1 As shown, a model processing method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0051] Step 101: Based on the diffusion kernel algorithm, low-dimensional transformation processing is performed on the input data, the target layer, and the activation function of the next layer of the target layer to obtain a transformed feature vector.

[0052] The diffusion kernel algorithm is used to convert the high-dimensional layer into a low-dimensional expression. The input data can be natural language text, images, etc. The input data, the function of the target layer, and the activation function of the next layer of the target layer can all be high-dimensional data. The model can be a large language model, which includes at least multiple target layers, and each target layer is arranged in sequence. The target layers included in the large language model can be neural network layers, such as multiple convolutional layers, multiple pooling layers, etc.; the terminal can determine the next layer of the target layer according to the arrangement order of each target layer.

[0053] Specifically, the target layer can be any one of the multiple neural network layers included in the large language model. After determining the target layer, the terminal can perform a low-dimensional conversion process on the target layer based on the diffusion kernel algorithm, input data, the activation function of the target layer, and the activation function of the next layer of the target layer to obtain a converted feature vector of the target layer.

[0054] Step 102: construct a similarity matrix between the target layer and the next layer of the target layer based on a preset inter-layer similarity algorithm and the converted feature vector.

[0055] The preset inter-layer similarity algorithm may be a normalized pairwise information similarity measurement algorithm. The similarity matrix may represent the similarity between the target layer and the next layer of the target layer.

[0056] Specifically, the terminal may calculate a similarity matrix between a target layer and a layer below the target layer by normalizing an information similarity measurement algorithm, a converted feature vector of the target layer, and a converted feature vector of a layer below the target layer.

[0057] Step 103: determine similar layers in the similarity matrix, and fuse the similar layers according to preset fusion parameters to obtain fused similar layers, and obtain a compressed model based on each fused similar layer.

[0058] The fused similarity layer may be a target layer and a next layer of the target layer whose similarity matrix satisfies a preset similarity condition.

[0059] Specifically, the terminal can determine the relationship between the similarity matrix and the preset similarity condition. The terminal can determine the target layer and the next layer of the target layer that meet the preset similarity condition as similar layers. The terminal can fuse the similar layers according to the preset fusion parameters to obtain the fused similar layers of the target layer and the next layer of the target layer. The terminal can calculate the preset fusion parameters based on the parameters of the same position of the target layer and the next layer of the target layer and the parameters when the mth compression is performed. The specific formula for calculating the preset fusion parameters can be:

[0060]

[0061] Among them, θ m is the parameter of the merged position of the target layer and the next layer of the target layer; m is the coefficient for the mth compression, θ i is the parameter of the same position of the target layer and the next layer of the target layer; θ j It is the parameter of the next layer of the target layer and the same position of the target layer.

[0062] In addition, the terminal may also fuse the target layer that meets the preset similarity condition and the next layer of the target layer through a weight distribution function to obtain a fused similar layer.

[0063] The terminal can execute the method described in the above embodiment for each layer in the model as the target layer, obtain fused similar layers corresponding to multiple target layers, and obtain a compressed model based on the fused similar layers corresponding to each target layer; optionally, the terminal can also determine the fused similar layers corresponding to each target layer and the layers that are not fused as the compressed model.

[0064] The above-mentioned model processing method calculates the converted feature vector through the diffusion kernel algorithm, calculates the similarity matrix of the target layer and the layer below the target layer according to the converted feature vector and the preset inter-layer similarity algorithm, fuses the similar layers based on the similarity matrix, obtains the compressed model based on the fused similar layers, extracts the model information through the streaming embedding technology, realizes the inter-layer fusion and merging, and effectively compresses the model. It can reduce the size of the model while ensuring the performance of the model, and at the same time make better use of the redundant information in the model to realize the efficient compression of the model.

[0065] In an exemplary embodiment, step 101 specifically includes:

[0066] A kernel matrix is ​​constructed based on input data, a target layer, and an activation function and a kernel bandwidth parameter of a next layer of the target layer, and a degree matrix is ​​constructed based on elements of the kernel matrix; a difference between the degree matrix and the kernel matrix is ​​determined as a Laplace matrix; an eigenvector corresponding to a target eigenvalue of the Laplace matrix is ​​determined; and a product of the eigenvector and the Laplace matrix is ​​determined as a transformed eigenvector.

[0067] Specifically, the terminal constructs a kernel matrix based on the input data, the activation function of the target layer and the next layer of the target layer, and the kernel bandwidth parameter; the kernel matrix can be expressed as:

[0068]

[0069] Among them, K is the kernel matrix, H i is the activation function of the target layer, H j is the activation function of the next layer of the target layer, and σ is the kernel bandwidth parameter.

[0070] The terminal can perform diagonal processing based on the elements of the kernel matrix to obtain the degree matrix. The degree matrix can be expressed as:

[0071]

[0072] The terminal may determine the difference between the degree matrix and the kernel matrix as a Laplace matrix. The terminal may determine the eigenvector corresponding to the target eigenvalue of the Laplace matrix. The terminal may determine the product of the eigenvector and the Laplace matrix as the converted eigenvector. The specific expression formula of the converted eigenvector may be:

[0073]

[0074] Where E is the transformed eigenvector; EigVectords d Refers to the eigenvectors corresponding to the d smallest eigenvalues ​​of the Laplace matrix L.

[0075] In this embodiment, by converting high-dimensional data into low-dimensional expressions, it is helpful to understand and compare the relationships at different levels.

[0076] In an exemplary embodiment, step 102 specifically includes:

[0077] Calculate the joint probability distribution function of the target layer and the layer next to the target layer, and calculate the marginal probability distribution function of the target layer and the marginal probability distribution function of the layer next to the target layer; construct a similarity matrix of the target layer and the layer next to the target layer based on the joint probability distribution function, the marginal probability distribution function of the target layer and the marginal probability distribution function of the layer next to the target layer.

[0078] Specifically, the terminal can calculate the joint probability distribution function of the target layer and the next layer of the target layer, and calculate the edge probability distribution function of the target layer and the edge probability distribution function of the next layer of the target layer. The terminal can calculate the shared information of the joint probability distribution function of the target layer and the next layer of the target layer and the edge probability distribution function of the target layer and the next layer of the target layer. The terminal can calculate the normalized value of the target layer and the next layer of the target layer, and determine the ratio of the shared information and the normalized value as the similarity matrix of the target layer and the next layer of the target layer. The specific calculation formula of the similarity matrix can be:

[0079]

[0080] Among them, S ij is the similarity matrix between the target layer and the next layer of the target layer. p(x,y) represents E i and E j The joint probability distribution of E i and E j The marginal probability distribution of E i is the transformed feature vector of the target layer, E j is the transformed feature vector of the next layer of the target layer.

[0081] In this implementation, the similarity matrix between the target layer and the next layer of the target layer is calculated by a preset inter-layer similarity algorithm, taking into account the shared information between layers and the amount of information of each layer itself, thereby improving the accuracy of the similarity calculation between layers and facilitating subsequent compression between layers.

[0082] In an exemplary embodiment, the model processing method further includes:

[0083] The sum of the multi-head attention value, the feedforward value and the layer normalization value of the activation function of the previous layer of the target layer is determined as the activation function of the target layer.

[0084] The model may also include a layer above the target layer.

[0085] Specifically, the terminal can perform a multi-head attention operation on the activation function of the previous layer of the target layer to obtain a multi-head attention value; the terminal can perform a feedforward operation on the activation function of the previous layer of the target layer to obtain a feedforward value; the terminal can add the multi-head attention value, the feedforward value and the activation function of the previous layer (residual connection) to obtain a sum value, and the terminal performs a layer normalization operation on the sum value to obtain a layer normalization value. The terminal can determine the sum of the multi-head attention value, the feedforward value and the layer normalization value of the activation function of the previous layer of the target layer as the activation function of the target layer. The specific expression formula of the activation function of the target layer can be:

[0086]

[0087] Among them, H l represents the activation of layer l, MultiHead(H l −1) can be the multi-head attention value of the activation function of the previous layer. LayerNorm(H l −1) can be the layer normalization value of the activation function of the previous layer. FeedForward(H l −1) can be the feedforward value of the activation function of the previous layer. The activation function represents the output results of each layer of the model when processing input data.

[0088] In addition, if the target layer is the first layer of the model, the terminal can calculate the weight value and bias value of the first layer, and the terminal can calculate the linear change value of the first layer based on the weight value, bias value and input data of the first layer. The terminal can calculate the activation function of the first layer based on the preset activation function and the linear change value of the first layer. The preset activation function can be a rectified linear unit (Rectified Linear Unit, ReLU), etc., which is not specifically limited here.

[0089] In this embodiment, by calculating the activation function of each layer, it is convenient to compress and merge the layers later.

[0090] In an exemplary embodiment, the model processing method further includes:

[0091] Calculate the performance parameters of the compressed model; if the performance parameters do not meet the preset performance conditions, continue to execute the steps of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the converted feature vector until the performance parameters meet the preset performance conditions.

[0092] The performance parameter may be a parameter reflecting the performance of the large language model, such as a running speed, resource consumption, etc. The preset performance condition may be a predetermined compression target or a performance threshold.

[0093] Specifically, the terminal can set a preset performance condition according to the model performance condition specified by the user or the performance threshold of the model itself. The terminal can calculate the performance parameters of the model after each compression. The terminal can compare the performance parameters with the preset performance conditions. If the performance parameters of the currently compressed model exceed the parameter range set by the preset performance conditions, it is determined that the performance parameters do not meet the preset performance conditions. The terminal compresses the model and continues to execute from step 102 until the compressed performance parameters meet the preset performance conditions. If the terminal calculates that the performance parameters of the currently compressed model are within the parameter range set by the preset performance conditions, it is determined that the performance parameters meet the preset performance conditions. The terminal terminates the compression of the model and performs real-time dialogue processing or other processing based on the compressed model.

[0094] In addition, the terminal may receive a termination instruction and terminate the compression of the model based on the termination instruction.

[0095] In this embodiment, by determining whether the model meets preset performance conditions, the model performance is continuously monitored during the compression process, and the optimal balance between the compression ratio and the performance is found, thereby ensuring the performance of the model.

[0096] In an exemplary embodiment, the model processing method further includes:

[0097] An initial inference tree of the model is constructed based on a preset inference tree inference algorithm. For each target node, the product value between the confidences of the nodes included in the path from the root node to the target node is calculated, and the product value is determined as the node value function of the target node; based on the node value function, the depth value of the target node and the hyperparameters for adjusting the influence of the depth, the modified node value function of the target node is determined; based on the modified node value function, the path from the root node to the target node is rearranged to obtain the rearranged path; based on the rearranged paths corresponding to each target node, the target inference tree is obtained.

[0098] The preset inference tree inference algorithm may be an adaptive dynamic inference tree inference acceleration algorithm. The initial inference tree includes at least one target node. The path from the root node to the target node may be a draft token path. The confidence may be the prediction probability of the model for the draft token.

[0099] Specifically, the terminal can determine the root node and at least one target node and one or more paths from the root node to each target node based on a preset inference tree inference algorithm. The terminal determines the root node, one target node, and one or more paths as the initial inference tree. For each target node, the terminal can calculate the product value between the confidences of each node included in the path from the root node to the target node, and determine the product value as the node value function of the target node. The specific calculation formula of the node value function can be:

[0100]

[0101] Among them, Value(t i ) can be the node value function of the target node, Path(root,t i ) represents the path from the root node to the target node, c j Indicates the confidence of the nodes from the root node to the target node.

[0102] The terminal may determine a modified node value function of the target node based on the node value function, the depth value of the target node, and the hyperparameters for adjusting the depth effect. The specific formula of the modified node value function of the target node may be:

[0103]

[0104] Among them, Value * (t i ) can be the modified node value function depth(t i ) represents the depth of the target node, and α is a hyperparameter for adjusting the influence of depth.

[0105] The terminal can rearrange the path from the root node to the target node based on the modified node value function to obtain a rearranged path; and obtain a target reasoning tree based on the rearranged paths corresponding to each target node.

[0106] In addition, in order to ensure that the generation process is consistent with the original autoregressive decoding, the terminal can determine all the ancestor nodes of each target node. If the current node is the ancestor node of the target node, the mask value of the current node is set to 1. If the current node is not the ancestor node of the target node, the mask value of the current node is set to 0. The mask value of 1 indicates that when the target node is processed, the current node can be paid attention to. The mask value of 1 indicates that when the target node is processed, the current node cannot be paid attention to. The calculation formula of mask M can be:

[0107]

[0108] Among them, Ancestors(i) represents the set of all ancestor nodes of node i.

[0109] In this embodiment, the token acceptance rate is improved by dynamically adjusting the inference tree structure, thereby accelerating the inference process of the large language model. It ensures that each token can only notice its ancestor node in the tree structure, thereby maintaining the consistency of the generation process. While ensuring the generation quality, the inference speed of the large language model is significantly improved. The adaptive dynamic inference tree structure is introduced to optimize the traditional autoregressive decoding process through steps such as expansion, rearrangement and attention mask adjustment to improve the inference efficiency.

[0110] In an exemplary embodiment, Figure 2 As shown, the model processing method specifically includes the following steps:

[0111] Step 201: The terminal extracts the activation function of each layer in the large language model.

[0112] Step 202: The terminal performs low-dimensional transformation processing on the input data, the target layer, and the activation function of the next layer of the target layer based on the diffusion kernel algorithm to obtain a transformed feature vector.

[0113] Step 203: The terminal constructs a similarity matrix between the target layer and the next layer of the target layer based on the normalized pairwise information similarity measurement algorithm and the converted feature vectors.

[0114] Step 204: The terminal selects similar layers of the target layer and the next layer of the target layer according to the similarity matrix, and fuses the similar layers according to preset fusion parameters to obtain fused similar layers, and obtains a compressed model based on each fused similar layer.

[0115] Step 205: The terminal calculates the performance parameters of the compressed model, and determines whether the performance parameters meet the preset performance conditions. If the performance parameters of the compressed model do not meet the preset performance conditions, continue to execute step 203.

[0116] Step 206: If the performance parameters of the compressed model meet the preset performance conditions, the compressed model is output.

[0117] In this embodiment, by mapping the model to a low-dimensional manifold space, the essential relationship and structure between the parameters are effectively captured. This makes the model compression process more accurate and efficient. Especially when processing very large-scale language models, the manifold embedding technology can better maintain the performance and expressiveness of the model while reducing the model parameters, thereby effectively improving the quality and efficiency of the compressed model. In the model compression stage, similar layers are merged by analyzing the similarity between layers in the manifold space. By constructing a similarity matrix and selecting the most similar layer pairs for merging, the compressed model structure is more compact and efficient, especially when processing deep networks, the key features of the model can be more accurately retained.

[0118] In an exemplary embodiment, Figure 3 As shown, the model processing method specifically includes the following steps:

[0119] Step 301: The terminal constructs an initial inference tree of the model based on a preset inference tree inference algorithm.

[0120] Step 302: The terminal calculates, for each target node, the product value of the confidences of the nodes included in the path from the root node to the target node, and determines the product value as the node value function of the target node.

[0121] Step 303: The terminal determines a modified node value function of the target node based on the node value function, the depth value of the target node, and a hyperparameter for adjusting the depth influence.

[0122] Step 304: The terminal rearranges the path from the root node to the target node based on the modified node value function to obtain a rearranged path.

[0123] Step 305: The terminal obtains a target inference tree based on the rearranged paths corresponding to each target node.

[0124] Step 306: The terminal adjusts the attention mask based on whether the current node belongs to the ancestor node of the target node, and sets the attention mask value of the current node that belongs to the ancestor node of the target node to 1, otherwise it is set to 0.

[0125] In this embodiment, the token acceptance rate is improved by dynamically adjusting the inference tree structure, thereby accelerating the inference process of the large language model. It mainly includes two core steps: the expansion stage and the rearrangement stage, and efficient inference acceleration is achieved by optimizing these two stages. By constructing a dynamic inference tree structure and combining steps such as expansion, rearrangement, and attention mask adjustment, the inference efficiency of the model is significantly improved, so that the large language model can process multiple possible output paths in parallel during the generation process, greatly reducing the inference delay. Compared with the traditional autoregressive decoding method, the adaptive dynamic inference tree algorithm has significant advantages in improving the inference speed, and is particularly suitable for application scenarios that require real-time response.

[0126] In an exemplary embodiment, Figure 4 As shown, the model processing method can specifically include two parts: a knowledge extraction and layer merging compression part based on manifold embedding and a reasoning acceleration part based on an adaptive dynamic inference tree, wherein the knowledge extraction and layer merging compression part based on manifold embedding includes manifold embedding knowledge extraction and similarity-based layer merging. Manifold embedding knowledge extraction may include extracting layer activation and diffusion kernel algorithm; similarity-based layer merging may include constructing a similarity matrix and merging similar layers. The reasoning acceleration part based on an adaptive dynamic inference tree may include reasoning tree expansion, tree node rearrangement, and attention mask adjustment.

[0127] In this embodiment, by combining the model compression based on manifold embedding with the adaptive dynamic inference tree inference acceleration algorithm, not only the model size is significantly reduced, but also the inference speed is greatly improved to maintain the performance of the model. Manifold embedding technology realizes efficient model compression, and the adaptive dynamic inference tree structure greatly improves the inference efficiency, achieving the technical effect of significantly compressing the model size and accelerating the inference process, and providing new solutions for related applications in the field of natural language processing, especially the deployment and application of large models in resource-constrained environments. It is particularly suitable for application scenarios that require the deployment of large-scale language models in resource-constrained environments. This application can be applied to application scenarios of fast-response real-time text generation and dialogue systems.

[0128] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a model processing device for implementing the model processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more model processing device embodiments provided below can refer to the limitations on the model processing method above, and will not be repeated here.

[0130] In an exemplary embodiment, Figure 5 As shown, a model processing device 50 is provided, comprising: a processing module 51, a construction module 52 and a fusion module 53, wherein:

[0131] A processing module 51 is used to perform low-dimensional transformation processing on the input data, the target layer and the activation function of the next layer of the target layer based on the diffusion kernel algorithm to obtain a transformed feature vector, and the model includes the target layer and the next layer of the target layer;

[0132] A construction module 52, configured to construct a similarity matrix between a target layer and a layer below the target layer based on a preset inter-layer similarity algorithm and the converted feature vector;

[0133] The fusion module 53 is used to determine similar layers in the similarity matrix, fuse the similar layers according to preset fusion parameters to obtain fused similar layers, and obtain a compressed model based on each fused similar layer.

[0134] In one embodiment, the processing module 51 is used to construct a kernel matrix based on input data, the activation function of the target layer and the next layer of the target layer, and the kernel bandwidth parameter, and construct a degree matrix based on the elements of the kernel matrix;

[0135] Determine the difference between the degree matrix and the kernel matrix as the Laplacian matrix;

[0136] Determine the eigenvector corresponding to the target eigenvalue of the Laplacian matrix;

[0137] The product of the eigenvector and the Laplacian matrix is ​​determined as the transformed eigenvector.

[0138] In one embodiment, the construction module 52 is used to calculate the joint probability distribution function of the target layer and the layer below the target layer, and calculate the marginal probability distribution function of the target layer and the marginal probability distribution function of the layer below the target layer;

[0139] Based on the joint probability distribution function, the marginal probability distribution function of the target layer and the marginal probability distribution function of the layer below the target layer, a similarity matrix of the target layer and the layer below the target layer is constructed.

[0140] In one of the embodiments, the processing module 51 is further used to determine the sum of the multi-head attention value, the feedforward value and the layer normalization value of the activation function of the previous layer of the target layer as the activation function of the target layer.

[0141] In one embodiment, the fusion module 53 is also used to calculate the performance parameters of the compressed model;

[0142] If the performance parameter does not meet the preset performance condition, continue to execute the step of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the converted feature vector until the performance parameter meets the preset performance condition.

[0143] In one embodiment, the fusion module 53 is further used to construct an initial inference tree of the model based on a preset inference tree inference algorithm, and the initial inference tree includes at least one target node;

[0144] For each target node, the product value between the confidences of the nodes included in the path from the root node to the target node is calculated, and the product value is determined as the node value function of the target node; based on the node value function, the depth value of the target node and the hyperparameter for adjusting the depth effect, the modified node value function of the target node is determined; based on the modified node value function, the path from the root node to the target node is rearranged to obtain the rearranged path;

[0145] Based on the rearranged paths corresponding to each target node, a target reasoning tree is obtained.

[0146] Each module in the above model device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0147] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a model processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0148] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0149] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0150] Based on the diffusion kernel algorithm, the input data, the target layer and the activation function of the next layer of the target layer are transformed into a low-dimensional form to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0151] Based on the preset inter-layer similarity algorithm and the converted feature vectors, a similarity matrix between the target layer and the next layer of the target layer is constructed;

[0152] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer, and a compressed model is obtained based on each fused similarity layer.

[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0154] A kernel matrix is ​​constructed based on input data, the activation function of the target layer and the next layer of the target layer, and a kernel bandwidth parameter, and a degree matrix is ​​constructed based on the elements of the kernel matrix;

[0155] Determine the difference between the degree matrix and the kernel matrix as the Laplacian matrix;

[0156] Determine the eigenvector corresponding to the target eigenvalue of the Laplacian matrix;

[0157] The product of the eigenvector and the Laplacian matrix is ​​determined as the transformed eigenvector.

[0158] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0159] Calculate the joint probability distribution function of the target layer and the layer below the target layer, and calculate the marginal probability distribution function of the target layer and the marginal probability distribution function of the layer below the target layer;

[0160] Based on the joint probability distribution function, the marginal probability distribution function of the target layer and the marginal probability distribution function of the layer below the target layer, a similarity matrix of the target layer and the layer below the target layer is constructed.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] The sum of the multi-head attention value, the feedforward value and the layer normalization value of the activation function of the previous layer of the target layer is determined as the activation function of the target layer.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] Calculate the performance parameters of the compressed model;

[0165] If the performance parameter does not meet the preset performance condition, continue to execute the step of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the converted feature vector until the performance parameter meets the preset performance condition.

[0166] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0167] Constructing an initial inference tree of the model based on a preset inference tree inference algorithm, wherein the initial inference tree includes at least one target node;

[0168] For each target node, the product value between the confidences of the nodes included in the path from the root node to the target node is calculated, and the product value is determined as the node value function of the target node; based on the node value function, the depth value of the target node and the hyperparameter for adjusting the depth effect, the modified node value function of the target node is determined; based on the modified node value function, the path from the root node to the target node is rearranged to obtain the rearranged path;

[0169] Based on the rearranged paths corresponding to each target node, a target reasoning tree is obtained.

[0170] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0171] Based on the diffusion kernel algorithm, the input data, the target layer and the activation function of the next layer of the target layer are transformed into a low-dimensional form to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0172] Based on the preset inter-layer similarity algorithm and the converted feature vectors, a similarity matrix between the target layer and the next layer of the target layer is constructed;

[0173] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer, and a compressed model is obtained based on each fused similarity layer.

[0174] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0175] Based on the diffusion kernel algorithm, the input data, the target layer and the activation function of the next layer of the target layer are transformed into a low-dimensional form to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0176] Based on the preset inter-layer similarity algorithm and the converted feature vectors, a similarity matrix between the target layer and the next layer of the target layer is constructed;

[0177] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer, and a compressed model is obtained based on each fused similarity layer.

[0178] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0180] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A model processing method, characterized in that: The method comprises: Based on the diffusion kernel algorithm, low-dimensional transformation processing is performed on the input data, the target layer and the activation function of the next layer of the target layer to obtain a transformed feature vector, and the model includes the target layer and the next layer of the target layer; Based on a preset inter-layer similarity algorithm and the converted feature vector, construct a similarity matrix between the target layer and a layer below the target layer; A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer, and a compressed model is obtained based on each of the fused similarity layers.

2. The method according to claim 1, characterized in that The low-dimensional transformation is performed based on the diffusion kernel algorithm and the activation function of the input data, the target layer, and the next layer of the target layer to obtain the transformed feature vector, including: Constructing a kernel matrix based on input data, activation functions of a target layer and a layer below the target layer, and kernel bandwidth parameters, and constructing a degree matrix based on elements of the kernel matrix; Determining the difference between the degree matrix and the kernel matrix as a Laplacian matrix; Determine an eigenvector corresponding to a target eigenvalue of the Laplace matrix; The product of the eigenvector and the Laplacian matrix is ​​determined as the transformed eigenvector.

3. The method according to claim 1, characterized in that The constructing a similarity matrix between the target layer and a layer next to the target layer based on a preset inter-layer similarity algorithm and the converted feature vectors includes: Calculating a joint probability distribution function of the target layer and a layer below the target layer, and calculating an edge probability distribution function of the target layer and an edge probability distribution function of a layer below the target layer; Based on the joint probability distribution function, the marginal probability distribution function of the target layer and the marginal probability distribution function of the next layer of the target layer, a similarity matrix of the target layer and the next layer of the target layer is constructed.

4. The method according to claim 1, characterized in that: The method further comprises: The sum of the multi-head attention value, the feedforward value and the layer normalization value of the activation function of the previous layer of the target layer is determined as the activation function of the target layer.

5. The method according to claim 1, characterized in that The method further comprises: Calculating performance parameters of the compressed model; If the performance parameter does not meet the preset performance condition, continue to execute the step of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the converted feature vector until the performance parameter meets the preset performance condition.

6. The method according to claim 1, characterized in that The method further comprises: Constructing an initial inference tree of the model based on a preset inference tree inference algorithm, wherein the initial inference tree includes at least one target node; For each of the target nodes, calculate the product value between the confidences of the nodes included in the path from the root node to the target node, and determine the product value as the node value function of the target node; determine a modified node value function of the target node based on the node value function, the depth value of the target node, and a hyperparameter for adjusting the depth effect; and rearrange the path from the root node to the target node based on the modified node value function to obtain a rearranged path; A target inference tree is obtained based on the rearranged paths corresponding to each of the target nodes.

7. A model processing device, characterized in that: The device comprises: A processing module, for performing low-dimensional transformation processing on the input data, the target layer and the activation function of the next layer of the target layer based on a diffusion kernel algorithm to obtain a transformed feature vector, wherein the model includes the target layer and the next layer of the target layer; A construction module, used for constructing a similarity matrix between the target layer and a layer below the target layer based on a preset inter-layer similarity algorithm and the converted feature vector; The fusion module is used to determine the similarity layer in the similarity matrix, and fuse the similarity layer according to the preset fusion parameters to obtain the fused similarity layer, and obtain the compressed model based on each of the fused similarity layers.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images