Model processing method and device, computer, storage medium and program product

By introducing business hidden vectors as intermediate states in the large language model, the video memory space occupied by key-value cache is reduced, the problem of excessive KV-Cache memory space is solved, and the model deployment efficiency and performance are improved.

CN119005264AActive Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411126490.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-22
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

The memory space occupied by key-value cache (KV-Cache) in large language model (LLM) is too large, resulting in low model deployment efficiency. If the input data is reduced to reduce the memory space, data damage and model performance will be degraded. .

Method used

By introducing a service hidden vector as an intermediate state, a projection matrix is ​​used to dimensionally compress the service hidden state, and convert it into a key vector and a value vector when needed, and only the intermediate state is cached, thereby reducing the cache requirement of key-value states. Reduce the memory space usage.

Benefits of technology

The memory usage is reduced to less than half of the original, improving model deployment efficiency and performance, reducing deployment costs, and avoiding data damage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005264A_ABST
    Figure CN119005264A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a model processing method and device, a computer, a storage medium and a program product, and the method comprises the steps: in a first business processing network, adding a business hidden vector obtained after compressing a first business hidden state of business data through a first projection matrix into a key value cache; carrying out attention processing on a first key vector and a first value vector after the business hidden vector is converted through a second projection matrix to obtain a second business hidden state; transmitting cross-layer transfer parameters in the first key vector, the first value vector and the service hidden vector to a second service processing network, and performing attention processing on a second key vector and a second value vector determined through the cross-layer transfer parameters based on a second service hidden state to obtain a third service hidden state; and performing information restoration on the service hiding state obtained by the last service processing network to obtain a service processing result. By adopting the method, the model video memory occupation amount can be reduced, and the model deployment efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model processing method, device, computer, storage medium and program product. Background Art

[0002] The current Large Language Model (LLM) is generally implemented based on the Transformer architecture, especially the Transformer in the decoder mode. In the process of autoregressive generation, in order to avoid repeated calculations in the model, the model will save some of the previous intermediate calculation results, namely the Key-Value Cache (KV-Cache). Among them, the size of the video memory space occupied by KV-Cache is mainly determined by several factors, which can be recorded as Sp = 2LSH, Sp is used to represent the size of the video memory space occupied by KV-Cache, L represents the number of Transformers included in the current LLM, S is used to represent the length of the vector elements included in the sample, H is used to represent the size of the KV state, and 2 is used to represent the key vector (key, K) and the value vector (value, V). The model scale of LLM is very large, and with the advent of the long text era, S is also getting larger and larger, which also leads to the video memory space occupied by KV-Cache (i.e., the proportion of memory) is getting larger and larger, which makes the batch size (batchsize) when the model is deployed limited, and then leads to low model deployment efficiency. Therefore, it is extremely important to reduce the size of the video memory space occupied by KV-Cache. At present, some unimportant vector elements in the input data are generally discarded, or multiple vector elements are merged to reduce the input length of the model. However, the input data of the model is the original attribute of the sample. The complete input data contains the complete information of the sample. Processing the vector elements in the input data will cause data loss, thereby reducing model performance. Summary of the invention

[0003] The embodiments of the present application provide a model processing method, device, computer, storage medium and program product, which can reduce the model memory usage, improve the model deployment efficiency, reduce the model deployment cost, and improve the model performance.

[0004] On the one hand, an embodiment of the present application provides a model processing method, the method comprising:

[0005] In the first service processing network, a first service hidden state corresponding to the service data is obtained, a first projection matrix is ​​used to perform dimension compression on the first service hidden state to obtain a service hidden vector, and the service hidden vector is added to the key-value cache;

[0006] Obtaining a business latent vector from the key-value cache, performing key-value conversion on the business latent vector using a second projection matrix to obtain a first key vector and a first value vector, performing attention processing on the first key vector and the first value vector to obtain a second business hidden state;

[0007] Obtaining a cross-layer transfer parameter from the first key vector, the first value vector and the service latent vector, and transmitting the cross-layer transfer parameter to the second service processing network;

[0008] In the second service processing network, a second key vector and a second value vector are determined based on the cross-layer transfer parameter, and attention processing is performed on the second key vector and the second value vector based on the second service hidden state to obtain a third service hidden state;

[0009] When the second service processing network is the last service processing network, information restoration is performed on the third service hidden state to obtain a service processing result for the service data.

[0010] On the one hand, an embodiment of the present application provides a model processing method, the method comprising:

[0011] In a first initial processing network of the initial processing model, a first sample hidden state corresponding to the business sample is obtained, a first initial projection matrix is ​​used to perform dimension compression on the first sample hidden state to obtain a sample latent vector, and the sample latent vector is added to a sample key value cache;

[0012] Obtain a sample latent vector from the sample key-value cache, perform key-value conversion on the sample latent vector using the second initial projection matrix to obtain a first sample key vector and a first sample value vector, perform attention processing on the first sample key vector and the first sample value vector to obtain a second sample hidden state;

[0013] Obtaining a sample transfer parameter from the first sample key vector, the first sample value vector, and the sample latent vector, and transferring the sample transfer parameter to a second initial processing network in the initial processing model;

[0014] In the second initial processing network, a second sample key vector and a second sample value vector are determined based on the sample transfer parameter, and based on the second sample hidden state, attention processing is performed on the second sample key vector and the second sample value vector to obtain a third sample hidden state;

[0015] When the second initial processing network is the last initial processing network, information restoration is performed on the hidden state of the third sample to obtain a sample processing result for the business sample;

[0016] The parameters of the initial processing model are adjusted based on the sample processing results to obtain a business processing model after parameter convergence; the business processing model includes a first projection matrix corresponding to the first initial projection matrix and a second projection matrix corresponding to the second initial projection matrix.

[0017] In one aspect, an embodiment of the present application provides a model processing device, the device comprising:

[0018] A parameter compression module, used to obtain, in the first service processing network, a first service hidden state corresponding to the service data, perform dimension compression on the first service hidden state using a first projection matrix to obtain a service hidden vector, and add the service hidden vector to a key-value cache;

[0019] A key-value processing module is used to obtain a business latent vector from a key-value cache, perform key-value conversion on the business latent vector using a second projection matrix to obtain a first key vector and a first value vector, perform attention processing on the first key vector and the first value vector, and obtain a second business hidden state;

[0020] A cross-layer transfer module, used to obtain a cross-layer transfer parameter from the first key vector, the first value vector and the service hidden vector, and transfer the cross-layer transfer parameter to the second service processing network;

[0021] The key-value processing module is further used to determine, in the second service processing network, a second key vector and a second value vector based on the cross-layer transfer parameter, and perform attention processing on the second key vector and the second value vector based on the second service hidden state to obtain a third service hidden state;

[0022] The service prediction module is used to restore information of the third service hidden state when the second service processing network is the last service processing network, so as to obtain a service processing result for the service data.

[0023] When the first projection matrix is ​​used to perform dimension compression on the first service hidden state to obtain the service hidden vector, the parameter compression module can be used to:

[0024] Using a first projection matrix to perform dimension compression on a first service hidden state to obtain a first latent vector;

[0025] Quantize and compress the first latent vector to obtain a business latent vector;

[0026] When the second projection matrix is ​​used to perform key-value conversion on the business latent vector to obtain the first key vector and the first value vector, the key-value processing module can be used to:

[0027] Quantize and restore the business latent vector to obtain a second latent vector;

[0028] The product of the first key projection matrix and the second latent vector in the second projection matrix is ​​determined as the first key vector, and the product of the first value projection matrix and the second latent vector in the second projection matrix is ​​determined as the first value vector.

[0029] Wherein, when the first latent vector is quantized and compressed to obtain the service latent vector, the parameter compression module can be used to:

[0030] The first latent vector is grouped to obtain N vector groups; N is a positive integer; each vector group includes vector elements;

[0031] Obtaining a quantization level, obtaining vector statistics corresponding to each of the N vector groups, and determining quantization parameters corresponding to each of the N vector groups based on the quantization level and the vector statistics corresponding to each of the N vector groups;

[0032] The quantization parameter corresponding to each vector group is used to compress the vector elements included in the vector group to obtain service latent vectors corresponding to the N vector groups respectively.

[0033] When the business latent vector is quantized and restored to obtain the second latent vector, the key value processing module can be used to:

[0034] Obtaining the service latent vectors and quantization parameters corresponding to the N vector groups respectively, using the quantization parameters corresponding to each vector group, performing quantization restoration processing on the service latent vectors corresponding to the vector group, and obtaining the restored latent vectors corresponding to the N vector groups respectively;

[0035] The restored latent vectors corresponding to the N vector groups are merged to obtain the second latent vector.

[0036] When performing attention processing on the first key vector and the first value vector to obtain the second service hidden state, the key value processing module can be used to:

[0037] Determine the product of the first query matrix and the first service hidden state as a first query vector;

[0038] Obtaining a vector dimension of the first query vector, and performing feature fusion on the first query vector and the first key vector based on the vector dimension to obtain vector attention;

[0039] Vector attention is used to perform weighted processing on the first value vector to obtain a second service hidden state.

[0040] The first service hidden state includes S first service sub-vectors, where S is a positive integer;

[0041] The key value processing module can be used for:

[0042] If i is 1, the i-th business latent vector is obtained from the key-value cache, the i-th business latent vector is converted into a key value using the second projection matrix to obtain the i-th first key vector and the i-th first value vector, and the i-th first key vector and the i-th first value vector are subjected to attention processing to obtain the i-th second business sub-vector; i is used to indicate the position of the i-th first business sub-vector in the S first business sub-vectors;

[0043] If i is a positive integer greater than 1 and less than or equal to S, then obtain the first query vector of the i-th first business sub-vector, obtain the business latent vectors corresponding to the first first business sub-vector to the i-th first business sub-vector from the key-value cache, use the second projection matrix to perform key-value conversion on the first business latent vector to the i-th business latent vector, and obtain the first key vectors and first value vectors corresponding to the first first business sub-vector to the i-th first business sub-vector, respectively; according to the first key vectors corresponding to the first first business sub-vector to the i-th first business sub-vector and the first query vector corresponding to the i-th first business sub-vector, determine the sub-attentions corresponding to the first first business sub-vector to the i-th first business sub-vector, use the sub-attentions corresponding to the first first business sub-vector to the i-th first business sub-vector, respectively, and perform weighted processing on the first value vectors corresponding to the first first business sub-vector to the i-th first business sub-vector, respectively, to obtain the i-th second business sub-vector;

[0044] When i is S, the first second service sub-vector to the Sth second service sub-vector form a second service hidden state.

[0045] Among them, when obtaining the cross-layer transfer parameter from the first key vector, the first value vector and the service hidden vector, the cross-layer transfer module can be used to:

[0046] Determining a service latent vector as a cross-layer transfer parameter;

[0047] In the second service processing network, when determining the second key vector and the second value vector based on the cross-layer transfer parameter, the key value processing module may be used to:

[0048] In the second service processing network, the third projection matrix in the second service processing network is used to perform key-value conversion on the cross-layer transfer parameters to obtain a second key vector and a second value vector.

[0049] Among them, when obtaining the cross-layer transfer parameter from the first key vector, the first value vector and the service hidden vector, the cross-layer transfer module can be used to:

[0050] determining the first key vector and the first value vector as cross-layer transfer parameters;

[0051] In the second service processing network, when determining the second key vector and the second value vector based on the cross-layer transfer parameter, the key value processing module may be used to:

[0052] In the second service processing network, a second key vector and a second value vector are obtained from the cross-layer transfer parameters.

[0053] When performing attention processing on the first key vector and the first value vector to obtain the second service hidden state, the key value processing module can be used to:

[0054] Acquire location-related information based on the first service hidden state, perform feature fusion on the first key vector, the first value vector, and the location-related information, and obtain full key-value data;

[0055] Determine the product of the first query matrix and the first service hidden state as a first query vector;

[0056] The first query vector is subjected to attention processing using the full amount of key-value data to obtain a second business hidden state.

[0057] The device further comprises:

[0058] A data acquisition module is used to obtain the number of business processing networks included in the business processing model and the range of parameter multiplexing layers;

[0059] A layer number determination module is used to obtain an approximate number of networks from a parameter multiplexing layer number range, and determine the approximate number of networks as the multiplexing layer number;

[0060] The network determination module is used to determine the a*d+1th business processing network in the business processing model as the first business processing network, and determine the business processing networks other than the first business processing network included in the business processing model as the second business processing network; d is the number of multiplexing layers; a is a natural number.

[0061] The device further comprises:

[0062] A storage transfer module, configured to store the key value cache corresponding to the first business object in a hardware memory if it is detected that the first business object is in an offline state; the first business object is a business object that provides business data;

[0063] The storage transfer module is also used to obtain the key value cache from the hardware memory and execute the process of obtaining the business latent vector from the key value cache when it is detected that the object state of the first business object changes from the offline state to the online state.

[0064] In one aspect, an embodiment of the present application provides a model processing device, the device comprising:

[0065] A sample compression module, used to obtain a first sample hidden state corresponding to a business sample in a first initial processing network of an initial processing model, perform dimension compression on the first sample hidden state using a first initial projection matrix to obtain a sample latent vector, and add the sample latent vector to a sample key value cache;

[0066] A sample processing module is used to obtain a sample latent vector from a sample key value cache, perform key value conversion on the sample latent vector using a second initial projection matrix to obtain a first sample key vector and a first sample value vector, perform attention processing on the first sample key vector and the first sample value vector, and obtain a second sample hidden state;

[0067] A parameter processing module, used to obtain a sample transfer parameter from the first sample key vector, the first sample value vector and the sample latent vector, and transfer the sample transfer parameter to a second initial processing network in the initial processing model;

[0068] The sample processing module is further used to determine, in the second initial processing network, a second sample key vector and a second sample value vector based on the sample transfer parameter, and perform attention processing on the second sample key vector and the second sample value vector based on the second sample hidden state to obtain a third sample hidden state;

[0069] A sample prediction module, used for restoring information of the third sample hidden state when the second initial processing network is the last initial processing network, to obtain a sample processing result for the business sample;

[0070] The model training module is used to adjust the parameters of the initial processing model based on the sample processing results to obtain a business processing model after parameter convergence; the business processing model includes a first projection matrix corresponding to the first initial projection matrix, and a second projection matrix corresponding to the second initial projection matrix.

[0071] The device further comprises:

[0072] A model acquisition module, used to acquire a basic model, and determine a first initial processing network and a second initial processing network from the basic model;

[0073] A dependency acquisition module, used to determine the network dependency and topology dependency in the basic model based on the first initial processing network, the second initial processing network and the model architecture of the basic model;

[0074] The sparse processing module is used to obtain the neurons to be optimized from the basic model based on the network dependency and the topological dependency, and perform sparse processing on the neurons to be optimized in the basic model to obtain an initial processing model.

[0075] The device further comprises:

[0076] The association acquisition module is used to obtain the basic model and the association model of the basic model, and obtain the association model parameters in the association model; the task similarity between the association model task corresponding to the association model and the basic model task corresponding to the basic model is greater than or equal to the model association threshold;

[0077] The parameter initialization module is used to initialize the basic model parameters in the basic model based on the associated model parameters to obtain an initial processing model.

[0078] On the one hand, an embodiment of the present application provides a computer device, including a processor, a memory, and an input and output interface;

[0079] The processor is connected to the memory and the input and output interface respectively, wherein the input and output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that a computer device including the processor executes the model processing method in one aspect of an embodiment of the present application.

[0080] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the model processing method in one aspect of the embodiment of the present application.

[0081] In one aspect, an embodiment of the present application provides a computer program product, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional ways in one aspect of the embodiment of the present application. In other words, when the computer instruction is executed by the processor, the method provided in various optional ways in one aspect of the embodiment of the present application is implemented.

[0082] Implementing the embodiments of the present application will have the following beneficial effects:

[0083] In an embodiment of the present application, in a first business processing network, a first business hidden state corresponding to business data is obtained, and a first projection matrix is ​​used to perform dimension compression on the first business hidden state to obtain a business hidden vector, and the business hidden vector is added to a key-value cache; the business hidden vector is obtained from the key-value cache, and a second projection matrix is ​​used to perform key-value conversion on the business hidden vector to obtain a first key vector and a first value vector, and attention processing is performed on the first key vector and the first value vector to obtain a second business hidden state; a cross-layer transfer parameter is obtained from the first key vector, the first value vector and the business hidden vector, and the cross-layer transfer parameter is passed to the second business processing network; in the second business processing network, a second key vector and a second value vector are determined based on the cross-layer transfer parameter, and based on the second business hidden state, attention processing is performed on the second key vector and the second value vector to obtain a third business hidden state; when the second business processing network is the last business processing network, information restoration is performed on the third business hidden state to obtain a business processing result for the business data. Through the above process, an intermediate state, namely the business hidden vector, is introduced between the business hidden state and the key-value state (i.e., the key vector and the value vector). There is no need to cache the key-value state, but the introduced intermediate state is cached, and the intermediate state is obtained after dimension compression of the business hidden state, so that the cache of the key-value state only needs to store one copy of data, and the data has been dimensionally compressed, so that the size of the video memory space occupied by the key-value cache is reduced to less than half of the original size, reducing the video memory occupancy of the model. The reduction in video memory occupancy can increase the batch size, thereby improving the model deployment efficiency, improving the video memory utilization, reducing the model deployment cost, and improving the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0085] Figure 1 This is a network interaction architecture diagram of a model processing provided by an embodiment of the present application;

[0086] Figure 2 It is a schematic diagram of a model processing scenario provided in an embodiment of the present application;

[0087] Figure 3 is a flow chart of a model processing method provided in an embodiment of the present application;

[0088] Figure 4 It is a schematic diagram of a model compression scenario provided in an embodiment of the present application;

[0089] Figure 5 is a flow chart of another model processing method provided in an embodiment of the present application;

[0090] Figure 6 This is a schematic diagram of a cross-layer multiplexing scenario provided by an embodiment of the present application;

[0091] Figure 7 This is a schematic diagram of a cross-layer multiplexing scenario after vector quantization provided by an embodiment of the present application;

[0092] Figure 8 It is a schematic diagram of model division provided in an embodiment of the present application;

[0093] Figure 9a This is a cross-layer multiplexing scenario provided by the embodiment of the present application. Figure 1 ;

[0094] Figure 9b This is a cross-layer multiplexing scenario provided by the embodiment of the present application. Figure 2 ;

[0095] Fig.9c This is a cross-layer multiplexing scenario provided by the embodiment of the present application. Figure 3 ;

[0096] Figure 9d This is a cross-layer multiplexing scenario provided by the embodiment of the present application. Figure 4 ;

[0097] Fig.10 It is a flow chart of another model processing method provided in an embodiment of the present application;

[0098] Fig.11 is a flow chart of another model processing method provided in an embodiment of the present application;

[0099] Fig.12 It is a flow chart of a training process method of a model processing provided in an embodiment of the present application;

[0100] Fig.13 It is a schematic diagram of the operating performance of a model provided in an embodiment of the present application;

[0101] Fig.14 is a schematic diagram of a model processing device provided in an embodiment of the present application;

[0102] Fig.15 is a schematic diagram of another model processing device provided in an embodiment of the present application;

[0103] Fig.16 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0105] Among them, if it is necessary to collect object (such as user, etc.) data in this application, a prompt interface or pop-up window will be displayed before and during the collection. The prompt interface or pop-up window is used to prompt the user that certain data is currently being collected. Only after the user confirms the prompt interface or pop-up window, the relevant steps of data acquisition are started, otherwise the process ends. Moreover, the acquired user data will be used in reasonable and legal scenarios or purposes. Optionally, in some scenarios where user data needs to be used but is not authorized by the user, authorization can be requested from the user, and the user data can be used when the authorization is passed. In other words, the use of user data in this application complies with the relevant provisions of laws and regulations.

[0106] In the examples of this application, see Figure 1 , Figure 1 is a network interaction architecture diagram of a model processing provided by an embodiment of the present application, such as Figure 1As shown, the computer device 101 can obtain business data from the local space, predict the business data to obtain the business processing result; or, the computer device 101 can obtain the business data sent by the business device (such as business device 102a, business device 102b or business device 102c, etc.), predict the business data to obtain the business processing result. Among them, when the computer device 101 predicts the business data to obtain the business processing result, it introduces an intermediate state between the business hidden state and the key value state, that is, the business hidden vector. The computer device 101 can perform dimension compression on the business hidden state to obtain the business hidden vector, cache the business hidden vector, and project the business hidden vector into a key vector and a value vector when necessary, so that the key value cache only needs to store one copy of data, and the data is obtained based on dimension compression, so that the video memory space occupied by the key value cache becomes less than half of the original, which greatly reduces the video memory space occupied by the key value cache and improves the model deployment efficiency. At the same time, the computer device 101 will reuse the business hidden vector across layers, that is, multiple business processing networks can cache the business hidden vector once, that is, cache the intermediate state introduced once, and multiple business processing networks reuse the intermediate state cached this time, so that the video memory space occupied by the key-value cache becomes 1 / d of the original, d is a positive integer, which is used to represent the number of business processing networks that reuse the same intermediate state, which also greatly reduces the video memory space occupied by the key-value cache. Among them, the key-value state is used to represent the key vector and the value vector in this application, and the intermediate state is used to represent the data to be cached, that is, the data added to the key-value cache. In this application, it is used to indicate the business hidden vector, which is the data introduced between the business hidden state and the key-value state.

[0107] For details, see Figure 2 , Figure 2 Schematic diagram of a model processing scenario provided by an embodiment of the present application. Figure 2As shown, the business processing model may include multiple business processing networks, wherein the business processing model can be considered as an LLM model, and a business processing network can be considered as a Transformer network. The LLM model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. The large language model can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc. The Transformer network is a sequence model based on the attention mechanism. Specifically, the business processing network included in the business processing model can be divided into two categories, which are respectively recorded as the first business processing network and the second business processing network. The first business processing network refers to a business processing network that caches business latent vectors, and the second business processing network refers to a business processing network that reuses the business latent vectors in the first business processing network. The computer device can obtain the first business hidden state 202 corresponding to the business data in the first business processing network 201, and the first business hidden state 202 is used to represent the input data of the first business processing network 201; the first projection matrix is ​​used to perform dimension compression on the first business hidden state 202 to obtain the business latent vector, and the business latent vector is added to the key value cache 203 to reduce the video memory usage of the key value cache. Further, the business latent vector is obtained from the key value cache 203, and the second projection matrix is ​​used to perform key-value conversion on the business latent vector to obtain the first key vector and the first value vector, and the first key vector and the first value vector are subjected to attention processing to obtain the second business hidden state, and the second business hidden state is used to represent the output data of the first business processing network. Furthermore, the computer device can obtain cross-layer transfer parameters from the first key vector, the first value vector, the business hidden vector and other data 204, and pass the cross-layer transfer parameters to the second business processing network 205; in the second business processing network 205, the second key vector and the second value vector are determined based on the cross-layer transfer parameters, so that the second business processing network 205 does not need to cache the intermediate state, and multiple business processing networks reuse the business hidden vector, thereby greatly reducing the memory usage of the key-value cache. The computer device can perform attention processing on the second key vector and the second value vector based on the second business hidden state to obtain a third business hidden state. When the second business processing network 205 is the last business processing network, the third business hidden state is restored to obtain the business processing result for the business data, and the predictive processing of the business data is realized.

[0108] It is understandable that the computer devices mentioned in the embodiments of the present application include but are not limited to terminal devices or servers. In other words, the computer device can be a server or a terminal device, or it can be a system composed of a server and a terminal device. Among them, the terminal device mentioned above can be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, PDAs, vehicle-mounted devices, augmented reality / virtual reality (Augmented Reality / Virtual Reality, AR / VR) devices, helmet displays, smart TVs, wearable devices, smart speakers, digital cameras, cameras and other mobile Internet devices (mobile internet device, MID) with network access capabilities, or terminal devices in scenarios such as trains, ships, and flights. Figure 1 As shown in , the terminal device may be a laptop computer (as shown in the service device 102b), a mobile phone (as shown in the service device 102c), or a vehicle-mounted device (as shown in the service device 102a), etc. Figure 1 Only some of the devices are listed, and optionally, the business device 102a refers to a device located in the vehicle 103, and the business device 102a can be used to run the business processing model and process the business data 1021. Among them, the server mentioned above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0109] Optionally, the data involved in the embodiments of the present application can be stored in a computer device, or the data can be stored based on cloud storage technology or a blockchain network, which is not limited here.

[0110] For further information, see Figure 3 , Figure 3 This is a flow chart of a model processing method provided in an embodiment of the present application. Figure 3 As shown, the computer device can introduce an intermediate state between the business hidden state and the key value state based on the multi-head latent vector attention (MLA) mechanism of the LLM model to compress the key value state and reduce the memory usage of the key value cache. Specifically, the model processing process includes the following steps:

[0111] Step S301, in the business processing network, obtain the business hidden state corresponding to the business data, use the first projection matrix to perform dimension compression on the business hidden state to obtain the business hidden vector, and add the business hidden vector to the key-value cache.

[0112] In the embodiments of this application, please refer to Figure 4 , Figure 4 Schematic diagram of a model compression scenario provided by an embodiment of the present application. Figure 4 As shown, the computer device can obtain the business hidden state 4011 corresponding to the business data in the business processing network 401, wherein the business hidden state corresponding to the business data can be considered as the input data of the business processing network 401. Further, the computer device can use the first projection matrix to perform dimension compression on the business hidden state 4011 to obtain a business latent vector, that is, project the business hidden state 4011 to a latent vector (latent) of a small hidden layer size, and add the business latent vector to the key-value cache 402. On the one hand, the vector is compressed, and on the other hand, the cache key vector and value vector are changed to the cache business latent vector, so that the video memory space occupied by the cache corresponding to a single vector element in the key-value cache is reduced from the size of the key vector and the value vector to the size of the business latent vector, which greatly reduces the video memory space occupied by the key-value cache. For example, the size of the key vector and the value vector are both 4096, and the size of the latent is 512, then the video memory space occupied by the cache corresponding to a single vector element in the key-value cache is reduced from 2*4096 to 512.

[0113] Step S302, obtain the business latent vector from the key-value cache, use the second projection matrix to perform key-value conversion on the business latent vector to obtain a key vector and a value vector, perform attention processing on the key vector and the value vector, and obtain the business hidden state corresponding to the business processing network.

[0114] In the embodiments of the present application, Figure 4As shown, the computer device can obtain the business hidden vector from the key-value cache 402, use the second projection matrix to perform key-value conversion on the business hidden vector to obtain a key vector and a value vector, perform attention processing on the key vector and the value vector, and obtain the business hidden state 4012 corresponding to the business processing network 401. Among them, in a business processing network, the business hidden state corresponding to the business data is the input data of the business processing network, and the business hidden state corresponding to the business processing network is the output data of the business processing network. Further, if the business processing network is not the last business processing network, the business hidden state corresponding to the business processing network can be determined as the business hidden state corresponding to the business data in the next business processing network, and return to execute step S301. If the business processing network is the last business processing network, step S303 can be executed.

[0115] Step S303: when the service processing network is the last service processing network, information restoration is performed on the service hidden state corresponding to the service processing network to obtain a service processing result for the service data.

[0116] In the embodiments of the present application, Figure 4 As shown, when the business processing network 401 is the last business processing network, the business hidden state 4012 corresponding to the business processing network 401 is restored to obtain the business processing result for the business data. The information restoration is determined based on the model processing task of the business processing model. For example, if the model processing task is a question-answering task, the information restoration can be to decode the business hidden state 4012 corresponding to the business processing network 401 to obtain the business processing result for the business data, and the business processing result is the answer to the business data.

[0117] In an embodiment of the present application, a computer device can add a first projection matrix and a second projection matrix between the input (i.e., business hidden states) and the key-value (KV) state of each layer of the Transformer to introduce an intermediate state, first using the first projection matrix to project the input to a hidden vector of a small hidden layer size, and then restore it to the KV state to be used, and when the model is deployed, the KV state is not retained, only the business latent vector (latent) is retained, thereby reducing the memory space occupied by the KV-Cache. Among them, when the model is actually deployed, it is necessary to obtain the cached latent, and then calculate the KV state, which is equivalent to using a small amount of calculation in exchange for a smaller memory usage when the model is deployed, and compressing 2H of the KV-Cache's memory space usage to the size of the latent, H is greater than the size of the latent, to achieve a reduction in the memory space occupied by the KV-Cache.

[0118] For further information, see Figure 5 , Figure 5 This is a flow chart of another model processing method provided in the embodiment of the present application. Figure 5 As shown, based on the key-value cache compression, data cross-layer multiplexing is performed. At this time, the business processing network for data caching can be recorded as the first business processing network, the business hidden state corresponding to the business data in the first business processing network can be recorded as the first business hidden state, and the business hidden state corresponding to the first business processing network can be recorded as the second business hidden state. Specifically, the model processing process includes the following steps:

[0119] Step S501: In a first business processing network, a first business hidden state corresponding to business data is obtained, a first projection matrix is ​​used to perform dimension compression on the first business hidden state to obtain a business hidden vector, and the business hidden vector is added to a key-value cache.

[0120] In an embodiment of the present application, a computer device can obtain a first business hidden state corresponding to business data in a first business processing network. When the first business processing network is the first business processing network in a business processing model, feature extraction can be performed on the business data to obtain the first business hidden state; when the first business processing network is not the first business processing network in the business processing model, output data of a previous business processing network of the first business processing network can be determined as the first business hidden state.

[0121] Furthermore, the computer device may use the first projection matrix to perform dimension compression on the first service hidden state to obtain a service latent vector. Specifically, the computer device may perform matrix multiplication of the first projection matrix (which may be denoted as W_L) and the first service hidden state to obtain a first latent vector (latent), and directly determine the first latent vector as the service latent vector. In this case, the service latent vector may be denoted as latent. For example, see Figure 6 , Figure 6 Schematic diagram of a cross-layer multiplexing scenario provided by an embodiment of the present application. Figure 6 As shown, the computer device can perform dimension compression on the first service hidden state 602 in the first service processing network 601 to obtain a service hidden vector, and add the service hidden vector to the key-value cache 603.

[0122] Alternatively, the first projection matrix may be used to perform dimension compression on the first business hidden state to obtain a first latent vector; the first latent vector may be quantized and compressed to obtain a business latent vector. In this case, the business latent vector may be denoted as Quant.

[0123] Among them, when the first latent vector is quantized and compressed to obtain the business latent vector, the computer device can group the first latent vector to obtain N vector groups; N is a positive integer; each vector group includes vector elements, which can be recorded as [A / group_size, group_size], where A is used to represent the size of the first latent vector, and group_size is used to represent the number of vector elements included in a vector group, N = A / group_size. Obtain the quantization level M, which refers to the maximum value corresponding to the quantization technology. The quantization technology can be any technology for quantizing the vector, such as int4 quantization technology, int8 quantization technology or other quantization technology. For example, when int4 quantization technology is used, the quantization level can be 7, and when int8 quantization technology is used, the quantization level can be 127, etc.; obtain the vector statistics corresponding to the N vector groups respectively, and determine the quantization parameters corresponding to the N vector groups respectively based on the quantization level and the vector statistics corresponding to the N vector groups respectively, which can be recorded as Scale j =M / |X j |, where Scale j Used to represent the quantization parameter corresponding to the jth vector group, X j Used to represent the vector statistic corresponding to the jth vector group. For example, the absolute value of the vector elements included in the jth vector group can be obtained, and the largest absolute value can be determined as the vector statistic corresponding to the jth vector group. Further, the quantization parameter corresponding to each vector group can be used to compress the vector elements included in the vector group to obtain the service latent vectors corresponding to the N vector groups. For example, the product of the vector elements included in the jth vector group and the quantization parameter corresponding to the jth vector group can be determined as the initial element quantization value of the vector elements included in the jth vector group, and the quantization value with the smallest distance from the initial element quantization value in the quantization value range corresponding to the quantization technology can be determined as the service latent vector of the vector elements included in the jth vector group. In this way, the memory space occupied by the key-value cache can be further reduced. For example, when the int4 quantization technology is used, the memory space occupied by the key-value cache can be further reduced by 4 times. For example, if the initial memory space occupied is 5120, the memory space occupied can be compressed to 512 / (5120*2*4)=1.25%. It can be seen that the memory space occupied by the key-value cache is greatly reduced.

[0124] For example, see Figure 7 , Figure 7 Schematic diagram of a cross-layer multiplexing scenario after vector quantization provided by an embodiment of the present application. Figure 7As shown, the computer device can perform dimension compression on the first business hidden state 702 in the first business processing network 701 to obtain a first latent vector; perform quantization processing on the first latent vector to obtain a business latent vector, and add the business latent vector to the key-value cache 703.

[0125] Step S502, obtain the business latent vector from the key-value cache, use the second projection matrix to perform key-value conversion on the business latent vector to obtain a first key vector and a first value vector, perform attention processing on the first key vector and the first value vector to obtain a second business hidden state.

[0126] In the embodiment of the present application, the computer device can obtain the service hidden vector from the key-value cache, such as Figure 7 , obtain the business latent vector from the key-value cache 703. The business latent vector is converted into a key value using the second projection matrix to obtain a first key vector and a first value vector. Specifically, when the business latent vector is the first latent vector, the product of the first key projection matrix (which can be recorded as W_K) in the second projection matrix and the business latent vector can be determined as the first key vector, and the product of the first value projection matrix (which can be recorded as W_V) in the second projection matrix and the business latent vector can be determined as the first value vector. Alternatively, when the business latent vector is obtained by quantizing and compressing the first latent vector, the computer device can quantize and restore the business latent vector to obtain the second latent vector; the product of the first key projection matrix and the second latent vector in the second projection matrix is ​​determined as the first key vector, and the product of the first value projection matrix and the second latent vector in the second projection matrix is ​​determined as the first value vector.

[0127] When the service latent vector is quantized and restored to obtain the second latent vector, the computer device can obtain the service latent vectors and quantization parameters corresponding to the N vector groups, respectively, and use the quantization parameters corresponding to each vector group to perform quantization and restoration processing on the service latent vector corresponding to the vector group, and obtain the restored latent vectors corresponding to the N vector groups, respectively, which can be recorded as X″ j =X′ j / Scale j , where X′ j The business latent vector used to represent the vector elements included in the jth vector group, Scale j Used to represent the quantization parameter corresponding to the jth vector group, X″ j It is used to represent the restored latent vector corresponding to the jth vector group. Further, the restored latent vectors corresponding to the N vector groups are merged to obtain the second latent vector.

[0128] Furthermore, the first key vector and the first value vector may be subjected to attention processing to obtain the second business hidden state. Specifically, the product of the first query matrix and the first business hidden state may be determined as the first query vector; the vector dimension of the first query vector is obtained, and the first query vector and the first key vector are subjected to feature fusion based on the vector dimension to obtain vector attention; the first value vector is subjected to weighted processing using vector attention to obtain the second business hidden state. Figure 7 As shown, attention processing is performed on the first key vector, the first value vector and the first query vector to obtain a second business hidden state.

[0129] Optionally, when the first key vector and the first value vector are subjected to attention processing to obtain the second business hidden state, the position-related information can be processed separately. Specifically, the position-related information can be obtained based on the first business hidden state, and the first key vector, the first value vector and the position-related information can be feature fused to obtain the full key-value data. Among them, the product of the position projection matrix (which can be denoted as W_rope) and the position-related data in the first business hidden state (which can be denoted as K_rope) can be determined as the position-related information. The product of the first query matrix and the first business hidden state is determined as the first query vector; the first query vector is subjected to attention processing using the full key-value data to obtain the second business hidden state.

[0130] The first business hidden state includes S first business sub-vectors, where S is a positive integer. If i is 1, the i-th business hidden vector is obtained from the key-value cache, and the i-th business hidden vector is converted into a key value using the second projection matrix to obtain the i-th first key vector and the i-th first value vector, and the i-th first key vector and the i-th first value vector are subjected to attention processing to obtain the i-th second business sub-vector; i is used to indicate the position of the i-th first business sub-vector in the S first business sub-vectors. If i is a positive integer greater than 1 and less than or equal to S, then obtain the first query vector of the i-th first business sub-vector, obtain the business hidden vectors corresponding to the first first business sub-vector to the i-th first business sub-vector from the key-value cache, use the second projection matrix to perform key-value conversion on the first business hidden vector to the i-th business hidden vector, and obtain the first key vector and first value vector corresponding to the first first business sub-vector to the i-th first business sub-vector; according to the first key vectors corresponding to the first first business sub-vector to the i-th first business sub-vector and the first query vector corresponding to the i-th first business sub-vector, determine the sub-attentions corresponding to the first first business sub-vector to the i-th first business sub-vector; use the sub-attentions corresponding to the first first business sub-vector to the i-th first business sub-vector to perform weighted processing on the first value vectors corresponding to the first first business sub-vector to the i-th first business sub-vector, and obtain the i-th second business sub-vector. When i is S, the first second business sub-vector to the S-th second business sub-vector form a second business hidden state.

[0131] Step S503: Obtain a cross-layer transfer parameter from the first key vector, the first value vector and the service latent vector, and transfer the cross-layer transfer parameter to the second service processing network.

[0132] In the embodiments of the present application, Figure 7 As shown, the cross-layer transfer parameters can be obtained from the first key vector, the first value vector and the service latent vector 704. Among them, in a data multiplexing method ①, the computer device can determine the service latent vector as the cross-layer transfer parameter. In a data multiplexing method ②, the computer device can determine the first key vector and the first value vector as the cross-layer transfer parameter. That is to say, the second service processing network can reuse the service latent vector of the first service processing network, or directly reuse the key vector and value vector of the first service processing network. Further, as Figure 6 In the embodiment, the computer device may transmit the cross-layer transfer parameter to the second service processing network 604, or as Figure 7 In the process, the cross-layer transfer parameters can be delivered to the second service processing network 705.

[0133] Step S504: In the second service processing network, a second key vector and a second value vector are determined based on the cross-layer transfer parameter, and attention processing is performed on the second key vector and the second value vector based on the second service hidden state to obtain a third service hidden state.

[0134] In an embodiment of the present application, in the second business processing network, the second key vector and the second value vector are determined based on the cross-layer transfer parameters. Specifically, in data multiplexing mode ①, the computer device can use the third projection matrix in the second business processing network in the second business processing network to perform key-value conversion on the cross-layer transfer parameters to obtain the second key vector and the second value vector. In data multiplexing mode ②, the computer device can obtain the second key vector and the second value vector from the cross-layer transfer parameters in the second business processing network. At this time, the second key vector is the first key vector, and the second value vector is the first value vector.

[0135] Further, if Figure 7 In step S504, the second key vector and the second value vector may be subjected to attention processing based on the second business hidden state to obtain a third business hidden state. The generation process of the third business hidden state may refer to the generation process of the second business hidden state in step S502, which will not be described in detail herein.

[0136] Step S505: when the second service processing network is the last service processing network, information restoration is performed on the hidden state of the third service to obtain a service processing result for the service data.

[0137] In the embodiment of the present application, when the second service processing network is the last service processing network, information restoration is performed on the third service hidden state to obtain a service processing result for the service data.

[0138] Optionally, KV-Cache will occupy video memory space (i.e., graphics processing unit, GPU memory for short). When the LLM model is used, context switching between different users and interruption of model use may occur. KV-Cache will continuously read and write back and forth between video memory space and other storage (such as hardware memory, etc.). Reducing the amount of video memory space occupied by KV-Cache can also improve the read and write efficiency of KV-Cache, reduce model deployment loss, and reduce model deployment costs. For example, if it is detected that the first business object is in an offline state, the key-value cache corresponding to the first business object is stored in the hardware memory; the first business object is a business object that provides business data. When it is detected that the object state of the first business object changes from an offline state to an online state, the key-value cache is obtained from the hardware memory, and the process of obtaining the business latent vector from the key-value cache is executed.

[0139] The service processing model may include multiple service processing networks, and the multiple service processing networks may be divided into a first service processing network and a second service processing network. Figure 8 , Figure 8 This is a schematic diagram of model division provided in the embodiment of the present application. Figure 8 As shown, the number of business processing networks included in the business processing model can be recorded as L, where L is a positive integer, i.e., L business processing networks 802, wherein d business processing networks reuse a business latent vector, where d is a positive integer. At this time, business processing network 1, business processing network d+1, etc. will add the obtained business latent vector to the key-value cache 801, and other business processing networks will reuse the cached business latent vector. Figure 8 As shown in , business processing network 1 can add business latent vector 1 to the key-value cache 801, and business processing network 2 to business processing network d can obtain the cross-layer transfer parameters corresponding to business processing network 1; business processing network d+1 can add business latent vector d+1 to the key-value cache 801, and business processing network d+2 to business processing network 2d can obtain the cross-layer transfer parameters corresponding to business processing network d+1, etc.

[0140] The division of multiple business processing networks may be performed during the training of the business processing model. In this case, the first business processing network and the second business processing network may be directly determined based on the training situation. Alternatively, the multiple business processing networks may be divided during the application of the business processing model, and the business processing model may be fine-tuned based on the division result to obtain a fine-tuned business processing model. In the fine-tuned business processing model, the first business processing network and the second business processing network may be directly determined based on the training situation. Figure 3 or Figure 5Processes such as the above. Model fine-tuning includes pruning neurons and pruning connections, etc. When dividing multiple business processing networks, the computer device can obtain the network number L of the business processing networks included in the business processing model, and obtain the parameter reuse layer range. The parameter reuse layer range is used to represent the number range of business processing networks that can reuse business latent vectors. By limiting the parameter reuse layer range, the excessive performance loss of the model can be avoided while reducing the memory space occupied by the key-value cache. For example, the parameter reuse layer range can be [2, 4], etc., which is used to balance the compression rate of the memory space occupied and the model performance. From the parameter reuse layer range, obtain the approximate number of the network number, and determine the approximate number of the network number as the reuse layer number. Optionally, the reuse layer number can also be provided by the management personnel. The a*d+1th business processing network in the business processing model is determined as the first business processing network, and the business processing networks included in the business processing model, except the first business processing network, are determined as the second business processing network; d is the number of multiplexing layers; a is a natural number, a*d+1 is less than or equal to L, that is, the 1st business processing network, the d+1th business processing network, ... are determined as the first business processing network.

[0141] In an embodiment of the present application, a computer device can obtain a first business hidden state corresponding to business data in a first business processing network, use a first projection matrix to perform dimension compression on the first business hidden state to obtain a business hidden vector, and add the business hidden vector to a key-value cache; obtain the business hidden vector from the key-value cache, use a second projection matrix to perform key-value conversion on the business hidden vector to obtain a first key vector and a first value vector, perform attention processing on the first key vector and the first value vector, and obtain a second business hidden state; obtain a cross-layer transfer parameter from the first key vector, the first value vector, and the business hidden vector, and pass the cross-layer transfer parameter to the second business processing network; in the second business processing network, determine a second key vector and a second value vector based on the cross-layer transfer parameter, and based on the second business hidden state, perform attention processing on the second key vector and the second value vector to obtain a third business hidden state; when the second business processing network is the last business processing network, perform information restoration on the third business hidden state to obtain a business processing result for the business data. Through the above process, an intermediate state, namely the business hidden vector, is introduced between the business hidden state and the key-value state (i.e., the key vector and the value vector). There is no need to cache the key-value state, but the introduced intermediate state is cached, and the intermediate state is obtained after the business hidden state is dimensionally compressed, so that the cache of the key-value state only needs to store one copy of data, and the data has been dimensionally compressed, so that the size of the video memory space occupied by the key-value cache is reduced to less than half of the original, reducing the video memory usage of the model. The reduction in video memory usage can increase the batch size, thereby improving the model deployment efficiency, improving the video memory utilization, reducing the model deployment cost, and improving the model performance. Among them, the batch size refers to the amount of data used each time when training or reasoning the model (i.e., the business processing model), that is, the number of data samples used at a time.

[0142] Optionally, when multiplexing across layers, the location association information can be processed separately. There are many situations in which the data multiplexed across layers can exist. For details, see Figures 9a to 9d Among them, a cross-layer multiplexing data case ①, see Figure 9a , Figure 9a This is a cross-layer multiplexing scenario provided by the embodiment of the present application. Figure 1 ,like Figure 9aAs shown, in the first business processing network 901, the first business hidden state can be dimensionally compressed to obtain a business hidden vector, the business hidden vector is determined as a cross-layer transfer parameter, and the cross-layer transfer parameter is transferred to the second business processing network 902. In the first business processing network 901, the first position association information can be obtained based on the first business hidden state, the first key vector, the first value vector and the first position association information are feature fused to obtain the first full key-value data, and the first full key-value data and the first query vector are subjected to attention processing to obtain the second business hidden state. For details, see Figure 5 In the second business processing network 902, the business latent vector is converted into a key value to obtain a second key vector and a second value vector, the second position association information is obtained based on the second business hidden state, the second key vector, the second value vector and the second position association information are feature fused to obtain the second full key value data, the second full key value data and the second query vector are subjected to attention processing to obtain the third business hidden state.

[0143] A case of cross-layer multiplexing data②, see Figure 9b , Figure 9b This is a cross-layer multiplexing scenario provided by the embodiment of the present application. Figure 2 ,like Figure 9b As shown, the key vector and the value vector are cross-layer multiplexed. In the first business processing network 903, the first business hidden state can be dimensionally compressed to obtain a business hidden vector, and the first key vector and the first value vector are determined as cross-layer transfer parameters. The cross-layer transfer parameters are passed to the second business processing network 904. In the first business processing network 903, the first position association information can be obtained based on the first business hidden state, and the first key vector, the first value vector and the first position association information are feature fused to obtain the first full key-value data. The first full key-value data and the first query vector are subjected to attention processing to obtain the second business hidden state. For details, see Figure 5 In the second business processing network 904, the first key vector is determined as the second key vector, the first value vector is determined as the second value vector, the second position association information is obtained based on the second business hidden state, the second key vector, the second value vector and the second position association information are feature fused to obtain the second full key value data, the second full key value data and the second query vector are subjected to attention processing to obtain the third business hidden state.

[0144] A case of cross-layer multiplexing data ③, see Fig.9c , Fig.9c This is a cross-layer multiplexing scenario provided by the embodiment of the present application. Figure 3 ,like Fig.9cAs shown, the service hidden vector and the first position association information are cross-layer multiplexed. When the service hidden vector and the first position association information are obtained in the first service processing network 905, the service hidden vector can be determined as a cross-layer transfer parameter, and the cross-layer transfer parameter and the first position association information are transferred to the second service processing network 906. In the first service processing network 905, the second service hidden state is obtained based on the first key vector, the first value vector, the first position association information and the first query vector; in the second service processing network 905, the third service hidden state is obtained based on the second key vector, the second value vector, the second position association information and the second query vector.

[0145] A case of cross-layer multiplexing data ④, see Figure 9d , Figure 9d This is a cross-layer multiplexing scenario provided by the embodiment of the present application. Figure 4 ,like Figure 9d As shown, the first key vector, the first value vector and the first position association information are cross-layer multiplexed. When the first key vector, the first value vector and the first position association information are obtained in the first service processing network 907, the first key vector and the first value vector can be determined as cross-layer transfer parameters, and the cross-layer transfer parameters and the first position association information are transferred to the second service processing network 908. In the first service processing network 907, the second service hidden state is obtained based on the first key vector, the first value vector, the first position association information and the first query vector; in the second service processing network 908, the third service hidden state is obtained based on the second key vector, the second value vector, the second position association information and the second query vector.

[0146] For further information, see Fig.10 , Fig.10 This is another flow chart of a model processing method provided by an embodiment of the present application. Fig.10 As shown, vector quantization compression is performed on the basis of key-value cache compression. Specifically, the model processing process includes the following steps:

[0147] Step S1001: In a business processing network, a business hidden state corresponding to business data is obtained, and a first projection matrix is ​​used to perform dimension compression on the business hidden state to obtain a first latent vector.

[0148] In the embodiment of the present application, this process can be referred to Figure 5 In step S501, the process of obtaining the first latent vector is not described in detail here.

[0149] Step S1002: quantize and compress the first latent vector to obtain a business latent vector, and add the business latent vector to the key-value cache.

[0150] In the embodiment of the present application, this process can be referred to Figure 5 The relevant description in step S501 will not be repeated here.

[0151] Step S1003, obtaining the business latent vector from the key-value cache, and quantizing and restoring the business latent vector to obtain a second latent vector.

[0152] In the embodiment of the present application, this process can be referred to Figure 5 The relevant description in step S502 will not be repeated here.

[0153] Step S1004: Use the second projection matrix to perform key-value conversion on the second latent vector to obtain a key vector and a value vector, perform attention processing on the key vector and the value vector, and obtain the business hidden state corresponding to the business processing network.

[0154] In the embodiment of the present application, this process can be referred to Figure 5 In step S502, the relevant description of the process of obtaining the second service hidden state is not repeated here. Further, if the service processing network is not the last service processing network, the service hidden state corresponding to the service processing network is determined as the service hidden state corresponding to the service data in the next service processing network, and step S1001 is returned to the next service processing network. If the service processing network is the last service processing network, step S1005 is executed.

[0155] Step S1005 , when the service processing network is the last service processing network, information restoration is performed on the service hidden state corresponding to the service processing network to obtain a service processing result for the service data.

[0156] In the embodiment of the present application, this process can be referred to Figure 5 The relevant description in step S505 will not be repeated here.

[0157] In an embodiment of the present application, by performing dimension compression and quantization compression on the business hidden state, the amount of video memory space occupied by the cache corresponding to a single vector element is updated from the size of the key vector and the value vector to the size of the business latent vector after quantization compression, that is, from 2H to latent size / quantization compression multiple, H is greater than the latent size, which greatly reduces the video memory space occupied by the key-value cache, thereby improving the efficiency and performance of model deployment.

[0158] For further information, see Fig.11 , Fig.11 This is a flow chart of another model processing method provided by the embodiment of the present application. Fig.11 As shown, based on the key-value cache compression, vector quantization compression and cross-layer multiplexing are performed. Specifically, the model processing process includes the following steps:

[0159] Step S1101: In a first business processing network, a first business hidden state corresponding to business data is obtained, and a first projection matrix is ​​used to perform dimension compression on the first business hidden state to obtain a first latent vector.

[0160] In the embodiments of this application, please refer to Figure 5 The relevant description in step S501 will not be repeated here.

[0161] Step S1102: quantize and compress the first latent vector to obtain a business latent vector, and add the business latent vector to the key-value cache.

[0162] In the embodiments of this application, please refer to Figure 5 The relevant description in step S501 will not be repeated here.

[0163] Step S1103, obtaining the business latent vector from the key-value cache, and quantizing and restoring the business latent vector to obtain a second latent vector.

[0164] In the embodiments of this application, please refer to Figure 5 The relevant description in step S502 will not be repeated here.

[0165] Step S1104: Use the second projection matrix to perform key-value conversion on the second latent vector to obtain a first key vector and a first value vector, perform attention processing on the first key vector and the first value vector, and obtain a second business hidden state.

[0166] In the embodiments of this application, please refer to Figure 5 The relevant description in step S502 will not be repeated here.

[0167] Step S1105: Obtain a cross-layer transfer parameter from the first key vector, the first value vector and the service latent vector, and transfer the cross-layer transfer parameter to the second service processing network.

[0168] In the embodiments of this application, please refer to Figure 5 The relevant description in step S503 is not repeated here. Optionally, the number of the second service processing networks is d-1, and the first service processing network can transmit the cross-layer transfer parameter to d-1 second service processing networks.

[0169] Step S1106: In the second service processing network, a second key vector and a second value vector are determined based on the cross-layer transfer parameter, and attention processing is performed on the second key vector and the second value vector based on the second service hidden state to obtain a third service hidden state.

[0170] In the embodiments of this application, please refer to Figure 5 The relevant description in step S504 will not be repeated here. Optionally, when d-1 is greater than 1, in the second business processing network, the second key vector and the second value vector in the k-th second business processing network are determined based on the cross-layer transfer parameter, and based on the second business hidden state corresponding to the k-th second business processing network, the second key vector and the second value vector in the k-th second business processing network are subjected to attention processing to obtain the third business hidden state corresponding to the k-th second business processing network, where k is a positive integer less than or equal to d-1. When k is less than d-1, the third business hidden state corresponding to the k-th second business processing network is determined as the second business hidden state corresponding to the k+1-th second business processing network, and the third business hidden state corresponding to the k+1-th second business processing network is determined. When k is d-1, execute step S1107.

[0171] Step S1107: The second service processing network is the last service processing network.

[0172] In the embodiment of the present application, if the second service processing network is not the last service processing network, step S1108 is executed; if the second service processing network is the last service processing network, step S1109 is executed.

[0173] Step S1108, entering the next service processing network.

[0174] Step S1109: restore the information of the hidden state of the third service to obtain a service processing result for the service data.

[0175] In the embodiments of this application, please refer to Figure 5 The relevant description in step S505 will not be repeated here.

[0176] In the embodiment of the present application, it can be recorded as Cross Layer Latent Attention (CLLA) plus quantization (Quant) mechanism, which can greatly reduce the memory space occupied by the key-value cache, and can compress the key-value cache to 1% of the original, so that the memory space occupied is greatly reduced. The reduction in the memory space occupied can increase the batch size, thereby improving the deployment efficiency of the model. Moreover, more efficient memory space utilization can reduce the deployment cost of the model, while saving costs and improving the performance of the model.

[0177] For further information, see Fig.12 , Fig.12 This is a flow chart of a training process method for model processing provided by an embodiment of the present application. Fig.12 As shown, the model processing process includes the following steps:

[0178] Step S1201, in the first initial processing network of the initial processing model, obtain the first sample hidden state corresponding to the business sample, use the first initial projection matrix to perform dimension compression on the first sample hidden state to obtain a sample latent vector, and add the sample latent vector to the sample key value cache.

[0179] In the embodiment of the present application, this process can be referred to Figure 5 In step S501, the process of generating and caching the service latent vector is not described in detail here.

[0180] Optionally, the computer device can directly obtain the initial processing model. Alternatively, the basic model can be obtained, and the first initial processing network and the second initial processing network can be determined from the basic model; based on the model architecture of the first initial processing network, the second initial processing network and the basic model, the network dependency and the topological dependency in the basic model are determined, the network dependency is used to represent the dependency between adjacent initial processing networks, and the topological dependency is used to represent the dependency of the residual structure in the model; based on the network dependency and the topological dependency, the neurons to be optimized are obtained from the basic model, and the neurons to be optimized in the basic model are sparsely processed to obtain the initial processing model. Among them, the sparse processing refers to the processing of deleting the neurons to be optimized and the connections associated with the neurons to be optimized.

[0181] Alternatively, the basic model and the associated model of the basic model can be obtained, and the associated model parameters in the associated model can be obtained; the task similarity between the associated model task corresponding to the associated model and the basic model task corresponding to the basic model is greater than or equal to the model association threshold; the basic model parameters in the basic model are initialized based on the associated model parameters to obtain the initial processing model. It can be considered that the teacher-student model framework is formed between the associated model and the initial processing model.

[0182] That is to say, the basic model can be compressed based on the model compression technology to obtain the initial processing model. The model compression technology may include but is not limited to methods such as model sparsification or knowledge distillation.

[0183] Optionally, a first initial processing network and a second initial processing network in the initial processing model may be determined. The determination process of the first initial processing network and the second initial processing network may be referred to in Figure 5 The division process of the first service processing network and the second service processing network will not be described in detail here.

[0184] Step S1202, obtain the sample latent vector from the sample key value cache, use the second initial projection matrix to perform key value conversion on the sample latent vector to obtain the first sample key vector and the first sample value vector, perform attention processing on the first sample key vector and the first sample value vector to obtain the second sample hidden state.

[0185] In the embodiment of the present application, this process can be referred to Figure 5 In step S502, the process of generating and caching the service latent vector is not described in detail here.

[0186] Step S1203, obtaining a sample transfer parameter from the first sample key vector, the first sample value vector and the sample latent vector, and passing the sample transfer parameter to the second initial processing network.

[0187] In the embodiment of the present application, this process can be referred to Figure 5 The relevant description in step S503 will not be repeated here.

[0188] Step S1204: In the second initial processing network, a second sample key vector and a second sample value vector are determined based on the sample transfer parameter, and based on the second sample hidden state, attention processing is performed on the second sample key vector and the second sample value vector to obtain a third sample hidden state.

[0189] In the embodiment of the present application, this process can be referred to Figure 5 In step S504, the process of generating and caching the service latent vector is not described in detail here.

[0190] Step S1205: when the second initial processing network is the last initial processing network, information restoration is performed on the third sample hidden state to obtain a sample processing result for the business sample.

[0191] In the embodiment of the present application, this process can be referred to Figure 5 In step S505, the process of generating and caching the service latent vector is not described in detail here.

[0192] Step S1206: Adjust parameters of the initial processing model based on the sample processing result to obtain a service processing model after parameter convergence.

[0193] In the embodiment of the present application, the business processing model includes a first projection matrix corresponding to the first initial projection matrix, and a second projection matrix corresponding to the second initial projection matrix.

[0194] The computer device for training the business processing model and the computer device for running the business processing model may be the same device or different devices.

[0195] In an embodiment of the present application, the input data can be first projected to a hidden vector of a small hidden layer size, and then restored to the KV state to be used when needed, so that when deployed, only the projected hidden vector needs to be retained, thereby reducing the memory space occupied by the key-value cache, and compressing 2H of the original memory space occupancy to the size of the hidden vector, greatly reducing the memory space occupancy of the key-value cache. At the same time, cross-layer data reuse is performed on the business processing network, so that multiple business processing networks only need to cache the business hidden vector once, and further compress the memory space occupancy of the key-value cache. In the autoregressive generation process of the business processing model, the required data can be directly obtained from the key-value cache, that is, fewer computing resources are used in exchange for more quantized space resources, thereby improving the model deployment efficiency.

[0196] Among them, compared with the prior art, the performance of the model of this application is shown in Table 1 below:

[0197] Table 1

[0198]

[0199]

[0200]

[0201] As shown in Table 1, the network impact index is used to indicate the impact of network congestion on model performance, BoolQ is used to evaluate the model's ability to handle simple fact-based queries, commonSenseQA is used to indicate the correctness of answer predictions in common sense question-answering text datasets, and PIQA is used to indicate the model's ability to solve physics-related problems. As can be seen from Table 1, compared with the existing group query attention-based model, the model implemented in this application has optimized various model indicators and improved model performance.

[0202] Among them, see Fig.13 , Fig.13 Schematic diagram of a model operation performance provided by the embodiment of the present application. Fig.13 As shown, the model accuracy index (Accuracy, ACC) used to describe the existing group query attention basic model and the model implemented in this application, that is, the ACC value of each model under different batch sizes. It can be seen that the model performance in this application has been improved.

[0203] For further information, see Fig.14 , Fig.14Schematic diagram of a model processing device provided in an embodiment of the present application. The model processing device may be a computer program (including program code, etc.) running in a computer device, for example, the model processing device may be an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig.14 As shown, the model processing device 1400 can be used to Figure 5 The computer device in the corresponding embodiment may specifically include: a parameter compression module 11 , a key value processing module 12 , a cross-layer transfer module 13 and a service prediction module 14 .

[0204] A parameter compression module 11 is used to obtain a first business hidden state corresponding to the business data in the first business processing network, perform dimension compression on the first business hidden state using a first projection matrix to obtain a business hidden vector, and add the business hidden vector to a key-value cache;

[0205] A key value processing module 12 is used to obtain a business latent vector from a key value cache, perform key value conversion on the business latent vector using a second projection matrix to obtain a first key vector and a first value vector, perform attention processing on the first key vector and the first value vector, and obtain a second business hidden state;

[0206] A cross-layer transfer module 13, configured to obtain a cross-layer transfer parameter from the first key vector, the first value vector and the service latent vector, and transfer the cross-layer transfer parameter to the second service processing network;

[0207] The key value processing module 12 is further used to determine the second key vector and the second value vector based on the cross-layer transfer parameter in the second service processing network, and perform attention processing on the second key vector and the second value vector based on the second service hidden state to obtain a third service hidden state;

[0208] The service prediction module 14 is used to restore information of the third service hidden state when the second service processing network is the last service processing network, so as to obtain a service processing result for the service data.

[0209] When the first projection matrix is ​​used to perform dimension compression on the first service hidden state to obtain the service hidden vector, the parameter compression module 11 can be used to:

[0210] Using a first projection matrix to perform dimension compression on a first service hidden state to obtain a first latent vector;

[0211] Quantize and compress the first latent vector to obtain a business latent vector;

[0212] When the second projection matrix is ​​used to perform key-value conversion on the business latent vector to obtain the first key vector and the first value vector, the key-value processing module 12 can be used to:

[0213] Quantize and restore the business latent vector to obtain a second latent vector;

[0214] The product of the first key projection matrix and the second latent vector in the second projection matrix is ​​determined as the first key vector, and the product of the first value projection matrix and the second latent vector in the second projection matrix is ​​determined as the first value vector.

[0215] When the first latent vector is quantized and compressed to obtain the service latent vector, the parameter compression module 11 can be used to:

[0216] The first latent vector is grouped to obtain N vector groups; N is a positive integer; each vector group includes vector elements;

[0217] Obtaining a quantization level, obtaining vector statistics corresponding to each of the N vector groups, and determining quantization parameters corresponding to each of the N vector groups based on the quantization level and the vector statistics corresponding to each of the N vector groups;

[0218] The quantization parameter corresponding to each vector group is used to compress the vector elements included in the vector group to obtain service latent vectors corresponding to the N vector groups respectively.

[0219] When the business latent vector is quantized and restored to obtain the second latent vector, the key value processing module 12 can be used to:

[0220] Obtaining the service latent vectors and quantization parameters corresponding to the N vector groups respectively, using the quantization parameters corresponding to each vector group, performing quantization restoration processing on the service latent vectors corresponding to the vector group, and obtaining the restored latent vectors corresponding to the N vector groups respectively;

[0221] The restored latent vectors corresponding to the N vector groups are merged to obtain the second latent vector.

[0222] When performing attention processing on the first key vector and the first value vector to obtain the second service hidden state, the key value processing module 12 can be used to:

[0223] Determine the product of the first query matrix and the first service hidden state as a first query vector;

[0224] Obtaining a vector dimension of the first query vector, and performing feature fusion on the first query vector and the first key vector based on the vector dimension to obtain vector attention;

[0225] Vector attention is used to perform weighted processing on the first value vector to obtain a second service hidden state.

[0226] The first service hidden state includes S first service sub-vectors, where S is a positive integer;

[0227] The key value processing module 12 can be used for:

[0228] If i is 1, the i-th business latent vector is obtained from the key-value cache, the i-th business latent vector is converted into a key value using the second projection matrix to obtain the i-th first key vector and the i-th first value vector, and the i-th first key vector and the i-th first value vector are subjected to attention processing to obtain the i-th second business sub-vector; i is used to indicate the position of the i-th first business sub-vector in the S first business sub-vectors;

[0229] If i is a positive integer greater than 1 and less than or equal to S, then obtain the first query vector of the i-th first business sub-vector, obtain the business latent vectors corresponding to the first first business sub-vector to the i-th first business sub-vector from the key-value cache, use the second projection matrix to perform key-value conversion on the first business latent vector to the i-th business latent vector, and obtain the first key vectors and first value vectors corresponding to the first first business sub-vector to the i-th first business sub-vector, respectively; according to the first key vectors corresponding to the first first business sub-vector to the i-th first business sub-vector and the first query vector corresponding to the i-th first business sub-vector, determine the sub-attentions corresponding to the first first business sub-vector to the i-th first business sub-vector, use the sub-attentions corresponding to the first first business sub-vector to the i-th first business sub-vector, respectively, and perform weighted processing on the first value vectors corresponding to the first first business sub-vector to the i-th first business sub-vector, respectively, to obtain the i-th second business sub-vector;

[0230] When i is S, the first second service sub-vector to the Sth second service sub-vector form a second service hidden state.

[0231] Among them, when obtaining the cross-layer transfer parameter from the first key vector, the first value vector and the service hidden vector, the cross-layer transfer module 13 can be used to:

[0232] Determining a service latent vector as a cross-layer transfer parameter;

[0233] In the second service processing network, when determining the second key vector and the second value vector based on the cross-layer transfer parameter, the key value processing module 12 may be used to:

[0234] In the second service processing network, the third projection matrix in the second service processing network is used to perform key-value conversion on the cross-layer transfer parameters to obtain a second key vector and a second value vector.

[0235] Among them, when obtaining the cross-layer transfer parameter from the first key vector, the first value vector and the service hidden vector, the cross-layer transfer module 13 can be used to:

[0236] determining the first key vector and the first value vector as cross-layer transfer parameters;

[0237] In the second service processing network, when determining the second key vector and the second value vector based on the cross-layer transfer parameter, the key value processing module 12 may be used to:

[0238] In the second service processing network, a second key vector and a second value vector are obtained from the cross-layer transfer parameters.

[0239] When performing attention processing on the first key vector and the first value vector to obtain the second service hidden state, the key value processing module 12 can be used to:

[0240] Acquire location-related information based on the first service hidden state, perform feature fusion on the first key vector, the first value vector, and the location-related information, and obtain full key-value data;

[0241] Determine the product of the first query matrix and the first service hidden state as a first query vector;

[0242] The first query vector is subjected to attention processing using the full amount of key-value data to obtain a second business hidden state.

[0243] The device 1400 further includes:

[0244] A data acquisition module 15 is used to acquire the number of service processing networks included in the service processing model and the range of parameter multiplexing layers;

[0245] The layer number determination module 16 is used to obtain the approximate number of networks from the parameter multiplexing layer number range, and determine the approximate number of networks as the multiplexing layer number;

[0246] The network determination module 17 is used to determine the a*d+1th business processing network in the business processing model as the first business processing network, and determine the business processing networks other than the first business processing network included in the business processing model as the second business processing network; d is the number of multiplexing layers; a is a natural number.

[0247] The device 1400 further includes:

[0248] The storage transfer module 18 is used to store the key value cache corresponding to the first business object in the hardware memory if it is detected that the first business object is in an offline state; the first business object is a business object that provides business data;

[0249] The storage transfer module 18 is also used to obtain the key value cache from the hardware memory and execute the process of obtaining the service latent vector from the key value cache when it is detected that the object state of the first service object is changed from the offline state to the online state.

[0250] An embodiment of the present application provides a model processing device, which can obtain a first business hidden state corresponding to business data in a first business processing network, use a first projection matrix to perform dimension compression on the first business hidden state to obtain a business hidden vector, and add the business hidden vector to a key-value cache; obtain the business hidden vector from the key-value cache, use a second projection matrix to perform key-value conversion on the business hidden vector to obtain a first key vector and a first value vector, perform attention processing on the first key vector and the first value vector, and obtain a second business hidden state; obtain a cross-layer transfer parameter from the first key vector, the first value vector and the business hidden vector, and pass the cross-layer transfer parameter to the second business processing network; in the second business processing network, determine a second key vector and a second value vector based on the cross-layer transfer parameter, and based on the second business hidden state, perform attention processing on the second key vector and the second value vector to obtain a third business hidden state; when the second business processing network is the last business processing network, perform information restoration on the third business hidden state to obtain a business processing result for the business data. Through the above process, an intermediate state, namely the business hidden vector, is introduced between the business hidden state and the key-value state (i.e., the key vector and the value vector). There is no need to cache the key-value state, but the introduced intermediate state is cached, and the intermediate state is obtained after dimension compression of the business hidden state, so that the cache of the key-value state only needs to store one copy of data, and the data has been dimensionally compressed, so that the size of the video memory space occupied by the key-value cache is reduced to less than half of the original size, reducing the video memory occupancy of the model. The reduction in video memory occupancy can increase the batch size, thereby improving the model deployment efficiency, improving the video memory utilization, reducing the model deployment cost, and improving the model performance.

[0251] For further information, see Fig.15 , Fig.15 Schematic diagram of another model processing device provided in an embodiment of the present application. The model processing device may be a computer program (including program code, etc.) running in a computer device, for example, the model processing device may be an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig.15 As shown, the model processing device 1500 can be used to Fig.12 The computer device in the corresponding embodiment, specifically, the device may include: a sample compression module 21, a sample processing module 22, a parameter processing module 23, a sample prediction module 24 and a model training module 25.

[0252] A sample compression module 21 is used to obtain a first sample hidden state corresponding to a business sample in a first initial processing network of an initial processing model, perform dimension compression on the first sample hidden state using a first initial projection matrix to obtain a sample hidden vector, and add the sample hidden vector to a sample key value cache;

[0253] A sample processing module 22 is used to obtain a sample latent vector from a sample key value cache, perform key value conversion on the sample latent vector using a second initial projection matrix to obtain a first sample key vector and a first sample value vector, perform attention processing on the first sample key vector and the first sample value vector, and obtain a second sample hidden state;

[0254] A parameter processing module 23, used to obtain a sample transfer parameter from the first sample key vector, the first sample value vector and the sample latent vector, and transfer the sample transfer parameter to the second initial processing network in the initial processing model;

[0255] The sample processing module 22 is further used to determine, in the second initial processing network, a second sample key vector and a second sample value vector based on the sample transfer parameter, and perform attention processing on the second sample key vector and the second sample value vector based on the second sample hidden state to obtain a third sample hidden state;

[0256] A sample prediction module 24, used for restoring information on the third sample hidden state when the second initial processing network is the last initial processing network, to obtain a sample processing result for the business sample;

[0257] The model training module 25 is used to adjust the parameters of the initial processing model based on the sample processing results to obtain a business processing model after parameter convergence; the business processing model includes a first projection matrix corresponding to the first initial projection matrix, and a second projection matrix corresponding to the second initial projection matrix.

[0258] The device 1500 further includes:

[0259] A model acquisition module 26 is used to acquire a basic model and determine a first initial processing network and a second initial processing network from the basic model;

[0260] A dependency acquisition module 27, configured to determine network dependency and topology dependency in the basic model based on the first initial processing network, the second initial processing network and the model architecture of the basic model;

[0261] The sparse processing module 28 is used to obtain the neurons to be optimized from the basic model based on the network dependency and the topological dependency, and perform sparse processing on the neurons to be optimized in the basic model to obtain an initial processing model.

[0262] The device 1500 further includes:

[0263] The association acquisition module 29 is used to acquire the basic model and the association model of the basic model, and acquire the association model parameters in the association model; the task similarity between the association model task corresponding to the association model and the basic model task corresponding to the basic model is greater than or equal to the model association threshold;

[0264] The parameter initialization module 30 is used to initialize the basic model parameters in the basic model based on the associated model parameters to obtain an initial processing model.

[0265] In an embodiment of the present application, the input data can be first projected to a hidden vector of a small hidden layer size, and then restored to the KV state to be used when needed, so that when deployed, only the projected hidden vector needs to be retained, thereby reducing the memory space occupied by the key-value cache, and compressing 2H of the original memory space occupancy to the size of the hidden vector, greatly reducing the memory space occupancy of the key-value cache. At the same time, cross-layer data reuse is performed on the business processing network, so that multiple business processing networks only need to cache the business hidden vector once, and further compress the memory space occupancy of the key-value cache. In the autoregressive generation process of the business processing model, the required data can be directly obtained from the key-value cache, that is, fewer computing resources are used in exchange for more quantized space resources, thereby improving the model deployment efficiency.

[0266] See also Fig.16 , Fig.16 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.16 As shown, the computer device in the embodiment of the present application may include: one or more processors 1601, a memory 1602, and an input-output interface 1603. The processor 1601, the memory 1602, and the input-output interface 1603 are connected via a bus 1604. The memory 1602 is used to store a computer program, which includes program instructions, and the input-output interface 1603 is used to receive data and output data, such as for data interaction between the computer device and the business device; the processor 1601 is used to execute the program instructions stored in the memory 1602.

[0267] The processor 1601 may perform the following operations when running the business processing model:

[0268] In the first service processing network, a first service hidden state corresponding to the service data is obtained, a first projection matrix is ​​used to perform dimension compression on the first service hidden state to obtain a service hidden vector, and the service hidden vector is added to the key-value cache;

[0269] Obtaining a business latent vector from the key-value cache, performing key-value conversion on the business latent vector using a second projection matrix to obtain a first key vector and a first value vector, performing attention processing on the first key vector and the first value vector to obtain a second business hidden state;

[0270] Obtaining a cross-layer transfer parameter from the first key vector, the first value vector and the service latent vector, and transmitting the cross-layer transfer parameter to the second service processing network;

[0271] In the second service processing network, a second key vector and a second value vector are determined based on the cross-layer transfer parameter, and attention processing is performed on the second key vector and the second value vector based on the second service hidden state to obtain a third service hidden state;

[0272] When the second service processing network is the last service processing network, information restoration is performed on the third service hidden state to obtain a service processing result for the service data.

[0273] The processor 1601 may perform the following operations when training the model:

[0274] In a first initial processing network of the initial processing model, a first sample hidden state corresponding to the business sample is obtained, a first initial projection matrix is ​​used to perform dimension compression on the first sample hidden state to obtain a sample latent vector, and the sample latent vector is added to a sample key value cache;

[0275] Obtain a sample latent vector from the sample key-value cache, perform key-value conversion on the sample latent vector using the second initial projection matrix to obtain a first sample key vector and a first sample value vector, perform attention processing on the first sample key vector and the first sample value vector to obtain a second sample hidden state;

[0276] Obtaining a sample transfer parameter from the first sample key vector, the first sample value vector, and the sample latent vector, and transferring the sample transfer parameter to a second initial processing network in the initial processing model;

[0277] In the second initial processing network, a second sample key vector and a second sample value vector are determined based on the sample transfer parameter, and based on the second sample hidden state, attention processing is performed on the second sample key vector and the second sample value vector to obtain a third sample hidden state;

[0278] When the second initial processing network is the last initial processing network, information restoration is performed on the hidden state of the third sample to obtain a sample processing result for the business sample;

[0279] The parameters of the initial processing model are adjusted based on the sample processing results to obtain a business processing model after parameter convergence; the business processing model includes a first projection matrix corresponding to the first initial projection matrix and a second projection matrix corresponding to the second initial projection matrix.

[0280] In some feasible implementations, the processor 1601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0281] The memory 1602 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1601 and the input / output interface 1603. A portion of the memory 1602 may also include a non-volatile random access memory. For example, the memory 1602 may also store information about the device type.

[0282] In a specific implementation, the computer device can execute the following operations through its built-in functional modules: Figure 5 or Fig.12 For details on the implementation methods provided in each step, please refer to the Figure 5 or Fig.12 The implementation methods provided in each step will not be repeated here.

[0283] The embodiment of the present application provides a computer device, including: a processor, an input / output interface, and a memory, wherein the processor obtains a computer program in the memory and executes the computer program. Figure 5The various steps of the method shown in the embodiment of the present application are used to perform model processing operations. The embodiment of the present application realizes that in the first business processing network, the first business hidden state corresponding to the business data is obtained, the first projection matrix is ​​used to perform dimension compression on the first business hidden state, and the business hidden vector is obtained, and the business hidden vector is added to the key-value cache; the business hidden vector is obtained from the key-value cache, and the business hidden vector is converted into a key value using a second projection matrix to obtain a first key vector and a first value vector, and the first key vector and the first value vector are subjected to attention processing to obtain a second business hidden state; from the first key vector, the first value vector and the business hidden vector, a cross-layer transfer parameter is obtained, and the cross-layer transfer parameter is passed to the second business processing network; in the second business processing network, the second key vector and the second value vector are determined based on the cross-layer transfer parameter, and the second key vector and the second value vector are subjected to attention processing based on the second business hidden state to obtain a third business hidden state; when the second business processing network is the last business processing network, the third business hidden state is subjected to information restoration to obtain a business processing result for the business data. Through the above process, an intermediate state, namely the business hidden vector, is introduced between the business hidden state and the key-value state (i.e., the key vector and the value vector). There is no need to cache the key-value state, but the introduced intermediate state is cached, and the intermediate state is obtained after dimension compression of the business hidden state, so that the cache of the key-value state only needs to store one copy of data, and the data has been dimensionally compressed, so that the size of the video memory space occupied by the key-value cache is reduced to less than half of the original size, reducing the video memory occupancy of the model. The reduction in video memory occupancy can increase the batch size, thereby improving the model deployment efficiency, improving the video memory utilization, reducing the model deployment cost, and improving the model performance.

[0284] The present application also provides a computer-readable storage medium storing a computer program suitable for being loaded and executed by the processor. Figure 5 or Fig.12 The model processing methods provided in each step are detailed in this Figure 5 or Fig.12 The implementation methods provided in each step are not repeated here. In addition, the description of the beneficial effects of adopting the same method is not repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the computer program can be deployed to execute on one computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0285] The computer-readable storage medium may be the model processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0286] The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 5 or Fig.12 The method provided in the various optional methods realizes the introduction of an intermediate state, namely the business hidden vector, between the business hidden state and the key-value state (namely the key vector and the value vector). There is no need to cache the key-value state, but to cache the introduced intermediate state, and the intermediate state is obtained after dimension compression of the business hidden state, so that the cache of the key-value state only needs to store one copy of data, and the data has been dimensionally compressed, so that the size of the video memory space occupied by the key-value cache is reduced to less than half of the original size, reducing the video memory usage of the model. The reduction in video memory usage can increase the batch size, thereby improving the model deployment efficiency, improving the video memory utilization, reducing the model deployment cost, and improving the model performance.

[0287] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0288] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0289] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0290] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable model processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable model processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable model processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable model processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0291] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0292] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0293] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A model processing method, characterized in that: The method comprises: In the first service processing network, a first service hidden state corresponding to the service data is obtained, a first projection matrix is ​​used to perform dimension compression on the first service hidden state to obtain a service hidden vector, and the service hidden vector is added to the key-value cache; Obtaining the business latent vector from the key-value cache, performing key-value conversion on the business latent vector using a second projection matrix to obtain a first key vector and a first value vector, performing attention processing on the first key vector and the first value vector to obtain a second business hidden state; Acquire a cross-layer transfer parameter from the first key vector, the first value vector and the service latent vector, and transfer the cross-layer transfer parameter to the second service processing network; In the second service processing network, a second key vector and a second value vector are determined based on the cross-layer transfer parameter, and attention processing is performed on the second key vector and the second value vector based on the second service hidden state to obtain a third service hidden state; When the second service processing network is the last service processing network, information restoration is performed on the third service hidden state to obtain a service processing result for the service data.

2. The method according to claim 1, characterized in that The step of using the first projection matrix to perform dimension compression on the first service hidden state to obtain a service hidden vector includes: Using the first projection matrix to perform dimension compression on the first service hidden state to obtain a first latent vector; quantizing and compressing the first latent vector to obtain a service latent vector; The step of using the second projection matrix to perform key-value conversion on the business latent vector to obtain a first key vector and a first value vector includes: Quantizing and restoring the service latent vector to obtain a second latent vector; The product of the first key projection matrix in the second projection matrix and the second latent vector is determined as the first key vector, and the product of the first value projection matrix in the second projection matrix and the second latent vector is determined as the first value vector.

3. The method according to claim 2, characterized in that The step of quantizing and compressing the first latent vector to obtain a service latent vector includes: The first latent vectors are grouped to obtain N vector groups, where N is a positive integer, and each vector group includes vector elements; Obtaining a quantization level, obtaining vector statistics corresponding to the N vector groups respectively, and determining quantization parameters corresponding to the N vector groups respectively based on the quantization level and the vector statistics corresponding to the N vector groups respectively; The quantization parameter corresponding to each vector group is used to compress the vector elements included in the vector group to obtain the service latent vectors corresponding to the N vector groups respectively.

4. The method according to claim 3, characterized in that The quantizing and restoring the service latent vector to obtain a second latent vector includes: Obtaining service latent vectors and quantization parameters corresponding to the N vector groups respectively, and using the quantization parameter corresponding to each vector group to perform quantization restoration processing on the service latent vector corresponding to the vector group, to obtain restored latent vectors corresponding to the N vector groups respectively; The restored latent vectors corresponding to the N vector groups are merged to obtain a second latent vector.

5. The method according to claim 1, characterized in that The performing attention processing on the first key vector and the first value vector to obtain a second service hidden state includes: Determine the product of the first query matrix and the first service hidden state as a first query vector; Acquire a vector dimension of the first query vector, and perform feature fusion on the first query vector and the first key vector based on the vector dimension to obtain vector attention; The first value vector is weightedly processed using the vector attention to obtain a second service hidden state.

6. The method according to claim 1, characterized in that The first service hidden state includes S first service sub-vectors, where S is a positive integer; The method includes: obtaining the service hidden vector from the key-value cache, performing key-value conversion on the service hidden vector using a second projection matrix to obtain a first key vector and a first value vector, performing attention processing on the first key vector and the first value vector to obtain a second service hidden state, including: If i is 1, the i-th business latent vector is obtained from the key-value cache, the i-th business latent vector is converted into a key value using the second projection matrix to obtain the i-th first key vector and the i-th first value vector, and the i-th first key vector and the i-th first value vector are subjected to attention processing to obtain the i-th second business sub-vector; i is used to indicate the position of the i-th first business sub-vector in the S first business sub-vectors; If i is a positive integer greater than 1 and less than or equal to S, then obtain the first query vector of the i-th first business sub-vector, obtain the business latent vectors corresponding to the first first business sub-vector to the i-th first business sub-vector from the key-value cache, use the second projection matrix to perform key-value conversion on the first business latent vector to the i-th business latent vector, and obtain the first key vector and the first value vector corresponding to the first first business sub-vector to the i-th first business sub-vector, and determine the sub-attentions corresponding to the first first business sub-vector to the i-th first business sub-vector according to the first key vectors corresponding to the first first business sub-vector to the i-th first business sub-vector and the first query vector corresponding to the i-th first business sub-vector, and use the sub-attentions corresponding to the first first business sub-vector to the i-th first business sub-vector to perform weighted processing on the first value vectors corresponding to the first first business sub-vector to the i-th first business sub-vector, and obtain the i-th second business sub-vector; When i is S, the first second service sub-vector to the Sth second service sub-vector form a second service hidden state.

7. The method according to claim 1, characterized in that The acquiring a cross-layer transfer parameter from the first key vector, the first value vector, and the service hidden vector includes: Determining the service latent vector as a cross-layer transfer parameter; The determining, in the second service processing network, a second key vector and a second value vector based on the cross-layer transfer parameter includes: In the second service processing network, a third projection matrix in the second service processing network is used to perform key-value conversion on the cross-layer transfer parameters to obtain a second key vector and a second value vector.

8. The method according to claim 1, characterized in that: The acquiring a cross-layer transfer parameter from the first key vector, the first value vector, and the service hidden vector includes: determining the first key vector and the first value vector as cross-layer transfer parameters; The determining, in the second service processing network, a second key vector and a second value vector based on the cross-layer transfer parameter includes: In the second service processing network, a second key vector and a second value vector are obtained from the cross-layer transfer parameters.

9. The method according to claim 1, characterized in that: The performing attention processing on the first key vector and the first value vector to obtain a second service hidden state includes: Acquire location-related information based on the first service hidden state, and perform feature fusion on the first key vector, the first value vector, and the location-related information to obtain full key-value data; Determine the product of the first query matrix and the first service hidden state as a first query vector; The first query vector is subjected to attention processing using the full amount of key-value data to obtain a second business hidden state.

10. The method according to claim 1, characterized in that The method further comprises: Obtain the number of business processing networks included in the business processing model, and obtain the range of parameter multiplexing layers; Obtaining a divisor of the number of networks from the parameter multiplexing layer number range, and determining the divisor of the number of networks as the number of multiplexing layers; The a*d+1th business processing network in the business processing model is determined as the first business processing network, and the business processing networks included in the business processing model, except the first business processing network, are determined as the second business processing network; d is the number of multiplexing layers; a is a natural number.

11. The method according to claim 1, characterized in that: The method further comprises: If it is detected that the first business object is in an offline state, the key value cache corresponding to the first business object is stored in the hardware memory; the first business object is the business object that provides the business data; When it is detected that the object state of the first business object is changed from the offline state to the online state, the key value cache is obtained from the hardware memory, and the process of obtaining the business latent vector from the key value cache is performed.

12. A model processing method, characterized in that: The method comprises: In a first initial processing network of the initial processing model, a first sample hidden state corresponding to the business sample is obtained, a first initial projection matrix is ​​used to perform dimension compression on the first sample hidden state to obtain a sample latent vector, and the sample latent vector is added to a sample key value cache; Obtaining the sample latent vector from the sample key value cache, performing key value conversion on the sample latent vector using a second initial projection matrix to obtain a first sample key vector and a first sample value vector, performing attention processing on the first sample key vector and the first sample value vector to obtain a second sample hidden state; Obtaining a sample transfer parameter from the first sample key vector, the first sample value vector and the sample latent vector, and transferring the sample transfer parameter to a second initial processing network in the initial processing model; In the second initial processing network, a second sample key vector and a second sample value vector are determined based on the sample transfer parameter, and based on the second sample hidden state, attention processing is performed on the second sample key vector and the second sample value vector to obtain a third sample hidden state; When the second initial processing network is the last initial processing network, performing information restoration on the third sample hidden state to obtain a sample processing result for the business sample; The initial processing model is parameter-adjusted based on the sample processing result to obtain a business processing model after parameter convergence; the business processing model includes a first projection matrix corresponding to the first initial projection matrix and a second projection matrix corresponding to the second initial projection matrix.

13. The method according to claim 12, characterized in that The method further comprises: Acquire a basic model, and determine the first initial processing network and the second initial processing network from the basic model; Determining network dependencies and topological dependencies in the basic model based on the first initial processing network, the second initial processing network, and the model architecture of the basic model; Based on the network dependency and the topological dependency, neurons to be optimized are obtained from the basic model, and sparse processing is performed on the neurons to be optimized in the basic model to obtain the initial processing model.

14. The method according to claim 12, characterized in that The method further comprises: Acquire a basic model and an associated model of the basic model, and acquire an associated model parameter in the associated model; the task similarity between the associated model task corresponding to the associated model and the basic model task corresponding to the basic model is greater than or equal to a model association threshold; The basic model parameters in the basic model are initialized based on the associated model parameters to obtain the initial processing model.

15. A model processing device, characterized in that: The device comprises: A parameter compression module, used to obtain, in a first service processing network, a first service hidden state corresponding to service data, perform dimension compression on the first service hidden state using a first projection matrix to obtain a service hidden vector, and add the service hidden vector to a key-value cache; A key value processing module, used to obtain the business latent vector from the key value cache, perform key value conversion on the business latent vector using a second projection matrix to obtain a first key vector and a first value vector, perform attention processing on the first key vector and the first value vector to obtain a second business hidden state; A cross-layer transfer module, configured to obtain a cross-layer transfer parameter from the first key vector, the first value vector and the service hidden vector, and transfer the cross-layer transfer parameter to a second service processing network; The key-value processing module is further used to determine, in the second service processing network, a second key vector and a second value vector based on the cross-layer transfer parameter, and perform attention processing on the second key vector and the second value vector based on the second service hidden state to obtain a third service hidden state; The service prediction module is used to restore the information of the third service hidden state when the second service processing network is the last service processing network, so as to obtain the service processing result for the service data.

16. A model processing device, characterized in that: The device comprises: A sample compression module, used to obtain a first sample hidden state corresponding to a business sample in a first initial processing network of an initial processing model, perform dimension compression on the first sample hidden state using a first initial projection matrix to obtain a sample latent vector, and add the sample latent vector to a sample key value cache; a sample processing module, configured to obtain the sample latent vector from the sample key value cache, perform key value conversion on the sample latent vector using a second initial projection matrix to obtain a first sample key vector and a first sample value vector, perform attention processing on the first sample key vector and the first sample value vector to obtain a second sample hidden state; a parameter processing module, configured to obtain a sample transfer parameter from the first sample key vector, the first sample value vector and the sample latent vector, and transfer the sample transfer parameter to a second initial processing network in the initial processing model; The sample processing module is further used to determine, in the second initial processing network, a second sample key vector and a second sample value vector based on the sample transfer parameter, and perform attention processing on the second sample key vector and the second sample value vector based on the second sample hidden state to obtain a third sample hidden state; A sample prediction module, used for restoring information of the third sample hidden state when the second initial processing network is the last initial processing network, to obtain a sample processing result for the business sample; A model training module is used to adjust the parameters of the initial processing model based on the sample processing results to obtain a business processing model after parameter convergence; the business processing model includes a first projection matrix corresponding to the first initial projection matrix, and a second projection matrix corresponding to the second initial projection matrix.

17. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive data and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1 to 11, or executes the method described in any one of claims 12 to 14.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 11, or executes the method according to any one of claims 12 to 14.

19. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method described in any one of claims 1 to 11 is implemented, or the method described in any one of claims 12 to 14 is executed.

Citation Information

Patent Citations

  • Compressed recurrent neural network models

    CN107145940A

  • Model deployment method and electronic equipment

    CN118313441A

  • Methods and apparatus to access data in non-volatile memory

    US20130054869A1

  • Condensed memory networks

    US20190087721A1

  • Systems and methods for cross-region data management in an active-active architecture

    US20210200771A1