Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining key value cache data of a first type according to a first storage space corresponding to a first network model; wherein the first storage space is used for storing key value cache data generated by the first network model; obtaining key value cache data of a second type according to a second storage space corresponding to the second network model; wherein the second storage space is used for transferring the key value cache data stored in the first storage space; and processing the current input data according to the first type of key value cache data and the second type of key value cache data. According to the mode, the data size of the key value cache data in the first storage space can be reduced, the problem that the data size of the key value cache data in the first storage space is continuously and linearly increased is avoided, and then the operation performance of the model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] KV Cache, or key-value cache, is a cache mechanism for storing key-value pairs. During the reasoning process of the language model, the same data often needs to be accessed multiple times, and KV Cache provides fast data access speed by caching this data in memory, thereby accelerating the reasoning process.

[0003] It can be seen that the key-value cache technology aims to save repeated calculations by increasing memory usage. For example, the Transformer model can use KV Cache to significantly reduce repeated calculations.

[0004] However, in related technologies, when language models such as Transformer process data, if the data length is very long, the amount of data in the KV Cache will continue to grow linearly, which will have a significant impact on the model performance. Summary of the invention

[0005] The present disclosure provides a data processing method and device, an electronic device, and a computer-readable storage medium.

[0006] In a first aspect, the present disclosure provides a data processing method, the method comprising:

[0007] Acquire key-value cache data of a first type according to a first storage space corresponding to the first network model; wherein the first storage space is used to store the key-value cache data generated by the first network model for historical input data;

[0008] Obtaining the key-value cache data of the second type according to the second storage space corresponding to the second network model; wherein the second storage space is used to transfer the key-value cache data stored in the first storage space when the amount of the key-value cache data stored in the first storage space reaches a preset value;

[0009] The current input data is processed according to the first type of key-value cache data and the second type of key-value cache data.

[0010] In a second aspect, the present disclosure provides a training method for a data processing model, wherein the data processing model includes: a first network model and a second network model; wherein the first network model includes: a first storage space for storing key-value cache data; the second network model includes: a second storage space; and the output end of the second network model is connected to the input end of the first network model; the method includes:

[0011] The data processing model in the initial state is trained by using a preset training data set to obtain a trained data processing model;

[0012] Among them, whenever it is detected that the amount of key-value cache data stored in the first storage space reaches a preset value, the key-value cache data stored in the first storage space is input into the second network model, and the first storage space is cleared; the stored key-value cache data is subjected to preset processing through the second network model, and the processing result is stored in the second storage space.

[0013] In a third aspect, the present disclosure provides a data processing device, the device comprising:

[0014] A first acquisition module, adapted to acquire a first type of key-value cache data according to a first storage space corresponding to a first network model; wherein the first storage space is used to store the key-value cache data generated by the first network model for historical input data;

[0015] a second acquisition module, adapted to acquire the second type of key-value cache data according to a second storage space corresponding to the second network model; wherein the second storage space is used to transfer the key-value cache data stored in the first storage space when the amount of key-value cache data stored in the first storage space reaches a preset value;

[0016] The processing module is adapted to process current input data according to the first type of key-value cache data and the second type of key-value cache data.

[0017] In a fourth aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned method.

[0018] In a fifth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above method when executed by a processor / processing core.

[0019] In the embodiment provided by the present disclosure, a second network model is further provided on the basis of the first network model. Accordingly, the key-value cache data in the first storage space can be transferred with the help of the second storage space corresponding to the second network model, thereby reducing the amount of key-value cache data in the first storage space, avoiding the problem of continuous linear growth of the amount of key-value cache data in the first storage space when the data length is long, thereby improving the operating performance of the model.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0022] Figure 1 A schematic diagram of a data processing method provided by an embodiment of the present disclosure is shown;

[0023] Figure 2 An example diagram showing the combination of the transformer model and the RNN model;

[0024] Figure 3 A schematic diagram of the data processing flow is shown;

[0025] Figure 4 A block diagram of a data processing device provided in an embodiment of the present disclosure;

[0026] Figure 5 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0029] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0030] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0031] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0032] In the related art, when the language model of the Transformer architecture processes data, if the data length is very long, it will cause the data volume of the KV Cache to continue to grow linearly, thereby causing a greater impact on the model performance. In order to solve the above problem, the embodiment of the present disclosure is further provided with a second network model on the basis of the first network model. Accordingly, the key-value cache data in the first storage space can be transferred with the help of the second storage space corresponding to the second network model, thereby reducing the data volume of the key-value cache data in the first storage space, avoiding the problem of continuous linear growth of the data volume of the key-value cache data in the first storage space when the data length is long, thereby improving the running performance of the model.

[0033] Figure 1 FIG. 1 is a flow chart of a data processing method provided by an embodiment of the present disclosure. Figure 1 As shown, the data processing method includes:

[0034] Step S110: acquiring key-value cache data of a first type according to a first storage space corresponding to the first network model; wherein the first storage space is used to store key-value cache data generated by the first network model for historical input data.

[0035] Among them, the first network model may include any data processing model that supports the key-value cache function. For example, the first network model may be a text conversion model including an attention module, such as a Transformer model. Correspondingly, the first storage space is used to store the key-value cache data generated by the first network model. Among them, the data volume of the key-value cache data stored in the first storage space increases dynamically with the increase of the data length of the historical input data. It can be seen that the data volume of the key-value cache data generated by the first network model for the historical input data stored in the first storage space continues to grow linearly with the increase of the length of the input data. In other words, the first network model does not perform overwrite operations or compression operations on the key-value cache data in the first storage space. Therefore, without intervening operations, the key-value cache data in the first storage space is always increasing linearly.

[0036] In this step, the first type of key-value cache data is obtained from the first storage space corresponding to the first network model. The first type of key-value cache data refers to: the original key-value cache data stored in the first storage space. It can be seen that the first type of key-value cache data is the key-value cache data directly generated by the first network model.

[0037] Step S120: Obtain the second type of key-value cache data according to the second storage space corresponding to the second network model; wherein the second storage space is used to transfer the key-value cache data stored in the first storage space when the amount of key-value cache data stored in the first storage space reaches a preset value.

[0038] In the present disclosure, when the amount of key-value cache data stored in the first storage space reaches a preset value, a transfer operation will be performed on the key-value cache data stored in the first storage space, so that the key-value cache data stored in the first storage space is cleared and transferred to the second storage space, thereby reducing the storage pressure of the first storage space and avoiding the problem of affecting the model running performance due to the excessive amount of data in the first storage space.

[0039] Among them, the key-value cache data from the first storage space transferred to the second storage space can be the key-value cache data after preset processing through the second network model. Specifically, the second network model is used to perform preset processing on the key-value cache data stored in the first storage space, and the specific processing method can be determined according to the model structure and model parameters of the second network model. This application does not limit the specific type and processing method of the second network model.

[0040] Step S130: Process the current input data according to the first type of key-value cache data and the second type of key-value cache data.

[0041] In this step, the first network model is not only processed according to the first type of key-value cache data stored in the first storage space, but also further processed in combination with the second type of key-value cache data stored in the second storage space. The second type of key-value cache data in the second storage space can share the pressure of the first storage space, thereby avoiding the problem of performance degradation of the first network model due to excessive data volume in the first storage space. In addition, the data volume of the key-value cache data is expanded with the help of the second storage space, thereby taking into account both the model operation performance and the comprehensiveness of the key-value cache data.

[0042] In the embodiment provided by the present disclosure, a second network model is further provided on the basis of the first network model. Accordingly, the key-value cache data in the first storage space can be transferred with the help of the second storage space corresponding to the second network model, thereby reducing the amount of key-value cache data in the first storage space, avoiding the problem of continuous linear growth of the amount of key-value cache data in the first storage space when the data length is long, thereby improving the operating performance of the model.

[0043] In addition, those skilled in the art may also make various modifications and variations to the embodiments of the present disclosure:

[0044] In an optional implementation, the amount of key-value cache data stored in the first storage space increases dynamically as the data length of historical input data increases. The first storage space may be, for example, a memory space or other storage space. Furthermore, the amount of key-value cache data stored in the second storage space is a fixed value. For example, the iterative compression capability of the second network model may be used to allow new data to continuously overwrite old data, thereby making the amount of data in the second storage space a fixed value, thereby avoiding linear growth as the data length of historical input data increases.

[0045] In an optional implementation, the second network model can be a recurrent neural network model, and accordingly, the above method further includes the following operations: whenever it is detected that the amount of key-value cache data stored in the first storage space reaches a preset value, the key-value cache data stored in the first storage space is input into the second network model, and the stored key-value cache data is subjected to preset processing by the second network model, and the processing results are stored in the second storage space. The specific processing logic of the preset processing can be determined according to the model parameters of the second network model. For example, the hidden layer state data corresponding to the stored key-value cache data can be obtained through the hidden layer in the second network model, and the hidden layer state data can be stored in the second storage space.

[0046] In an optional implementation, the second network model performs preset processing on the stored key-value cache data in the following manner: when the second network model receives the key-value cache data stored in the first storage space for the i-th time, it iteratively processes the key-value cache data received for the i-th time according to the i-1-th processing result stored in the second storage space to obtain the i-th processing result; the i-th processing result is updated and stored in the second storage space to cover the i-1-th processing result; wherein i is a natural number. Among them, the i-th time is this time, and the specific value of i is dynamically updated with the number of times the first storage space is cleared and data is transferred, the i-1-th time is the previous time, and when i is 1, the i-1-th processing result stored in the second storage space is empty. It can be seen that in the present disclosure, the processing results stored in the second storage space are continuously updated with the increase in the number of times the first storage space is cleared and data is transferred. Unlike the first storage space that continues to grow linearly, the second storage space is subject to the processing constraints of the second network model, and the space size is fixed. The data input next time will update and overwrite the data input last time, thereby realizing continuous iteration of data. In addition, because the second network model combines the processing results of the key-value cache data stored in the first storage space received last time when performing preset processing on the key-value cache data stored in the first storage space received this time, it can ensure that the key-value cache data after iteration retains some characteristics of the key-value cache data before iteration, thereby helping to improve the accuracy of data processing operations. In other words, the second storage space of the present application does not simply overwrite old data with new data, but processes new data based on the old data. Therefore, it can continue to retain some characteristics of the previous old data, thereby achieving data processing accuracy while saving storage space.

[0047] In an optional implementation, the current input data includes: the current text unit in the text sequence data, and the historical input data includes: the historical text unit located before the current text unit in the text sequence data. The text sequence data can be a text segment arranged in order, and the text unit can be a basic unit of model processing, such as tokens. Tokens are the basic units for large language models (LLMs) to process input text, and can be words, subwords, characters or other text fragments. Accordingly, the text is converted into a digital representation called a token for the model to process. The first network model is a text conversion model including an attention module, and the processing of the current input data includes: performing attention processing on the current input data through the attention module. The number of attention modules included in the first network model can be one or more, and the attention processing can include various methods such as self-attention processing and / or multi-head attention processing.

[0048] In an optional implementation, the spatial size of the second storage space can be determined according to a preset value. For example, the spatial size of the second storage space can match the preset value. And, in the case where there are multiple attention modules included in the first network model, the second network model may include: multiple recurrent neural network models corresponding to the multiple attention modules, and each recurrent neural network model corresponds to at least one second storage space. Among them, the recurrent neural network model includes: a single-layer recurrent neural network model, and / or a multi-layer recurrent neural network model. It can be seen that, in the case of multiple attention modules, each attention module can correspond to one recurrent neural network or multiple recurrent neural networks; or, multiple attention modules can correspond to one recurrent neural network, and the present application does not limit the specific correspondence.

[0049] In the present application, when multiple attention modules correspond to multiple recurrent neural network models, the multiple recurrent neural network models can be independent of each other and have no connection relationship with each other. Alternatively, multiple recurrent neural network models can also have a connection relationship. For example, in an optional implementation method, multiple recurrent neural network models corresponding to multiple attention modules have a connection relationship; wherein the output end of the recurrent neural network model corresponding to the attention module located in the upper layer is further connected to the input end of the recurrent neural network model corresponding to the attention module located in the lower layer. Correspondingly, in this method, when the recurrent neural network model corresponding to the attention module located in the lower layer performs the preset processing, it is further combined with the processing result corresponding to the recurrent neural network model corresponding to the attention module located in the upper layer (multiple attention modules are processed in sequence from bottom to top, of course, they can also be processed in sequence from top to bottom). In this way, the recurrent neural network model corresponding to the attention module located in the lower layer can learn the processing result of the recurrent neural network model corresponding to the attention module located in the upper layer, thereby helping to improve the accuracy and comprehensiveness of the final processing result. For example, the output end of the recurrent neural network model corresponding to the attention module located at the mth layer is further connected to the input end of the recurrent neural network model corresponding to the attention module located at the m+1th layer; and, when the recurrent neural network model corresponding to the attention module located at the m+1th layer performs preset processing, it is further implemented in combination with the processing result of the recurrent neural network model corresponding to the attention module located at the mth layer; wherein m is a natural number.

[0050] In an optional implementation, the first network model may include: a transformer model, and the second network model includes: a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, and / or a gated recurrent unit (GRU) model.

[0051] Another embodiment of the present disclosure also provides a training method for a data processing model, wherein the data processing model includes: a first network model and a second network model; wherein the first network model includes: a first storage space for storing key-value cache data; the second network model includes: a second storage space; and the output end of the second network model is connected to the input end of the first network model; the method includes: training the data processing model in the initial state through a preset training data set to obtain a trained data processing model; wherein, whenever it is detected that the amount of key-value cache data stored in the first storage space reaches a preset value, the key-value cache data stored in the first storage space is input into the second network model, and the first storage space is cleared; the stored key-value cache data is subjected to preset processing through the second network model, and the processing result is stored in the second storage space. The specific process of the preset processing can refer to the description of the previous embodiment.

[0052] It can be seen that by connecting the output end of the second network model to the input end of the first network model, the first network model and the second network model can be jointly trained, so that the preset processing method in the trained second network model matches the data processing requirements in the actual business, thereby improving the overall performance of the data processing model.

[0053] In an optional implementation, the data processing model in the initial state includes: a first network model in the initial state, and a second network model in the initial state; and training the data processing model in the initial state to obtain the trained data processing model includes: jointly training the first network model in the initial state and the second network model in the initial state to obtain the trained data processing model. In this method, the first network model and the second network model are both untrained initial models, and the two are jointly trained together to obtain the final data processing model.

[0054] In another optional implementation, the data processing model in the initial state includes: a first network model obtained by pre-training, and a second network model in the initial state; and the preset training data set is a calibration data set; and the data processing model in the initial state is trained to obtain the trained data processing model, including: training the model parameters of the second network model through the calibration data set to obtain the trained data processing model. In this method, the first network model is a pre-trained model, so the focus is on training the second network model, thereby improving the training efficiency. Optionally, when training the data processing model in the initial state, the parameters of the pre-trained first network model can be further adjusted through the calibration data set to obtain the trained data processing model. In other words, in this method, the parameters of the pre-trained first network model can also be fine-tuned to improve the accuracy of the model.

[0055] Among them, in the training process, various types of loss functions and convergence conditions can be flexibly selected. In addition, the present disclosure does not limit the specific training method of the model, and those skilled in the art can flexibly select according to needs.

[0056] For ease of understanding, an example is used below to introduce the implementation of the model processing method provided in an embodiment of the present disclosure.

[0057] First, let's introduce the technical implementation scheme related to this example: Large Model (LM), especially Large Language Model (LLM) such as GPT-3 (Generative Pre-trained Transformer-3) and GLM (General Language Model), has a huge number of parameters exceeding 100 billion, which brings great challenges to deployment and application. Most large models are implemented based on the transformer model architecture. The Transformer model contains many isomorphic layers, each of which contains an attention layer and a feed forward network (FFN) layer. The attention layer mainly processes the correlation of the token dimension (i.e., the time dimension), so it can be considered to mainly involve short-term memory. The forward connection layer mainly processes the correlation between the dimensions of a single token, so it can be considered to mainly involve long-term memory. Among them, the attention layer generally performs a linear transformation on the input vector, generating three vectors: query (query vector), key (key value vector), and value (value vector), and performs a self-attention operation, and then a linear transformation to generate output. The forward connection layer is generally composed of two layers of linear transformation layers. The role of the first linear transformation layer is similar to the matrix composed of keys. Under the query of the input vector (similar to the query), it outputs the degree of matching and then weights the second linear transformation layer (similar to the value) to obtain the output.

[0058] When a large language model is inferencing, the output result is serial and autoregressive, that is, the decoded output token will be fed back to the input layer of the model, and then decoded to generate the next token, and the cycle repeats until the termination condition is met. During the decoding process of a single token, the key and value vectors of the previously processed tokens at each layer can be saved to avoid recalculation and reduce the amount of calculation, which is generally called kv cache. This part of the kv cache is the main carrier of short-term memory and stores the information of each token in a certain context.

[0059] However, as the length of the conversation increases, the kv cache will expand linearly, causing a huge storage and communication burden. Moreover, in applications such as personal assistants that need to record user conversations and other information (large time scales, long conversation sequences), the need to convert short-term memory into long-term memory in large language models is further increased. On the other hand, the linear growth of kvcache will cause the amount of computation required to generate each token to also increase linearly, seriously increasing the computational burden of the model.

[0060] In related technologies, in order to convert the short-term memory of a large model into long-term memory, or to compress the kv cache, it can usually be achieved in the following ways:

[0061] In the first implementation, the kv cache can be quantized. Specifically, the kv cache is quantized to low-bit storage, such as int8, int4, and even int2. However, this method reduces the accuracy of the kv cache but cannot reduce the length.

[0062] In the second implementation, sparse attention can be used. Specifically, when processing the attention of a single token to past tokens (i.e., calculating the linear comparison between the query of the current token and the key of other tokens), not all tokens are involved, but only a part is selected according to certain rules. However, this type of method generally causes a significant performance degradation.

[0063] In the third implementation method, it can be implemented by storing past conversation records and other materials. Specifically, the conversation records and other materials are stored in the database in the original form such as text, and then extracted when used. However, this method requires the model to reprocess the data, resulting in additional costs, and also requires an additional retrieval module.

[0064] In the fourth implementation, it can be achieved through fine-tuning. Specifically, the conversation records are used to train the model through fine-tuning so that the model can remember past data. However, this method is costly and easily causes forgetting.

[0065] It can be seen that the related art lacks a solution that can effectively solve the problem of model performance degradation caused by the linear growth of kv cache. In order to solve the above problems, in this example, a recurrent neural network RNN ​​model is further introduced on the basis of the transformer model. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer unconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. Among them, the state in RNN refers to the memory or internal state of the network at a certain time step when processing sequence data. This state is one of the core features of RNN, which enables the network to remember previous information and use this information when processing the current input. Specifically, each time step of the RNN receives two inputs: (1) the current input data (e.g., a word in a sentence). In this example, the current input data mainly refers to the stored key-value cache data from the transformer model received by the input layer of the RNN model this time; (2) the hidden state of the previous time step. In this example, the hidden state of the previous time step mainly refers to the processing result obtained after the RNN model performs preset processing on the stored key-value cache data received from the transformer model last time. Based on these two inputs, the RNN calculates the output of the current time step and the updated hidden state. This hidden state (usually represented as H(t)) contains the network's summary of all previous inputs and can be regarded as the network's "memory". This state is then passed to the next time step as part of the input of that time step, thereby achieving continuous information transmission. In practical applications of RNN, output usually refers to the immediate result of each time step, such as a prediction of a word or an estimate of a value. The state is the information maintained within the network and is used to transfer memory between different time steps of the sequence. This structure makes RNN very suitable for processing sequence data, such as speech recognition, natural language processing and other tasks, because such tasks need to consider the time or order relationship in the data.

[0066] The core of this example is to combine the transformer model (i.e., the first network model) and the RNN model (i.e., the second network model) to compress the short-term memory with the kv cache as the main carrier to support the processing of long dialogue sequences. Among them, the transformer model is used to maintain an upper limit of the kv cache capacity. When the amount of data in the kv cache exceeds the upper limit, an RNN (whose state is the kv cache size) is used to iterate once to compress the newly generated kv cache into the hidden state of the RNN.

[0067] Figure 2 An example diagram showing the combination of the transformer model and the RNN model is shown. Figure 2 (a) shows the case where there is no connection between the RNN models between different transformer layers. Figure 2 (b) in the figure shows the case where there are connections between the RNN models of different transformer layers. Figure 2 As shown, the transformer model includes two transformer layers, each of which can be an attention layer in the transformer model or a composite layer consisting of an attention layer and a forward connection layer. This example does not limit the specific implementation method of each transformer layer.

[0068] exist Figure 2 In the manner shown, each transformer layer corresponds to an RNN model, and each RNN model includes two RNN layers (i.e., the RNN model is a multi-layer recurrent neural network model). In other implementations, two transformer layers may share the same RNN model. Alternatively, each transformer layer may be connected to multiple RNN models. Furthermore, each RNN model may be a single-layer recurrent neural network model (i.e., only one RNN layer), or a multi-layer recurrent neural network model including three or even more RNN layers. In short, this example does not limit the specific implementation details.

[0069] It should be noted that regardless of Figure 2In terms of which implementation method, when the RNN model is a multi-layer recurrent neural network model, the output result or state result of the last RNN layer in the multiple RNN layers will participate in the self-attention operation of the corresponding transformer layer. In other words, for each transformer layer, the input end of the transformer layer is further connected to the output end of the last RNN layer in the corresponding RNN model. In addition, the input end of the first RNN layer of the RNN model is used to receive the key-value cache data generated by the corresponding transformer layer.

[0070] Among them, Figure 2 In the case where there is no connection between the RNN models corresponding to different transformer layers shown in (a), the input of the RNN model corresponding to each transformer layer only includes the kv cache generated by the transformer layer. This connection method helps to improve data processing efficiency.

[0071] exist Figure 2 In the case where the RNN models corresponding to different transformer layers shown in (b) are connected, the input of the RNN model corresponding to each transformer layer includes not only the kv cache generated by the transformer layer, but also the output result of the RNN model corresponding to the previous transformer layer of the transformer layer. Among them, when the transformer layer is the m+1th transformer layer, the previous transformer layer of the transformer layer is the mth transformer layer. This connection method further considers the correlation between the kv caches corresponding to multiple transformer layers, so the accuracy of data processing is better. Among them, when there are connections between multiple RNN models and they are trained layer by layer, a method of training from front to back can be adopted. In addition, the output result of each RNN model is used to participate in the self-attention operation of the transformer layer corresponding to the RNN model.

[0072] It can be seen that the data processing model in this example mainly includes a transformer model or its variant, and an RNN model for kv cache compression. Among them, the hidden state size of the RNN model (i.e., the size of the second storage space) can be the maximum storage capacity of the kv cache corresponding to the preset transformer model (i.e., the preset maximum storage capacity of the first storage space). For example, if the preset maximum storage length of the first storage space is 1024, that is, the maximum kvcache length in the first storage space is 1024, and the dimension of the kv cache is 4096 (this dimension is usually the same as the dimension of the transformer hidden layer), then the hidden state size of the RNN model is 1024*4096*2 (*2 is because there are two parts, k cache and v cache). It can be seen that the hidden state size of the RNN model usually matches the preset maximum storage capacity of the first storage space (such as the two are equal). It can be seen that when the first network model is a transformer model and the second network model is a recurrent neural network model, the spatial size of the second storage space matches the maximum storage capacity of the first storage space. Among them, the spatial size of the second storage space is usually associated with the hidden state size of the RNN model. Furthermore, the matching may include: the two are equal, or the space size of the second storage space is slightly larger than the maximum storage capacity of the first storage space, etc. In short, setting the space size of the second storage space according to the preset maximum storage capacity of the first storage space can improve space utilization.

[0073] Among them, k cache and v cache can be compressed by the same RNN model, or k cache and v cache can be compressed by two RNN models respectively. If two RNN models are used for compression, the hidden layer dimension of each RNN model is 1024*4096.

[0074] In addition, each transformer layer can use the same RNN model for compression (that is, each transformer layer shares the same RNN model), or different RNN models for compression (that is, multiple transformer layers use multiple different RNN models respectively), or partially identical RNN models for compression (that is, some transformer layers in multiple transformer layers share the same RNN model).

[0075] The RNN model in this example refers to a recurrent neural network with feedback connections, including but not limited to the classic RNN model, LSTM (long short term memory) model, GRU (gated recurrent unit) model and other variants. The common feature of these models is that the hidden state size is constant and does not increase with the increase of sequence length.

[0076] Figure 3 The following is a schematic diagram of the data processing flow. Figure 3 The implementation process of data processing (i.e., model reasoning process) through the above-mentioned data processing model is introduced as follows:

[0077] First, the RNN model performs state initialization, which can be done by all-zero initialization. Then, when the transformer model processes a token, it not only needs to perform attention operations with the kv cache generated by the transformer model itself (located in the first storage space mentioned above), but also needs to perform attention operations with the hidden state of the RNN model (i.e., the kv cache compression state, corresponding to the "second type of kv cache data stored in the second storage space" described in the figure). Therefore, the kv cache that needs to be operated on for attention in this example includes both the hidden state of the RNN model (corresponding to the second type of kv cache data) and the kv cache generated by the transformer model (corresponding to the first type of kv cache data). It can be seen that the attention module needs to perform attention operations based on the first type of key-value cache data stored in the first storage space and the second type of key-value cache data stored in the second storage space to obtain the attention processing result of the current input data.

[0078] It can be seen that during the execution of this example, when the amount of the first type of kvcache data that has not been emptied in the first storage space is 0, the amount of kv cache data that needs to be paid attention to is the least, which is limited to the second type of kv cache data. The specific amount of data is usually the hidden state size of the RNN model (corresponding to the maximum kv cache capacity preset in the second storage space). This situation mainly corresponds to the moment when the data processing model processes the first token (at this time, the second type of kv cache data may actually be empty), or the moment when the first storage space has just completed the clearing operation (at this time, the first type of kv cache stored in the first storage space has been compressed into the RNN model).

[0079] In addition, when the amount of first-type kv cache data that has not been emptied in the first storage space is about to reach the preset maximum storage capacity of the first storage space, the amount of kv cache data that needs to be attended to is the largest, which not only includes the second-type kv cache data, corresponding to the hidden state size of the RNN model; it also further includes the first-type kv cache that has not been emptied in the first storage space (whose maximum data amount is approximately equal to the preset maximum storage capacity of the first storage space), that is, the maximum value of the amount of kv cache data that needs to be attended to is approximately twice the preset maximum storage capacity of the first storage space (that is, the RNN hidden state + the kv cache generated by the transformer).

[0080] like Figure 3 As shown in the figure, when processing the first token, the transformer layer needs to perform an attention operation with the RNN hidden layer state (i.e., the second type of kv cache data stored in the second storage space). At this time, the second type of kv cache data stored in the second storage space is actually empty or zero. When processing the second token, it is necessary to perform an attention operation with the RNN hidden layer state and the kv cache generated by the first token (i.e., the first type of kv cache data that has not been cleared in the first storage space)... and so on, until the kv cache generated by the transformer reaches the preset maximum capacity (the maximum length is 4 as an example in the figure), the RNN model iterates once, compresses the kv cache generated by the transformer (at this time, the same size as the RNN hidden layer state) into the RNN hidden layer state, and clears the transformer's kv cache.

[0081] In other optional implementations, the data involved in the attention operation includes not only the current RNN hidden layer state, but also the RNN hidden layer state within a certain time window, such as the current RNN state and the RNN state of the previous iteration. Among them, the RNN state of the previous iteration can be stored through an additional storage space. In this way, more state data can be retained, thereby improving the accuracy of data processing. Accordingly, after the above-mentioned update and storage of the i-th processing result to the second storage space to cover the i-1th processing result, the i-1th processing result can be further cached to the third storage space. Accordingly, the attention module performs attention operation processing based on the first type of key-value cache data, the second type of key-value cache data, and the cache data in the third storage space mentioned above. Similarly, a fourth storage space can be further introduced to cache the i-2th processing results, or even a fifth storage space can be introduced to cache the i-3th processing results, so that the attention module can perform attention operation processing based on the first type of key-value cache data, the second type of key-value cache data, the cache data in the third storage space, the cache data in the fourth storage space and / or the cache data in the fifth storage space mentioned above.

[0082] For example, in an optional implementation, after the i-th processing result is updated and stored in the second storage space to cover the i-1th processing result, the i-1th to in-th processing results can be further cached in the third storage space; wherein n is greater than or equal to 2. In short, by caching the processing results in more time windows (i.e., the RNN hidden layer state), the results of the attention operation can be made more accurate. In actual situations, the length of the time window can be flexibly set in combination with specific business needs to take into account both the accuracy and speed of the operation.

[0083] The following describes the model training process of the above data processing model:

[0084] In one training method, training can be done from scratch, that is, the hybrid model composed of RNN and transformer is trained from scratch on a certain pre-training data set and trained according to the above reasoning process.

[0085] In another training method, the pre-trained transformer is directly used, so only the RNN needs to be trained. The transformer has been pre-trained, so the parameters of the RNN are only trained on a certain calibration data set. The specific objective function and optimization method can refer to the trained transformer, that is, before and after compression, the performance of the transformer on the given task does not decrease, or decreases slightly.

[0086] In another training method, a few parameters are fine-tuned on the pre-trained transformer and the RNN is trained. In addition to training the RNN, the transformer is also fine-tuned, including but not limited to few parameter fine-tuning, such as lora (lowrank adaptation), adding an adapter, or full parameter fine-tuning.

[0087] The constraint method can be the final output of the transformer, or the hidden layer output of the transformer can be constrained layer by layer. This example does not limit the specific constraint method.

[0088] In summary, this example can dynamically compress the kv cache (short-term memory) of a large language model, reducing the storage, communication, and computing burdens, and can support the processing of ultra-long time scales and dialogue sequences. This example can also be extended to large models of other modalities, such as large visual models, multimodal large models, etc.

[0089] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0090] In addition, the present disclosure also provides a model training device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any model training method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0091] Figure 4 A block diagram of a data processing device provided in an embodiment of the present disclosure.

[0092] Reference Figure 4 , an embodiment of the present disclosure provides a data processing device, the device comprising:

[0093] A first acquisition module 41 is adapted to acquire a first type of key-value cache data according to a first storage space corresponding to a first network model; wherein the first storage space is used to store key-value cache data generated by the first network model for historical input data;

[0094] The second acquisition module 42 is adapted to acquire the second type of key-value cache data according to the second storage space corresponding to the second network model; wherein the second storage space is used to transfer the key-value cache data stored in the first storage space when the amount of the key-value cache data stored in the first storage space reaches a preset value;

[0095] The processing module 43 is adapted to process current input data according to the first type of key-value cache data and the second type of key-value cache data.

[0096] In an optional implementation, the data volume of the key-value cache data stored in the first storage space increases dynamically as the data length of the historical input data increases; and the data volume of the key-value cache data stored in the second storage space is a fixed value.

[0097] In an optional implementation, the second acquisition module is further used to:

[0098] Whenever it is detected that the amount of key-value cache data stored in the first storage space reaches a preset value, the key-value cache data stored in the first storage space is input into the second network model, and the stored key-value cache data is subjected to preset processing through the second network model, and the processing result is stored in the second storage space.

[0099] In an optional implementation, when the second network model receives the key-value cache data stored in the first storage space for the i-th time, iteratively processes the key-value cache data received for the i-th time according to the i-1-th processing result stored in the second storage space to obtain the i-th processing result;

[0100] The i-th processing result is updated and stored in the second storage space to cover the i-1-th processing result; wherein i is a natural number.

[0101] In an optional implementation, the current input data includes: a current text unit in the text sequence data, and the historical input data includes: a historical text unit in the text sequence data that is located before the current text unit;

[0102] The first network model is a text conversion model including an attention module, and the processing of the current input data includes: performing attention processing on the current input data through the attention module.

[0103] In an optional implementation, the space size of the second storage space is determined according to the preset value; and, in the case where the first network model includes multiple attention modules, the second network model includes: multiple recurrent neural network models corresponding to the multiple attention modules, and each recurrent neural network model corresponds to at least one second storage space;

[0104] Wherein, the recurrent neural network model includes: a single-layer recurrent neural network model and / or a multi-layer recurrent neural network model.

[0105] In an optional implementation, multiple recurrent neural network models corresponding to multiple attention modules are connected to each other; wherein the output end of the recurrent neural network model corresponding to the attention module located at the m-th layer is further connected to the input end of the recurrent neural network model corresponding to the attention module located at the m+1-th layer; and, when executing preset processing, the recurrent neural network model corresponding to the attention module located at the m+1-th layer is further implemented in combination with the processing result of the recurrent neural network model corresponding to the attention module located at the m-th layer; wherein m is a natural number.

[0106] In an optional implementation, the first network model includes: a Transformer model, and the second network model includes: a recurrent neural network model, a long short-term memory network model, and / or a gated recurrent unit model.

[0107] Figure 5 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0108] Reference Figure 5 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 901; at least one memory 902, and one or more I / O interfaces 903, connected between the processor 901 and the memory 902; wherein the memory 902 stores one or more computer programs that can be executed by at least one processor 901, and the one or more computer programs are executed by at least one processor 901 so that the at least one processor 901 can execute the above-mentioned model training method.

[0109] In some embodiments, the processing device may be a brain-like chip, which can use vectorized computing and needs to load the weight information and other parameters of the neural network model through an external memory such as a double data rate (DDR) synchronous dynamic random access memory. Therefore, the batch processing used in the embodiments of the present disclosure has a higher computing efficiency.

[0110] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned model training method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0111] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned model training method.

[0112] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0113] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0114] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0115] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0116] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0117] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0119] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0120] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0121] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A data processing method, the method comprising: Acquire key-value cache data of a first type according to a first storage space corresponding to the first network model; wherein the first storage space is used to store the key-value cache data generated by the first network model for historical input data; Obtaining the key-value cache data of the second type according to the second storage space corresponding to the second network model; wherein the second storage space is used to transfer the key-value cache data stored in the first storage space when the amount of the key-value cache data stored in the first storage space reaches a preset value; The current input data is processed according to the first type of key-value cache data and the second type of key-value cache data.

2. The method according to claim 1, wherein: The data volume of the key-value cache data stored in the first storage space increases dynamically as the data length of the historical input data increases; and the data volume of the key-value cache data stored in the second storage space is a fixed value.

3. The method according to claim 2, wherein: The method further comprises: Whenever it is detected that the amount of key-value cache data stored in the first storage space reaches a preset value, the key-value cache data stored in the first storage space is input into the second network model, and the stored key-value cache data is subjected to preset processing through the second network model, and the processing result is stored in the second storage space.

4. The method according to claim 3, wherein: The performing preset processing on the stored key-value cache data by using the second network model includes: When the second network model receives the key-value cache data stored in the first storage space for the i-th time, iteratively processes the key-value cache data received for the i-th time according to the i-1-th processing result stored in the second storage space to obtain the i-th processing result; The i-th processing result is updated and stored in the second storage space to cover the i-1-th processing result; wherein i is a natural number.

5. The method according to claim 4, wherein: The current input data includes: a current text unit in the text sequence data, and the historical input data includes: a historical text unit in the text sequence data that is located before the current text unit; The first network model is a text conversion model including an attention module, and the processing of the current input data includes: performing attention processing on the current input data through the attention module.

6. The method according to claim 5, wherein: The performing attention processing on the current input data by the attention module includes: The attention module performs attention operation processing according to the first type of key-value cache data and the second type of key-value cache data to obtain an attention processing result of the current input data.

7. The method according to claim 6, wherein: After the i-th processing result is updated and stored in the second storage space to cover the i-1th processing result, the method further includes: caching the i-1th to inth processing results in a third storage space; wherein n is greater than or equal to 2; Then the attention module performs attention operation processing according to the first type of key-value cache data and the second type of key-value cache data, including: The attention module performs attention operation processing according to the first type of key-value cache data, the second type of key-value cache data, and the cache data in the third storage space.

8. The method according to claim 5, wherein: The space size of the second storage space is determined according to the preset value; and, in the case where the first network model includes multiple attention modules, the second network model includes: multiple recurrent neural network models corresponding to the multiple attention modules, and each recurrent neural network model corresponds to at least one second storage space; Wherein, the recurrent neural network model includes: a single-layer recurrent neural network model and / or a multi-layer recurrent neural network model.

9. The method according to claim 8, wherein: There is a connection relationship between the multiple recurrent neural network models corresponding to the multiple attention modules; wherein the output end of the recurrent neural network model corresponding to the attention module located at the m-th layer is further connected to the input end of the recurrent neural network model corresponding to the attention module located at the m+1-th layer; and, when executing the preset processing, the recurrent neural network model corresponding to the attention module located at the m+1-th layer is further combined with the processing result of the recurrent neural network model corresponding to the attention module located at the m-th layer; wherein m is a natural number.

10. The method according to any one of claims 1 to 4, wherein: The first network model includes: a transformer model, and the second network model includes: a recurrent neural network model, a long short-term memory network model, and / or a gated recurrent unit model.

11. The method according to claim 10, wherein: When the first network model is a transformer model and the second network model is a recurrent neural network model, the spatial size of the second storage space matches the maximum storage capacity of the first storage space.

12. A method for training a data processing model, wherein: The data processing model includes: a first network model and a second network model; wherein the first network model includes: a first storage space for storing key-value cache data; the second network model includes: a second storage space; and the output end of the second network model is connected to the input end of the first network model; the method includes: The data processing model in the initial state is trained by using a preset training data set to obtain a trained data processing model; Among them, whenever it is detected that the amount of key-value cache data stored in the first storage space reaches a preset value, the key-value cache data stored in the first storage space is input into the second network model, and the first storage space is cleared; the stored key-value cache data is subjected to preset processing through the second network model, and the processing result is stored in the second storage space.

13. The method according to claim 12, wherein: The data processing model in the initial state includes: a first network model in the initial state, and a second network model in the initial state; and the training of the data processing model in the initial state to obtain the trained data processing model includes: jointly training the first network model in the initial state and the second network model in the initial state to obtain the trained data processing model; or The data processing model in the initial state includes: a first network model obtained by pre-training, and a second network model in the initial state; and the preset training data set is a calibration data set; and the training of the data processing model in the initial state to obtain the trained data processing model includes: training the model parameters of the second network model through the calibration data set to obtain the trained data processing model.

14. The method according to claim 13, wherein: The training of the data processing model in the initial state to obtain the trained data processing model further includes: adjusting parameters of the first network model obtained by pre-training through the calibration data set to obtain the trained data processing model.

15. A data processing device, comprising: A first acquisition module, adapted to acquire a first type of key-value cache data according to a first storage space corresponding to a first network model; wherein the first storage space is used to store the key-value cache data generated by the first network model for historical input data; a second acquisition module, adapted to acquire the second type of key-value cache data according to a second storage space corresponding to the second network model; wherein the second storage space is used to transfer the key-value cache data stored in the first storage space when the amount of key-value cache data stored in the first storage space reaches a preset value; The processing module is adapted to process current input data according to the first type of key-value cache data and the second type of key-value cache data.

16. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program implements the method according to any one of claims 1 to 14 when executed by a processor.

Images