A data processing method and related apparatus thereof

Abstract

The application discloses a data processing method and a related device thereof, and relates to the technical field of data processing. The method comprises the following steps: acquiring first key-value data corresponding to each first word element based on the first word element; storing each first key-value data in a cache; acquiring a second word element based on each first word element; and transmitting a plurality of second key-value data in the cache to a memory based on the second word element. The second key-value data in the memory is used for participating in attention calculation to obtain the attention weight of the second word element. The first key-value data corresponding to all the first word elements is stored in the cache, and when the attention weight of any second word element is calculated, only the required second key-value data needs to be retrieved from the cache, that is, the corresponding first key-value data does not need to be recalculated, the overhead of repeated generation of the key-value data is completely avoided, and the calculation cost of long sequence processing is significantly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data processing method and a related device thereof. BACKGROUND

[0002] The traditional attention mechanism encounters technical bottlenecks of memory capacity and computing performance when processing long sequences. The attention mechanism calculates the similarity between the query vector and the key vector to weight and sum the value vector. The time complexity and space complexity are both n² (n is the number of tokens in the long sequence), which results in extremely high computing cost and memory occupation when processing long sequences. For example, a sequence with 1000 tokens requires about 1 million floating point operations and stores about 1 million attention scores.

[0003] The prior art improves the traditional attention mechanism to reduce the computing cost and memory occupation of long sequences, for example, the sparse attention mechanism and the block attention mechanism can both reduce the computing cost and memory occupation of long sequences. It should be noted that the sparse attention mechanism reduces the amount of calculation and memory occupation by only focusing on some key tokens in the sequence, so some context information may be lost because not all tokens participate in the calculation. The block attention mechanism divides the sequence into multiple blocks, and each query item only calculates the attention weight with the keys and values in the same block. Since the information between blocks cannot be fused, the Transformer model may not be able to capture the long-distance dependency relationship across blocks. That is, the improved attention mechanism of the prior art sacrifices the ability of the Transformer model to capture the integrity of long sequence information in order to reduce the computing cost and memory occupation when processing long sequences. SUMMARY

[0004] The purpose of the present application is to provide a data processing method and a related device thereof to solve the problem of how to reduce the computing cost and memory occupation of long sequence processing without changing the algorithm of the traditional attention mechanism, that is, without sacrificing the ability to capture the integrity of long sequence information, by improving the transmission and storage method of the data processing process.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a technical solution of a data processing method applied to a tensor processor, wherein the tensor processor includes a memory and a cache. The data processing method includes:

[0007] Based on a token sequence, a plurality of first tokens are obtained; the first token is any token in the token sequence;

[0008] obtain, based on each first word piece, first key-value data corresponding to each first word piece; the first key-value data is data formed by a key vector and a value vector corresponding to the first word piece;

[0009] store each first key-value data in the cache;

[0010] obtain, based on each first word piece, a second word piece; the second word piece is any word piece for which attention weight is not calculated between each first word piece;

[0011] based on the second word piece, transmit a plurality of second key-value data in the cache to the memory; the second key-value data is any key-value data used to calculate the attention weight of the second word piece in the cache;

[0012] the second key-value data in the memory is used to participate in attention calculation to obtain the attention weight of the second word piece.

[0013] In a second aspect, a data processing device is provided, which is applied to a tensor processor including a memory and a cache. The data processing device includes:

[0014] a reading module configured to obtain a plurality of first word pieces based on a word piece sequence; the first word piece is any word piece in the word piece sequence;

[0015] a processing module configured to obtain, based on each first word piece, first key-value data corresponding to each first word piece; the first key-value data is data formed by a key vector and a value vector corresponding to the first word piece;

[0016] and store each first key-value data in the cache;

[0017] and obtain, based on each first word piece, a second word piece; the second word piece is any word piece for which attention weight is not calculated between each first word piece;

[0018] and based on the second word piece, transmit a plurality of second key-value data in the cache to the memory; the second key-value data is any key-value data used to calculate the attention weight of the second word piece in the cache; the second key-value data in the memory is used to participate in attention calculation to obtain the attention weight of the second word piece.

[0019] In a third aspect, the present application provides a technical solution of a computing device, which comprises a processor and a memory; the processor is connected with the memory, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program to enable the computing device to perform the data processing method according to the first aspect.

[0020] In a fourth aspect, the present application provides a technical solution of a computer readable storage medium, which stores a computer program; when the computer program is executed by a processor, the data processing method according to the first aspect is implemented.

[0021] Compared with the prior art, the present application has the following beneficial effects:

[0022] In the present application, all the first key-value data corresponding to all the first word pieces are stored in the cache, and when the attention weight of any second word piece is calculated, only the required second key-value data needs to be retrieved from the cache, that is, the corresponding first key-value data does not need to be calculated again, thereby completely avoiding the overhead of repeated generation of key-value data and significantly reducing the calculation cost of long sequence processing. Moreover, by pre-storing the key-value data (i.e., the first key-value data) in the cache and then transmitting the key-value data (i.e., the second key-value data) to the memory when needed, the two core pain points of repeated calculation and high memory occupation in the prior art are solved without sacrificing the integrity capturing capability of long sequence information. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of a data processing method according to an embodiment of the present application is shown in the figure.

[0024] Figure 2 A structural diagram of a data processing device according to an embodiment of the present application is shown in the figure.

[0025] Figure 3 A structural diagram of a server according to an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0026] The terms "first", "second", and the like in the description and in the claims of the present application and the above drawings are used to distinguish similar objects (for example, the first word and the second word are respectively represented as different word, and other similar objects) and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the names used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or modules does not have to be limited to those steps or modules clearly listed, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules in the embodiments of the present application is only a logical division, and in actual application, other division methods can be used, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling between modules, communication connection can be electrical or other similar forms, which are not limited in the embodiments of the present application. In addition, the modules or sub-modules described as separate components can be or can not be physically separated, can be or can not be physical modules, or can be distributed to a plurality of circuit modules, and some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0027] The scheme provided by the embodiments of the present application relates to technologies such as artificial intelligence (AI) and machine learning (ML), which are specifically explained as follows:

[0028] Among them, AI is to simulate, extend and expand human intelligence by using digital computers or machines controlled by digital computers, to perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0029] AI technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0030] ML is an important branch of AI, which focuses on developing algorithms that allow computers to automatically learn and improve through data without being explicitly programmed to perform specific tasks. In simple terms, the core of machine learning is to let machines "learn" rules from experience (data), and then use these rules to make predictions or decisions on new unknown data.

[0031] In order to enable those skilled in the art to understand the embodiments of the present application, it is also necessary to clearly understand the definitions of the following professional terms:

[0032] Attention mechanism: Attention mechanism is an artificial intelligence technology that simulates human "selective attention" ability. Its core idea is that when processing information, different parts are given different attention weights. That is, important information has high attention weight and is processed first; irrelevant or secondary information has low attention weight to reduce resource consumption, thereby improving the efficiency of capturing key information and task performance of the Transformer model.

[0033] Transformer model: Transformer model is a deep learning architecture proposed by Vaswani et al. in 2017. The core breakthrough of the Transformer model is to completely rely on attention mechanism to replace traditional recurrent neural networks (such as RNN or LSTM, etc.) to process sequence data, which has completely changed the field of natural language processing (NLP). The Transformer model architecture is divided into encoder and decoder, both of which are stacked by multiple identical modules. The encoder is responsible for extracting the context semantics of the input sequence (such as the text to be translated or user input text, etc.); the decoder combines the encoder output to generate the target sequence (such as the translation result or the result the user wants to get, etc.). Compared with traditional recurrent neural networks, the Transformer model can be calculated in parallel and efficiently capture long-distance dependencies, becoming the basis of large models such as BERT and GPT, and is widely used in machine translation, text generation, question and answer systems, etc. Application scenarios.

[0034] Token Sequence: Token Sequence is the sequence formed by the ordered arrangement of the minimum processing unit (i.e., Token) of text data in natural language processing. Token Sequence is the basic input form for AI models (e.g., Transformer or RNN, etc.) to understand and process text. In simple terms, the core of Token Sequence is to first break down the text into the smallest semantic or grammatical unit (i.e., Token), and then arrange them in order. For example, the sentence "I love natural language processing" may be split into Token Sequence as ["I", "love", "natural language processing"] (i.e., in word units) or ["I", "love", "self", "natural", "language", "processing"] (i.e., in character units), depending on the "Tokenization Strategy".

[0035] Tensor Processing Unit (TPU): Tensor Processing Unit is a customized Application-Specific Integrated Circuit chip designed specifically for machine learning tasks. The core goal of TPU is to optimize the hardware architecture for matrix operations to provide energy efficiency and computational density far superior to Central Processing Unit (CPU) and Graphics Processing Unit (GPU). It adopts a Systolic Array architecture, driven by data flow, to achieve efficient pipeline computation of matrix multiplication, reducing memory access delay. At the same time, TPU supports low-precision computing such as BF16, FP16, INT8, etc., which can reduce the number of transistors required for each operation without significantly sacrificing accuracy, thereby reducing energy consumption.

[0036] Random Access Memory (RAM): Random Access Memory is the core hardware in a computer for temporarily storing data and programs, and is the key bridge between TPU and external storage devices (e.g., mechanical hard disk or solid state disk). Its core feature is random access, i.e., TPU can directly read or write data at any address without the need to retrieve it in order, which makes its read / write speed much faster than external storage devices.

[0037] Cache: Cache is a high-speed temporary storage component in the computer system, and its core role is to solve the problem of speed mismatch between high-speed processors (such as CPU, TPU or GPU, etc.) and low-speed memory or storage devices, avoid the processor idle due to waiting for data, and improve the overall running efficiency. The working logic of the cache is based on the principle of locality, that is, when the computer runs, the data or instructions frequently accessed by the processor are often concentrated in a small area (such as frequently called program code or recently processed file fragments, etc.). Cache will copy these high-frequency data from low-speed storage devices in advance, when the processor needs high-frequency data again, it does not need to be read slowly by the low-speed storage device, but directly from the cache with much faster speed, greatly shortening the data waiting time.

[0038] In order to solve the problem of how to reduce the calculation cost and memory occupation of long sequence processing by improving the transmission and storage mode of data processing process without changing the algorithm of traditional attention mechanism, that is, without sacrificing the ability to capture the integrity of long sequence information, an embodiment of a data processing method is proposed. Specifically, the data processing method is applied to a tensor processor, and the tensor processor includes a memory and a cache.

[0039] In this embodiment, the memory is a storage unit in the TPU for storing key data (such as the second key value data below) to be calculated attention weight and temporary calculation result (such as attention score, etc.). That is, the memory is the core data transfer source of attention weight calculation. Its core role is to accept the data transmitted from the cache, which is strongly related to the current attention weight calculation, and to provide stable data support for subsequent calculation. In this embodiment, the type of memory is not limited, for example, the memory can be static random access memory (SRAM), dynamic random access memory (DRAM) or virtual memory (VMEM), etc.

[0040] In the embodiment, the cache is a high-speed storage unit in the TPU for temporarily storing the first key-value data corresponding to all the first tokens, and is the initial storage and scheduling center of the key-value data. Its core role is to cache the full amount of first key-value data in advance, and then according to the calculation requirements (for example, as can be known from the following, the attention weight calculation of the second token needs the participation of the second key-value data), the required second key-value data is efficiently transmitted to the memory, avoiding the high delay problem of directly reading data from the external storage. In the embodiment, the type of the cache is not limited, for example, the cache can be a level one cache, a level two cache in a tensor processor, or a solid state disk or a high-bandwidth memory (HBM) in a computer, etc.

[0041] In the embodiment, as shown in Figure 1 The data processing method includes steps S100 to S600.

[0042] Step S100: Based on the token sequence, a plurality of first tokens are obtained.

[0043] In the embodiment, the token sequence is obtained in advance, and the first token is any token in the token sequence. In the embodiment, the token sequence is not limited, for example, the token sequence can be an input sequence of an AI model (for example, Transformer or RNN, etc.) input by a user, or an output sequence generated by the AI model based on the user's input. The token sequence can be a short sequence with only a few or tens of tokens, or a long sequence with several thousand or tens of thousands of tokens.

[0044] Step S200: Based on each first token, first key-value data corresponding to each first token is obtained.

[0045] In the embodiment, the first key-value data is data formed by the key vector and the value vector corresponding to the first token.

[0046] In the embodiment, the method of obtaining the first key-value data corresponding to the first token is not limited. For example, in the embodiment, the first key-value data (i.e. key and value) corresponding to the first token can be obtained through a Transformer model, which is pre-trained. It should be noted that training the Transformer model and obtaining the key vector and value vector of a certain token (for example, the first token) based on the trained Transformer model are mature technologies, which will not be described here.

[0047] It should be noted that in the application scenario of long sequence, there may be a large number of repeated meaning word in the long sequence (i.e. word sequence). For example, in the application scenario of generating multiple articles based on a certain topic (for example, with the topic of “My Ideal”), a large number of word repeated with the topic (for example, “I” or “ideal” and the like) will appear in the output sequence. In order to avoid repeated calculation and storage of the same meaning word in the output sequence, and reduce the occupation of computing resources and storage resources, in an embodiment of the present application, step S200, based on each first word, obtains first key value data corresponding to each first word, including step S210 and step S220.

[0048] Step S210: based on each first word, obtaining third word and fourth word.

[0049] In the present embodiment, the third word is any word in the first word that has not obtained the corresponding first key value data. The fourth word is any word in the first word that has obtained the corresponding first key value data.

[0050] Step S220: based on the third word, traversing each fourth word, if each fourth word has the same word as the third word, that is, the fifth word, then the first key value data corresponding to the fifth word is taken as the first key value data corresponding to the third word, otherwise, the first key value data of the third word is obtained based on the Transformer model. That is, before obtaining the first key value data of the third word based on the Transformer model, each fourth word needs to be traversed to determine whether there is a word (i.e. fifth word) with the same word as the third word in the word that has obtained the corresponding first key value data. If so, there is no need to calculate the first key value data corresponding to the third word again, thereby saving computing resources. In step S330, only the physical address of the second storage area corresponding to the third word and the fifth word in the physical mapping table needs to be the same, thereby saving storage resources.

[0051] Step S300: storing each first key value data in the cache.

[0052] In the embodiments of the present application, any reasonable way can be used to store each first key value data in the cache. For example, if the word sequence corresponds to 1000 first key value data, the 1000 first key value data can be stored as a whole in the cache; if the word sequence corresponds to 50 first key value data, the 50 first key value data can be stored as a whole in the cache.

[0053] It should be noted that in some application scenarios (for example, application scenarios of large models such as GPT or BERT) requiring a large amount of data of a token sequence, because the token sequence itself is numerous, a large amount of idle fragments in the cache may be caused. For example, a token sequence (hereinafter referred to as a first token sequence) has 100 first key value data, and after use, 100 storage spaces of the first key value data can be released from the cache; if the number of first key value data of the next token sequence (hereinafter referred to as a second token sequence) is less than or equal to 100, the plurality of first key value data corresponding to the second token sequence can be stored in the storage space released by the first token sequence; if the number of first key value data corresponding to the second token sequence is greater than 100, the plurality of first key value data corresponding to the second token sequence cannot be stored in the storage space released by the first token sequence. That is, if the number of first key value data corresponding to the latter token sequence (that is, the second token sequence) is greater than the number of first key value data corresponding to the former token sequence (that is, the first token sequence), it is likely that each first key value data of the latter cannot be stored in the storage space released by each first key value data of the former, that is, a large amount of idle storage fragments in the cache may be caused.

[0054] In order to avoid a large amount of idle storage fragments in the cache, in an embodiment of the present application, the cache is divided into a plurality of first storage areas in advance, and the capacity of each first storage area is the same. In this embodiment, step S300 stores each first key value data in the cache, including steps S310 to S330.

[0055] Step S310: Based on each first key value data, a plurality of key value data groups are obtained.

[0056] In this embodiment, each key value data group includes at least one first key value data, and the storage capacity required by each key value data group is less than or equal to the capacity of the first storage area. For example, the maximum number of first key value data included in each key value data group is 128, and the maximum number of first key value data that each first storage area can store is also 128; or the maximum number of first key value data included in each key value data group is 256, and the maximum number of first key value data that each first storage area can store is also 256. That is, in this embodiment, each key value data group can be stored in any idle first storage area.

[0057] In the embodiment, each first key-value data in the key-value data group can come from a different token sequence. For example, in a specific embodiment, the maximum number of first key-value data in each key-value data group is 128, and the maximum number of first key-value data that each first storage area can store is also 128. Assuming that the first key-value data corresponding to the third token sequence is 257, and the first key-value data corresponding to the fourth token sequence is 255. In the case where the first key-value data corresponding to the third token sequence and the fourth token sequence are not combined, the first key-value data corresponding to the third token sequence requires 3 first storage areas for storage; the first key-value data corresponding to the fourth token sequence requires 2 first storage areas for storage. If the first key-value data corresponding to the third token sequence and the fourth token sequence are combined to obtain a key-value data group for storage, only 4 first storage areas are required.

[0058] Step S320: Based on each first storage area, a second storage area corresponding to each first key-value data is obtained.

[0059] In the embodiment, the second storage area is any storage area in each first storage area that does not store data.

[0060] Step S330: Each key-value data is stored in the corresponding second storage area, and a corresponding physical mapping table is established.

[0061] In the embodiment, the physical mapping table at least includes the physical address of the corresponding second storage area in the cache where the first key-value data corresponding to any first token is located. That is, in the embodiment, the physical addresses of the storage areas corresponding to each first key-value data in the same key-value data group are the same.

[0062] In the embodiment, each first key-value data is integrated into a key-value data group, and the cache space is efficiently utilized by storing the key-value data group as a whole, thereby reducing the fragmentation and waste of the cache. For example, if the maximum number of first key-value data that a first storage area can store is 100, the first key-value data corresponding to the first token sequence is 200 (i.e., occupying two first storage areas for storage), and the first key-value data corresponding to the second token sequence is 300 (i.e., occupying three first storage areas for storage), after the storage space occupied by the first token sequence is released, the 200 first key-value data (i.e., any two of the three key-value data groups) corresponding to the second token sequence can be stored in the storage space released by the first token sequence.

[0063] It should be noted that in the present embodiment, if the storage capacity of the buffer is greater than the storage capacity of the memory, in order to avoid the capacity of the first storage area being greater than the capacity of the memory, and further causing data overflow after the key-value data set is stored in the memory by the buffer, in an embodiment of the present application, the buffer is divided into a plurality of first storage areas in advance, including steps S340 to S370.

[0064] Step S340: based on the first key-value data, obtaining a first data capacity.

[0065] In the present embodiment, the first data capacity is equal to the data amount of the first key-value data (i.e. the storage space required when storing the first key-value data).

[0066] Step S350: based on the memory, obtaining a second data capacity.

[0067] In the present embodiment, the second data capacity is the capacity of the memory.

[0068] Step S360: based on the second data capacity and the first data capacity, obtaining a maximum number.

[0069] In the present embodiment, the maximum number is equal to the ratio of the second data capacity and the first data capacity after taking the integer part. That is, if the maximum number is x, x is any positive integer greater than or equal to 1, which means that the memory can store a maximum of x first key-value data.

[0070] Step S370: based on the maximum number, dividing the buffer to obtain a plurality of first storage areas.

[0071] In the embodiment, the capacity of the first storage area is greater than or equal to the capacity required for storing the maximum number of first key-value data. It should be noted that if the capacity of the first storage area is large, the utilization efficiency of the cache space may be reduced. Since each key-value data group is stored in a single first storage area, if the first storage area meets the storage requirement (i.e., the capacity of the first storage area is greater than or equal to the capacity required for storing the maximum number of first key-value data), if the capacity of a certain first storage area is greater than the actual required capacity of the key-value data group stored therein, the first storage area will have idle space that is not utilized. That is, if the first storage area meets the storage requirement, the less the number of key-value data stored in the first storage area, the larger the idle space; the more the number of key-value data stored in the first storage area, the smaller the idle space. For example, if the capacity of the first storage area is equal to the actual required capacity of one key-value data, idle space will not be generated. It should be noted that since the total storage resource space of the cache is limited, the above idle space will reduce the number of key-value data groups that can be accommodated by the cache, and it may not be possible to completely store all first key-value data corresponding to the first token, thereby requiring additional external storage interaction and increasing data scheduling delay. If the capacity of the first storage area is too small, the data management cost will increase. Since the capacity of a single first storage area is small, it is necessary to split each first key-value data from the same token sequence into multiple key-value data groups and store them in multiple first storage areas, which not only increases the computational overhead of data splitting, but also the more the number of split key-value data groups, the more times the physical addresses of different storage areas need to be searched when locating multiple first key-value data of the same token sequence based on the physical mapping table, which prolongs the data locating time and reduces the transmission efficiency from the cache to the memory.

[0072] In the embodiment, the capacity of the first storage area can be reasonably set. For example, the inventors have found through a large number of verifications that if the capacity of the first storage area is equal to the actual required storage space of 128 or 256 first key-value data, the cache is neither likely to generate a large amount of idle space nor likely to greatly increase the data management cost.

[0073] Step S400: Obtain a second token based on each first token.

[0074] In the embodiment, the second token is an arbitrary token for which the attention weight between the second token and each first token is not calculated. In the embodiment, the source of the second token is not limited. For example, the source of the second token can have at least the following four application scenarios.

[0075] First, in the application scenarios such as text classification, sentiment analysis, named entity recognition, which depend on the internal semantic association of the input sequence, the second word element can be any word element from the input sequence, that is, the second word element needs to calculate the attention weight with all the first word elements in the input sequence. In the encoder module of the Transformer model, the input sequence (for example, the source text of machine translation, the text to be analyzed for text classification) itself needs to capture the internal context association through the attention mechanism. At this time, each first word element is all the word elements in the input sequence, and the second word element is any word element in the input sequence for which the attention weight has not been calculated. For example, the input sequence is "Artificial intelligence helps data processing", and each first word element is the 5 word elements in the input sequence (that is, "artificial", "intelligence", "helps", "data" and "processing"). When "data" is the second word element, the second key-value data (that is, the key vector and value vector corresponding to the 5 word elements) of all the first word elements in the transmission buffer needs to be transmitted to the memory, and the attention score of "data" and the remaining 4 word elements is calculated, and finally the context semantic representation of "data" is formed.

[0076] Second, in the application scenarios such as machine translation, speech to text, image-text generation, the second word element can be any word element from the output sequence, that is, the second word element needs to calculate the attention weight with all the first word elements in the input sequence. In the decoder module of the Transformer model, the output sequence (for example, the target text of machine translation or the result text of text generation, etc.) needs to associate the semantic information of the input sequence through the cross-attention mechanism. At this time, each first word element is all the word elements in the input sequence, and the second word element is any word element in the output sequence for which the cross-attention weight has not been calculated. For example, the input sequence is English "AI improves efficiency" (that is, there are 3 first word elements), and the output sequence is Chinese "Artificial intelligence improves efficiency" (the word elements of the output sequence are generated gradually in the generation process). When "improves" in the output sequence is the second word element, the second key-value data of the 3 first word elements in the input sequence needs to be transmitted from the buffer to the memory, and the attention weight of "improves" with "AI", "improves" and "efficiency" is calculated, to ensure the consistency of the output second word element with the input semantics.

[0077] Third, in the application scenarios of text generation, reply generation of dialogue robots, code generation, and other self-regressive generation of GPT-based large models, the second token can be any token from the output sequence. That is, the second token needs to calculate the attention weight with each token (that is, each first token) of the generated output sequence. In the decoder attention mechanism module of the Transformer model, the output sequence generation needs to follow the "self-regression" logic, that is, the subsequently generated token needs to refer to the semantics of the generated token. At this time, each first token is a generated token in the output sequence, and the second token can be a token in the output sequence to be generated and for which the attention weight has not been calculated. For example, in the text generation task, the generated output sequence is "data processing is the core" (a total of 5 first tokens), and the to-be-generated token "efficient" is the second token. At this time, the second key-value data of the generated 5 first tokens needs to be transmitted from the cache to the memory, and the attention weight of "efficient" with "data", "processing", "of", "core" and "is" is calculated to ensure the grammatical coherence and semantic fluency of the generated content.

[0078] Fourth, in the application scenarios of cross-domain question answering systems, domain-adaptive text understanding, and cross-language semantic alignment, the second token can be any token from the cross-domain input sequence, that is, the second token needs to calculate the attention weight with each first token of the source domain. In cross-domain data processing tasks, the semantic knowledge of the source domain needs to be transferred to the target domain. At this time, each first token is a plurality of tokens in the token sequence of the source domain (for example, tokens related to "tumor diagnosis standard" in the medical field), and the second token is any token in the target domain input sequence for which the attention weight has not been calculated (for example, tokens related to "abnormal body lumps" in daily text), and the domain knowledge reuse is realized through the attention mechanism association. For example, in a medical question answering system, each first token of the source domain is "lung cancer, cough, chest pain, and hemoptysis" (4 first tokens in total), and the target domain input sequence is "recent frequent cough". If the to-be-calculated second token is "cough", the second key-value data of the 4 first tokens in the source domain needs to be transmitted from the cache to the memory, and the attention weight of "cough" with "lung cancer", "cough", "chest pain", and "hemoptysis" is calculated to assist in judging the correlation between the user's symptoms and medical knowledge.

[0079] So far, the application scenarios of the second token source have been introduced.

[0080] It should be noted that, due to the numerous application scenarios of the attention mechanism, it is impossible to exhaust all its application scenarios. In this embodiment, only several typical application scenarios of the first token and the second token source are listed, which do not represent that the data processing method proposed in the present application is only applicable to the above-mentioned listed application scenarios, and it should be understood that the data processing method proposed in the present application is applicable to any application scenario of the attention mechanism.

[0081] Step S500: Based on the second word element, transfer multiple second key-value data in the buffer to the memory.

[0082] In the embodiment, the second key-value data is any key-value data used to calculate the attention weight of the second word element in the buffer.

[0083] It should be noted that, as known from the background art, in the conventional attention mechanism or the existing improved scheme, the key-value data (i.e., the first key-value data) is not pre-stored, and each time the attention weight of a different second word element is calculated, the key-value data corresponding to the first word element needs to be calculated based on the Transformer model (i.e., the conversion process of "converting a word element into a key vector and a value vector" is repeatedly performed), thereby causing the calculation amount of the attention weight of a long sequence to exponentially increase. In the embodiment, after the first key-value data corresponding to all first word elements is calculated, it is stored in the buffer, and when the attention weight of any second word element is calculated subsequently, only the required second key-value data needs to be retrieved from the buffer (i.e., the first key-value data does not need to be recalculated), thereby completely avoiding the overhead of repeated generation of key-value data and significantly reducing the calculation cost of long sequence processing.

[0084] In the embodiment, when calculating the attention weight of the second word element, the full amount of first key-value data related to the second word element can be transferred from the buffer to the memory. For example, in step S300, if each first key-value data is stored in the buffer as a whole, then subsequently the full amount of first key-value data needs to be transferred from the buffer to the memory.

[0085] It should be noted that, in the conventional attention mechanism, the full amount of first key-value data needs to be stored in the memory when processing a long sequence, thereby causing the memory occupancy to increase with the number of word elements. Although the sparse attention mechanism and the block attention mechanism can reduce the occupancy of the memory, they discard part of the key-value data (for example, the sparse attention mechanism only uses the key-value data of the key word element to participate in the calculation of the attention weight, and the block attention mechanism only uses the key-value data of the word elements in the same block to participate in the calculation of the attention weight), thereby sacrificing the integrity of the information of the long sequence. In the process of using the conventional attention mechanism to process a long sequence, in order to avoid loading the full amount of first key-value data in step S500, i.e., to reduce the occupancy of the memory in the application scenario of long sequence attention weight calculation, in the embodiment of the present application, step S500, based on the second word element, transfers multiple second key-value data in the buffer to the memory, which includes step S510.

[0086] Step S510: Based on the physical mapping table, transfer the key-value data group corresponding to each second key-value data to the memory.

[0087] In the embodiment, the steps S310 to S330 store each first key-value data in a different storage area (i.e., a first storage area) in the cache, and only the required first key-value data (i.e., second key-value data) is transmitted to the memory when needed, i.e., without loading the full amount of first key-value data. That is, in the embodiment, the memory only receives the "second key-value data required for calculating the current second token attention weight" (i.e., not the full amount of first key-value data), and the second key-value data covers all information related to the second token, which not only avoids the high occupancy problem caused by full storage in the memory, but also ensures the integrity of long sequence information capture. That is, the embodiment pre-stores key-value data (i.e., first key-value data) in the cache, and then transmits the key-value data (i.e., second key-value data) to the memory when needed, which simultaneously solves the two core pain points of repeated calculation and high memory occupancy in the prior art without sacrificing the ability to capture the integrity of long sequence information.

[0088] It should be noted that in the long sequence application scenario, since there are many second key-value data to be called, in order to avoid data overflow caused by the memory being unable to accommodate the numerous second key-value data, in an embodiment of the present application, step S510, based on the physical mapping table, transmits each second key-value data corresponding key-value data group to the memory, including steps S511 to S513.

[0089] Step S511: based on each second key-value data, obtaining a first transmission instruction, the first transmission instruction is used to indicate transmission of at least one third key-value data.

[0090] In the embodiment, the third key-value data is any second key-value data currently required to be transmitted from the cache to the memory. That is, in the embodiment, each second key-value data can be transmitted from the cache to the memory in batches, i.e., ensuring that there is always enough space in the memory to store the current batch of required data (i.e., each third key-value data), thereby avoiding the risk of data overflow. For example, in a specific embodiment, the number of each second key-value data corresponding to the second token is 1000, and the maximum capacity of the memory can only accommodate 128 second key-value data, then each time 128 second key-value data is transmitted to the memory, after the 128 second key-value data are used for calculation, the 128 second key-value data are released from the memory, and the next batch of 128 second key-value data is transmitted, until all 1000 second key-value data are completely transmitted and used for attention weight calculation.

[0091] In the embodiment, the first transmission instruction at least includes physical address information of a storage region in which the third key-value data is located in the cache.

[0092] Step S512: based on the physical mapping table and the first transmission instruction, obtaining physical addresses of a plurality of third storage regions.

[0093] In the embodiment, the third storage region is a storage region in which the corresponding third key-value data is stored in the cache.

[0094] Step S513: based on the physical address of each third storage region, transmitting the key-value data group in each third storage region to the memory.

[0095] It should be noted that transmitting the storage data in the fixed physical address in a storage device (i.e., the cache) to another storage device (i.e., the memory) is a mature technology, which will not be described here.

[0096] It should be noted that in a long sequence application scenario, the more the number of second key-value data, the more batches of data transmission are required. For example, if the number of each second key-value data corresponding to the second word element is 1000, and the maximum capacity of the memory is only capable of accommodating 128 second key-value data, at least 7 times are required to complete the transmission of all data; if the number of each second key-value data corresponding to the second word element is 2000, and the maximum capacity of the memory is only capable of accommodating 128 second key-value data, at least 16 times are required to complete the transmission of all data, and so on.

[0097] As known from the foregoing, in the application scenario of batch transmission of a plurality of second key-value data, only after the key-value data of the previous batch participates in the calculation of the attention weight and the key-value data of the previous batch is released from the memory, the transmission of the next batch of key-value data can be performed (i.e., data calculation and data transmission are serial). In order to reduce the time consumption of data processing, in an embodiment of the present application, the memory can include a first buffer and a second buffer. Step S513, based on the physical address of each third storage region, transmitting the key-value data group in each third storage region to the memory, includes: transmitting the key-value data group in each third storage region to the first buffer.

[0098] Specifically, after transmitting the key-value data group in each third storage region to the first buffer, the method further includes steps S514 to S516.

[0099] Step S514: based on each second key-value data, obtaining a second transmission instruction, the second transmission instruction being used to indicate the transmission of at least one fourth key-value data.

[0100] In the embodiment, the fourth key-value data is any second key-value data in the cache that needs to be transmitted to the memory after the third key-value data.

[0101] Step S515: Obtain the physical addresses of the fourth storage areas based on the physical mapping table and the second transmission instruction.

[0102] In the embodiment, the fourth storage area is a storage area in the cache that stores the corresponding fourth key-value data.

[0103] Step S516: Transmit the key-value data group in each fourth storage area to the second buffer based on the physical address of each fourth storage area.

[0104] In the embodiment, the third key-value data is transmitted to the first buffer of the memory, and the fourth key-value data is transmitted to the second buffer of the memory, so that the next batch of key-value data can be transmitted without waiting for the calculation to be completed. For example, after the third key-value data is transmitted to the first buffer, the third key-value data can be used to participate in the attention weight calculation of the second token. During the calculation, the fourth key-value data can be transmitted to the second buffer. After the calculation of the third key-value data is completed, the third key-value data is released from the first buffer, and the fourth key-value data is used to participate in the attention weight calculation of the second token. During the calculation, the next batch of key-value data that needs to participate in the calculation is transmitted to the first buffer. In this way, the transmission and calculation of the key-value data are parallelized, which further reduces the time consumption of long sequence processing while avoiding data overflow in the memory.

[0105] As shown in step S220, the first key-value data corresponding to the same meaning token (i.e., the third token and the fifth token in the above) can be the same (i.e., the physical addresses of the corresponding second storage areas can be the same). In order to avoid repeated transmission of key-value data corresponding to the same token (i.e., multiple transmissions from the cache to the memory), in an embodiment of the present application, the tensor processor further includes a counter corresponding to each key-value data group. The initial value of the counter is equal to the sum of the number of first tokens corresponding to each second key-value data in the key-value data group. For example, a key-value data group consists of 128 first key-value data, of which 56 first key-value data is needed for the attention weight calculation of the second token (i.e., second key-value data). If each second key-value data corresponds to only one first token, the initial value of the counter corresponding to the key-value data group is 56 1=56; if 28 second key-value data pairs correspond to two first word items, 28 second key-value data pairs correspond to one first word item, the initial value of the counter corresponding to the key-value data group is 28 2+28 1=84, and so on. In this embodiment, after the step S510 of transmitting the key-value data group corresponding to each second key-value data to the memory, the method further includes steps S520 and S530.

[0106] Step S520: for any key-value data group, each time any second key-value data in the key-value data group is called, the value of the counter corresponding to the key-value data group is reduced by 1.

[0107] It should be noted that the second word item attention weight is calculated as follows:

[0108] First, the Transformer model is used to generate a query vector based on the second word item, which is used to represent the semantic requirements of the second word item; at the same time, the key vector and value vector corresponding to the first word item are extracted from the second key-value data transmitted from the cache to the memory, wherein the key vector is used to represent the semantic features of the first word item, and the value vector is used to carry the semantic information of the first word item.

[0109] Secondly, the similarity between the query vector of the second word item and the key vector of the first word item is calculated by a predetermined similarity calculation method (for example, dot product, scaled dot product or cosine similarity), and an attention score is obtained.

[0110] Finally, the attention score is normalized by the Softmax function, so that the sum of the attention scores of all first word items is 1, and the reasonable allocation of the semantic correlation of different first word items is realized; then the normalized attention score and the value vector corresponding to the first word item are weighted and summed, and finally the attention weight of the second word item is obtained, which integrates the context semantic information of all related first word items, and provides accurate semantic representation support for subsequent tasks (for example, text generation or semantic understanding, etc.).

[0111] In this embodiment, each time the key-value data of a first word item in the key-value data group is read, it means that the key vector and value vector corresponding to the first word item must participate in the calculation, that is, the value of the counter needs to be reduced by 1.

[0112] Step S530: for any key-value data group, if the value of the counter corresponding to the key-value data group is 0, the key-value data group is released from the memory.

[0113] In the embodiment, if the value of the counter is 0, it indicates that all key-value data of the first token in the key-value data group participates in the attention weight calculation of the second token, that is, the key-value data group can be released.

[0114] The embodiment can accurately track the use state of the key-value data group by setting the counter, ensure that all second key-value data that needs to participate in the calculation in the key-value data group is not missed or repeatedly called, avoid incomplete attention weight calculation due to data missing, or increase the calculation overhead due to repeated calling. After the key-value data group fully participates in the calculation, the memory space is released in time to avoid memory overflow caused by accumulated used data, and space is freed for the key-value data group to be transmitted in the subsequent batch, ensuring that the memory is continuously and efficiently available. Ultimately, without sacrificing the integrity of long sequence information, the memory occupation is significantly reduced, and the ability of the tensor processor to process long sequence data is improved. In an application scenario where one key-value data corresponds to multiple first tokens, repeated transmission of the corresponding key-value data group can be avoided by only setting the initial value of the counter, thereby saving the overhead of data transmission.

[0115] Step S600: The second key-value data in the memory is used to participate in attention calculation to obtain the attention weight of the second token.

[0116] As known from the foregoing, the calculation of the attention weight of the second token by the key-value data corresponding to the second token (i.e., the key vector and the value vector) is a mature technology, and will not be described here.

[0117] It should be noted that the calculation of the attention weight of the second token requires loading all key-value data related to the second token at one time. As known from steps S511 and S516, in an application scenario where the second key-value data is large, it is impossible to transmit all second key-value data required for the second token attention weight calculation to the memory at one time, that is, each second key-value data can only be transmitted to the memory in batches. It should be noted that if each second key-value data cannot be transmitted to the memory at one time, it is impossible to load all key-value data related to the second token at one time, that is, it is impossible to calculate the attention weight of the second token. In order to calculate the attention weight of the second token in the case where all key-value data cannot be loaded at one time, in an embodiment of the present application, step S600 obtains the attention weight of the second token based on the second key-value data in the memory, including steps S610 to S640.

[0118] Step S610: Obtain the historical attention weight of the second token.

[0119] In the embodiment, the historical attention weight is calculated by the first quantity of second key-value data. The first quantity refers to the quantity of second key-value data participating in the calculation of the historical attention weight of the second token.

[0120] Step S620: Obtain all fifth key-value data in the memory.

[0121] In the embodiment, the fifth key-value data is any second key-value data in the memory that does not participate in the calculation of the historical attention weight.

[0122] Step S630: Obtain new attention weight based on the second token and each fifth key-value data.

[0123] In the embodiment, the new attention weight refers to new weight data supplemented to the historical attention weight, which is calculated based on the attention score between the second token and each fifth key-value data in the current batch in the memory that does not participate in the calculation in the batch-by-batch calculation process of the attention weight of the second token. That is, the calculation method of the new attention weight of the second token is completely the same as the calculation of the attention weight of the second token, except that the key-value data participating in the calculation is different. For example, the attention weight of the second token needs full key-value data to participate in the calculation, and the new attention weight of the second token only needs any second key-value data in the memory that does not participate in the calculation of the historical attention weight to participate in the calculation.

[0124] Step S640: Update the historical attention weight and the first quantity based on the new attention weight to obtain a current attention weight, until the first quantity is equal to a second quantity; if the first quantity is equal to the second quantity, the current attention weight is taken as the attention weight of the second token.

[0125] In the embodiment, the second quantity is the total quantity of second key-value data needed to be used in the calculation of the attention weight of the second token. That is, if the first quantity is equal to the second quantity, it means that all key-value data participate in the calculation of the attention weight of the second token.

[0126] In the embodiment, the calculation formula of updating the first quantity based on the new attention weight is as follows:

[0127]

[0128] wherein, represents the first quantity after the update; represents the first quantity before the update; represents the quantity of fifth key-value data participating in the calculation of the new attention weight.

[0129] In the embodiment, the history attention weight can be updated based on the new attention weight in any reasonable manner to obtain the current attention weight, as long as the updating manner is consistent with the difference between the new attention weight and the history attention weight. For example, if the new attention weight is greater than the history attention weight, the history attention weight is appropriately increased to obtain the current attention weight; if the new attention weight is less than the history attention weight, the history attention weight is appropriately decreased to obtain the current attention weight; and if the new attention weight is equal to the history attention weight, the history attention weight can be directly taken as the current attention weight.

[0130] In an embodiment of the present application, the calculation formula for updating the history attention weight based on the new attention weight to obtain the current attention weight can be as follows:

[0131]

[0132] wherein, the current attention weight is represented by a; the history attention weight is represented by a; the new attention weight is represented by a; and n represents the number of iteration updates of the current attention weight.

[0133] In another embodiment of the present application, the calculation formula for updating the history attention weight based on the new attention weight to obtain the current attention weight can be as follows:

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] wherein, the current attention weight is represented by a; the history attention weight is represented by a; the new attention weight is represented by a; the first number represents the number of attention scores calculated to obtain the history attention weight; the i-th attention score calculated to obtain the history attention weight is represented by ai; the maximum value of the attention scores calculated to obtain the history attention weight is represented by amax. denotes an exponential function with a natural constant e as the base; denotes the number of attention scores calculated to obtain the new attention weight; denotes the kth attention score in the attention scores calculated to obtain the new attention weight; denotes the maximum value in the attention scores calculated to obtain the new attention weight; denotes the maximum value in the multiple values in the parentheses.

[0141] The embodiment of the data processing method proposed in the present application stores all first key-value data corresponding to the first token in the cache. When calculating the attention weight of any second token, only the required second key-value data needs to be retrieved from the cache, that is, there is no need to recalculate the corresponding first key-value data, which completely avoids the overhead of repeated generation of key-value data and significantly reduces the calculation cost of long sequence processing. Moreover, by pre-storing the key-value data (i.e., first key-value data) in the cache and then transmitting the key-value data (i.e., second key-value data) to the memory when needed, the two core pain points of repeated calculation and high memory occupation in the prior art are solved without sacrificing the integrity of the long sequence information capturing capability.

[0142] After introducing the data processing method proposed in the embodiment of the present application, the embodiment of a data processing device proposed in the present application is introduced below. Specifically, the data processing device is applied to a tensor processor, and the tensor processor includes a memory and a cache. As shown in Figure 2 the data processing device 10 includes:

[0143] The reading module 11 is configured to obtain a plurality of first tokens based on a token sequence; the first token is any token in the token sequence;

[0144] The processing module 12 is configured to obtain first key-value data corresponding to each first token based on each first token; the first key-value data is data formed by a key vector and a value vector corresponding to the first token;

[0145] and store each first key-value data in the cache;

[0146] and obtain a second token based on each first token; the second token is any token for which the attention weight with each first token has not been calculated;

[0147] and transmit, based on the second word piece, a plurality of second key-value data in the buffer to the memory; the second key-value data is any key-value data used to calculate the attention weight of the second word piece; and the second key-value data in the memory is used to participate in attention calculation to obtain the attention weight of the second word piece.

[0148] As a specific embodiment in the present application, the buffer is divided into a plurality of first storage areas, and each first storage area has the same capacity; the processing module 12 is further configured to obtain a plurality of key-value data groups based on each first key-value data; each key-value data group includes at least one first key-value data; and the required storage capacity of each key-value data group is less than or equal to the capacity of the first storage area.

[0149] and obtain, based on each first storage area, a second storage area corresponding to each first key-value data; the second storage area is any storage area in each first storage area that does not store data;

[0150] and store each key-value data in the corresponding second storage area, and establish a corresponding physical mapping table; the physical mapping table includes at least the physical address of the corresponding second storage area in the buffer where the first key-value data corresponding to any first word piece is located.

[0151] and transmit, based on the physical mapping table, the key-value data group corresponding to each second key-value data to the memory.

[0152] As a specific embodiment in the present application, the processing module 12 is further configured to obtain a third word piece and a fourth word piece based on each first word piece; the third word piece is any word piece in each first word piece for which the corresponding first key-value data has not been obtained; and the fourth word piece is any word piece in each first word piece for which the corresponding first key-value data has been obtained.

[0153] and based on the third word piece, traverse each fourth word piece; if each fourth word piece has the same word piece as the third word piece, i.e., a fifth word piece, then the first key-value data corresponding to the fifth word piece is taken as the first key-value data corresponding to the third word piece, and the physical address of the second storage area corresponding to the third word piece and the fifth word piece is the same; otherwise, the first key-value data of the third word piece is obtained based on a Transformer model.

[0154] As a specific embodiment in the present application, the tensor processor further comprises a counter corresponding to each key-value data group; an initial value of the counter is equal to a sum of the number of first tokens corresponding to each second key-value data in the key-value data group; the processing module 12 is further configured to, for any key-value data group, decrease the value of the counter corresponding to the key-value data group by 1 each time any second key-value data in the key-value data group is called;

[0155] and, for any key-value data group, if the value of the counter corresponding to the key-value data group is 0, release the key-value data group from the memory.

[0156] As a specific embodiment in the present application, the processing module 12 is further configured to, based on the first key-value data, obtain a first data capacity; the first data capacity is equal to the data amount of the first key-value data;

[0157] and, based on the memory, obtain a second data capacity; the second data capacity is the capacity of the memory;

[0158] and, based on the second data capacity and the first data capacity, obtain a maximum number; the maximum number is equal to the ratio of the second data capacity and the first data capacity after rounding down;

[0159] and, based on the maximum number, divide the cache into a plurality of first storage areas.

[0160] As a specific embodiment in the present application, the processing module 12 is further configured to, based on each second key-value data, obtain a first transmission instruction; the first transmission instruction is used to instruct to transmit at least one third key-value data; the third key-value data is any second key-value data in the cache that currently needs to be transmitted to the memory;

[0161] and, based on the physical mapping table and the first transmission instruction, obtain physical addresses of a plurality of third storage areas; the third storage area is a storage area in the cache that stores corresponding third key-value data;

[0162] and, based on the physical addresses of each third storage area, transmit the key-value data group in each third storage area to the memory.

[0163] As a specific embodiment in the present application, the memory comprises a first buffer and a second buffer; the processing module 12 is further configured to transmit the key-value data group in each third storage area to the first buffer;

[0164] and, based on each second key-value data, obtain a second transmission instruction, the second transmission instruction being used for instructing to transmit at least one fourth key-value data; the fourth key-value data being any second key-value data in the cache that is located after the third key-value data and needs to be transmitted to the memory;

[0165] and, based on the physical mapping table and the second transmission instruction, obtain physical addresses of a plurality of fourth storage regions; the fourth storage region being a storage region in the cache that stores corresponding fourth key-value data;

[0166] and, based on the physical address of each fourth storage region, transmit the key-value data group in each fourth storage region to the second buffer.

[0167] As one specific embodiment in the present application, the processing module 12 is further configured to obtain a historical attention weight of the second token; the historical attention weight is obtained by calculation of a first number of second key-value data;

[0168] and, obtain all fifth key-value data in the memory; the fifth key-value data being any second key-value data in the memory that does not participate in the calculation of the historical attention weight;

[0169] and, based on the second token and each fifth key-value data, obtain an added attention weight;

[0170] and, based on the added attention weight, update the historical attention weight and the first number to obtain a current attention weight, until the first number is equal to a second number; the second number being a total number of second key-value data that needs to be used in the calculation of the attention weight of the second token; if the first number is equal to the second number, the current attention weight is taken as the attention weight of the second token.

[0171] The embodiment of the data processing device proposed in the present application stores all first key-value data corresponding to the first token in the cache, and when calculating the attention weight of any second token, only the required second key-value data needs to be retrieved from the cache, that is, there is no need to recalculate the corresponding first key-value data, which completely avoids the overhead of repeated generation of key-value data and significantly reduces the calculation cost of long sequence processing. Moreover, by pre-storing the key-value data (i.e. first key-value data) in the cache and then transmitting the key-value data (i.e. second key-value data) to the memory when needed, the two core pain points of repeated calculation and high memory occupation in the prior art are solved without sacrificing the integrity of the long sequence information capturing capability.

[0172] After the data processing apparatus proposed by the embodiments of the present application is introduced, an embodiment of a computer readable storage medium proposed by the present application is introduced below. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the data processing method described in any one of the above embodiments.

[0173] The embodiment of the computer readable storage medium proposed by the present application stores all first key-value data corresponding to all first word pieces in the cache, and when calculating the attention weight of any second word piece subsequently, only the required second key-value data needs to be retrieved from the cache, that is, the corresponding first key-value data does not need to be recalculated, thereby completely avoiding the overhead of repeated generation of key-value data and significantly reducing the calculation cost of long sequence processing. Moreover, by pre-storing the key-value data (i.e., the first key-value data) in the cache and then transmitting the key-value data (i.e., the second key-value data) to the memory when needed, the two core pain points of repeated calculation and high memory occupation in the prior art are solved at the same time without sacrificing the integrity capturing capability of long sequence information.

[0174] After the embodiment of the computer readable storage medium proposed by the embodiments of the present application is introduced, an embodiment of a computing device proposed by the present application is introduced below. The computing device includes a processor and a memory; the processor is connected with the memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program, so that the computing device executes the data processing method described in any one of the above embodiments.

[0175] The embodiment of the computing device proposed by the present application stores all first key-value data corresponding to all first word pieces in the cache, and when calculating the attention weight of any second word piece subsequently, only the required second key-value data needs to be retrieved from the cache, that is, the corresponding first key-value data does not need to be recalculated, thereby completely avoiding the overhead of repeated generation of key-value data and significantly reducing the calculation cost of long sequence processing. Moreover, by pre-storing the key-value data (i.e., the first key-value data) in the cache and then transmitting the key-value data (i.e., the second key-value data) to the memory when needed, the two core pain points of repeated calculation and high memory occupation in the prior art are solved at the same time without sacrificing the integrity capturing capability of long sequence information.

[0176] It should be noted that the computing device involved in the embodiments of the present application can be a server and / or a terminal device.

[0177] The server related to the embodiments of the present application can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and basic cloud computing services such as big data and artificial intelligence platform.

[0178] The terminal device related to the embodiments of the present application can be a device providing voice and / or data connectivity to users, a handheld device with wireless connection function, or other processing devices connected to a wireless modem. For example, mobile phones (also known as "cellular" phones) and computers with mobile terminals, for example, portable, pocket, handheld, built-in or vehicle-mounted mobile devices, which exchange voice and / or data with a wireless access network. For example, personal communication service (English: Personal Communication Service, English: PCS) phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (Wireless Local Loop, English: WLL) stations, personal digital assistants (English: Personal Digital Assistant, English: PDA) and other devices.

[0179] The embodiments of the present application also provide a terminal device, as shown in Figure 3 For the convenience of description, only the parts related to the embodiments of the present application are shown, and the specific technical details not disclosed are referred to the method part of the embodiments of the present application. The terminal device can be any terminal device including a server, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), a point of sales (Point of Sales, POS), a vehicle-mounted computer, etc. Taking the terminal device as a server for example:

[0180] Please refer to Figure 3 , Figure 3Fig. 11 is a schematic diagram of a server structure according to an embodiment of the present application. The server 1100 can have a great difference due to different configurations or performances, and can include one or more central processing units (CPU) 1122 (e.g., one or more processors) and a memory 1132, one or more storage media 1130 (e.g., one or more mass storage devices) storing application programs 1142 or data 1144. The memory 1132 and the storage media 1130 can be temporary or persistent storage. The programs stored in the storage media 1130 can include one or more modules (not shown in the figure), each of which can include a series of instruction operations in the server. Further, the central processing unit 1122 can be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the server 1100.

[0181] The server 1100 can also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0182] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0183] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and module can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0184] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed system, device and method can be implemented by other means. For example, the above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0185] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, i.e., may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0186] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can be stored in a computer readable storage medium.

[0187] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part can be realized in the form of a computer program product.

[0188] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0189] The technical solutions provided by the embodiments of the present application are described in detail above, and the principles and implementation manners of the embodiments of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and the core idea of the embodiments of the present application. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the ideas of the embodiments of the present application. In conclusion, the content of the specification should not be understood as a limitation of the embodiments of the present application.

Claims

1. A data processing method applied to a tensor processor, the tensor processor comprising a memory and a cache; characterized in that, include: Based on the word sequence, obtain multiple first words; The first lexical unit is any lexical unit in the lexical unit sequence; Based on each first word element, obtain the first key value data corresponding to each first word element; The first key-value data is the data formed by the key vector and value vector corresponding to the first word element; Each first key-value data is stored in the cache; Based on each first word element, obtain the second word element; The second lexical unit is any lexical unit for which attention weights between it and each of the first lexical units have not been calculated; Based on the second word, multiple second key-value data in the cache are transferred to the memory; the second key-value data is any key-value data required to calculate the attention weight of the second word in the cache; The second key-value data in the memory is used to participate in attention calculation and obtain the attention weight of the second word element; The cache is pre-divided into multiple first storage areas, each with the same capacity. The step of storing each first key-value data into the cache includes: Based on each first key-value data, multiple key-value data groups are obtained; each key-value data group includes at least one first key-value data; and the storage capacity required for each key-value data group is less than or equal to the capacity of the first storage area. Based on each first storage area, a second storage area corresponding one-to-one with each first key-value data is obtained; the second storage area is any storage area in each first storage area that does not store data. Each key-value data is stored in the corresponding second storage area, and a corresponding physical mapping table is established; the physical mapping table includes at least the physical address of the first key-value data corresponding to any first word in the corresponding second storage area in the cache; The step of transferring multiple second key-value pairs from the cache to the memory based on the second lexical unit includes: Based on the physical mapping table, the key-value data groups corresponding to each second key-value data are transferred to the memory.

2. The data processing method according to claim 1, characterized in that, The step of obtaining the first key-value data corresponding one-to-one with each first word element includes: Based on each first word element, a third word element and a fourth word element are obtained; the third word element is any word element among the first word elements for which the corresponding first key value data has not been obtained; the fourth word element is any word element among the first word elements for which the corresponding first key value data has been obtained. Based on the third lexical unit, each fourth lexical unit is traversed. If each fourth lexical unit has a lexical unit that is the same as the third lexical unit, that is, a fifth lexical unit, then the first key-value data corresponding to the fifth lexical unit is used as the first key-value data corresponding to the third lexical unit. The physical address of the second storage area corresponding to the third lexical unit and the fifth lexical unit is the same. Otherwise, the first key-value data of the third lexical unit is obtained based on the Transformer model.

3. The data processing method according to claim 2, characterized in that, The tensor processor further includes a counter corresponding one-to-one with each key-value data group; the initial value of the counter is equal to the sum of the number of first tokens corresponding to each second key-value data in the key-value data group; after transferring the key-value data groups corresponding to each second key-value data to the memory, the method further includes: For any key-value data group, each time any second key-value data in the key-value data group is called, the value of the counter corresponding to the key-value data group is decremented by 1; For any key-value data group, if the value of the counter corresponding to the key-value data group is 0, then the key-value data group is released from the memory.

4. The data processing method according to claim 1, characterized in that, The cache is pre-divided into multiple first storage areas, including: Based on the first key-value data, a first data capacity is obtained; the first data capacity is equal to the data volume of the first key-value data. Based on the memory device, a second data capacity is obtained; the second data capacity is the capacity of the memory device. Based on the second data capacity and the first data capacity, obtain the maximum number; the maximum number is equal to the ratio of the second data capacity and the first data capacity rounded down; Based on the maximum number, multiple first storage regions are obtained from the cache.

5. The data processing method according to any one of claims 1 to 4, characterized in that, The step of transferring the key-value data groups corresponding to each second key-value data to the memory based on the physical mapping table includes: Based on each second key value data, a first transmission instruction is obtained, which is used to instruct the transmission of at least one third key value data; the third key value data is any second key value data in the buffer that needs to be transmitted to the memory. Based on the physical mapping table and the first transmission instruction, the physical addresses of multiple third storage areas are obtained; the third storage area is a storage area in the cache that stores corresponding third key-value data. Based on the physical address of each third storage area, the key-value data groups in each third storage area are transferred to the memory.

6. The data processing method according to claim 5, characterized in that, The memory includes a first buffer and a second buffer; the step of transferring key-value data groups from each third storage area to the memory based on the physical address of each third storage area includes: Transfer the key-value data groups from each of the third storage areas to the first buffer; After transferring the key-value data groups from each of the third storage areas to the first buffer, the method further includes: Based on each second key value data, a second transmission instruction is obtained, which is used to instruct the transmission of at least one fourth key value data; the fourth key value data is any second key value data in the buffer that is located after the third key value data and needs to be transmitted to the memory. Based on the physical mapping table and the second transmission instruction, the physical addresses of multiple fourth storage areas are obtained; the fourth storage area is the storage area in the cache that stores the corresponding fourth key-value data. Based on the physical address of each fourth storage area, the key-value data groups in each fourth storage area are transferred to the second buffer.

7. The data processing method according to claim 5, characterized in that, Based on the second key-value data in the memory, the attention weight of the second word element is obtained, including: Obtain the historical attention weight of the second word element; the historical attention weight is calculated by using a first number of second key-value data. Obtain all fifth key-value data in the memory; the fifth key-value data is any second key-value data in the memory that did not participate in the historical attention weight calculation; Based on the second word element and each fifth key value data, obtain the new attention weight; The historical attention weight and the first quantity are updated based on the newly added attention weight to obtain the current attention weight until the first quantity equals the second quantity; the second quantity is the total number of second key-value data required to calculate the attention weight of the second word; if the first quantity equals the second quantity, the current attention weight is used as the attention weight of the second word.

8. A data processing apparatus applied to a tensor processor, the tensor processor comprising a memory and a cache; characterized in that, The data processing device includes: The reading module is used to obtain multiple first word elements based on the word element sequence; the first word element is any word element in the word element sequence; The processing module is used to obtain the first key-value data corresponding to each first word element; the first key-value data is the data formed by the key vector and value vector of the corresponding first word element; And, store each first key-value data into the cache; And, based on each first word element, obtain the second word element; the second word element is any word element for which the attention weight between it and each first word element has not been calculated; Furthermore, based on the second lexical unit, multiple second key-value data in the cache are transferred to the memory; the second key-value data are any key-value data required in the cache to calculate the attention weight of the second lexical unit; the second key-value data in the memory is used to participate in attention calculation to obtain the attention weight of the second lexical unit; The cache is pre-divided into multiple first storage areas, each with the same capacity; the processing module is further configured to obtain multiple key-value data groups based on each first key-value data; each key-value data group includes at least one first key-value data; and the storage capacity required for each key-value data group is less than or equal to the capacity of the first storage area. Furthermore, based on each first storage area, a second storage area corresponding one-to-one with each first key-value data is obtained; the second storage area is any storage area in each first storage area that does not store data. In addition, each key-value data is stored in the corresponding second storage area, and a corresponding physical mapping table is established; the physical mapping table includes at least the physical address of the first key-value data corresponding to any first word element located in the corresponding second storage area in the cache; Furthermore, based on the physical mapping table, the key-value data groups corresponding to each second key-value data are transmitted to the memory.

9. The data processing apparatus according to claim 8, characterized in that, The processing module is further configured to obtain a third word element and a fourth word element based on each first word element; the third word element is any word element among the first word elements for which the corresponding first key value data has not been obtained; the fourth word element is any word element among the first word elements for which the corresponding first key value data has been obtained. Furthermore, based on the third lexical unit, each fourth lexical unit is traversed. If each fourth lexical unit has a lexical unit that is the same as the third lexical unit, that is, a fifth lexical unit, then the first key-value data corresponding to the fifth lexical unit is used as the first key-value data corresponding to the third lexical unit. The physical address of the second storage area corresponding to the third lexical unit and the fifth lexical unit is the same. Otherwise, the first key-value data of the third lexical unit is obtained based on the Transformer model.

10. The data processing apparatus according to claim 9, characterized in that, The tensor processor also includes a counter that corresponds one-to-one with each key-value data group; the initial value of the counter is equal to the sum of the number of first tokens corresponding to each second key-value data in the key-value data group; the processing module is further configured to, for any key-value data group, decrement the value of the counter corresponding to the key-value data group by 1 each time any second key-value data in the key-value data group is called. Furthermore, for any key-value data group, if the value of the counter corresponding to the key-value data group is 0, then the key-value data group is released from the memory.

11. The data processing apparatus according to claim 8, characterized in that, The processing module is further configured to obtain a first data capacity based on the first key-value data; the first data capacity is equal to the data volume of the first key-value data; And, based on the memory, obtain the second data capacity; The second data capacity is the capacity of the memory; And, based on the second data capacity and the first data capacity, obtain the maximum number; The maximum number is equal to the ratio of the second data capacity to the first data capacity rounded down; Furthermore, based on the maximum number, multiple first storage regions are divided within the cache.

12. The data processing apparatus according to any one of claims 8 to 11, characterized in that, The processing module is further configured to obtain a first transmission instruction based on each second key value data, the first transmission instruction being used to instruct the transmission of at least one third key value data; the third key value data being any second key value data in the buffer that currently needs to be transmitted to the memory. Furthermore, based on the physical mapping table and the first transmission instruction, the physical addresses of multiple third storage areas are obtained; the third storage area is a storage area in the cache that stores corresponding third key-value data; Furthermore, based on the physical address of each third storage area, the key-value data groups in each third storage area are transferred to the memory.

13. The data processing apparatus according to claim 12, characterized in that, The memory includes a first buffer and a second buffer; the processing module is further configured to transfer key-value data groups from each of the third storage areas to the first buffer; Furthermore, based on each second key-value data, a second transmission instruction is obtained, the second transmission instruction being used to instruct the transmission of at least one fourth key-value data; the fourth key-value data is any second key-value data in the buffer that is located after the third key-value data and needs to be transmitted to the memory; Furthermore, based on the physical mapping table and the second transmission instruction, the physical addresses of multiple fourth storage areas are obtained; the fourth storage area is a storage area in the cache that stores corresponding fourth key-value data; Furthermore, based on the physical address of each fourth storage area, the key-value data groups in each fourth storage area are transferred to the second buffer.

14. The data processing apparatus according to claim 12, characterized in that, The processing module is further configured to obtain the historical attention weight of the second word element; the historical attention weight is calculated by a first number of second key-value data. In addition, all fifth key-value data in the memory are obtained; the fifth key-value data is any second key-value data in the memory that did not participate in the historical attention weight calculation; Furthermore, based on the second lexical unit and each fifth key value data, the newly added attention weights are obtained; And, based on the newly added attention weight, update the historical attention weight and the first quantity to obtain the current attention weight, until the first quantity equals the second quantity; The second quantity is the total number of second key-value data required when calculating the attention weight of the second word; If the first quantity equals the second quantity, then the current attention weight is used as the attention weight of the second word.

15. A computing device, characterized in that, It includes a processor and a memory; the processor is connected to the memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to cause the computing device to perform the data processing method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1 to 7.

Images