MLA-based sequence processing method and related device, system and storage medium

Abstract

The present application discloses a sequence processing method based on MLA and related devices, systems and storage media. The method includes: decompressing the first query feature based on low-rank compression and performing rotation position embedding to obtain the target query feature; segmenting the first key-value feature based on low-rank compression to obtain the first sub-key-value feature to be processed by different computing units; rotating the first sub-key-value feature to be processed by different computing units to obtain the second sub-key-value feature; communicating based on each computing node to obtain the second key-value feature of the first key-value feature after the rotation position embedding; obtaining the target key feature and target value feature based on the second key-value feature; and executing a multi-head attention mechanism based on the target query feature, target key feature and target value feature. The above scheme can reduce the communication volume when performing sequence processing based on MLA to improve processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a sequence processing method based on MLA and related devices, systems and storage media. Background Art

[0002] In recent years, with the significant development of large language models, many high-performance network architectures have emerged. For example, Multi-head Latent Attention (MLA) significantly reduces the memory requirements of the KV cache during inference, thanks to the fact that MLA performs low-rank compression on Q, K, and V before executing MHA.

[0003] This study found that the existing MLA, which performs parallel aggregation operations within the MHA, leads to a dramatic increase in compute node communication traffic during MLA-based sequence processing, thereby impacting the ultimate processing efficiency of the compute nodes. Therefore, reducing the compute node communication traffic during MLA-based sequence processing to improve the ultimate processing efficiency has become an urgent issue. Summary of the Invention

[0004] The main technical problem solved by this application is to provide an MLA-based sequence processing method and related devices, systems and storage media, which can reduce the communication volume of computing nodes when performing sequence processing based on MLA, so as to improve the final processing efficiency of the computing nodes.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a sequence processing method based on MLA, including: the first query feature based on low-rank compression is decompressed and then rotated and embedded to obtain the target query feature of the multi-head attention mechanism; wherein the first query feature is obtained from the text sequence to be responded; the first key-value feature based on low-rank compression is split to obtain the first sub-key-value feature to be processed by different computing units in each computing node; the first sub-key-value feature assigned for processing is rotated and embedded by different computing units in each computing node to obtain the second sub-key-value feature; communication is performed based on each computing node to obtain the second key-value feature of the first key-value feature after the rotated position embedding; based on the second key-value feature, the target key feature and target value feature of the multi-head attention mechanism are obtained; the multi-head attention mechanism is executed based on the target query feature, target key feature and target value feature to complete the MLA processing.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a sequence processing device based on MLA, including: a position embedding module, a grouping and segmenting module, a distribution processing module, a sequence aggregation module, a feature processing module, and a multi-head processing module. The position embedding module is used to rotate the position of the first query feature compressed based on low rank compression after decompression to obtain the target query feature of the multi-head attention mechanism; wherein the first query feature is obtained from the text sequence to be responded; the grouping and segmenting module is used to split the first key-value feature based on low rank compression to obtain the first sub-key-value feature to be processed by different computing units in each computing node; the distribution processing module is used to rotate the position of the first sub-key-value feature assigned for processing by different computing units in each computing node to obtain the second sub-key-value feature; the sequence aggregation module is used to communicate based on each computing node to obtain the second key-value feature of the first key-value feature after the rotated position embedding; the feature processing module is used to obtain the target key feature and target value feature of the multi-head attention mechanism based on the second key-value feature; the multi-head processing module is used to execute the multi-head attention mechanism based on the target query feature, target key feature and target value feature to complete the MLA processing.

[0007] In order to solve the above technical problems, the third aspect of the present application provides a sequence processing system, including multiple computing nodes, each computing node contains several computing units, and the sequence processing system is used to execute program instructions to implement the MLA-based sequence processing method in the above first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the MLA-based sequence processing method of the first aspect.

[0009] In the above scheme, the first query feature based on low-rank compression is compressed and then rotated and embedded to obtain the target query feature of the multi-head attention mechanism. The first query feature is obtained from the text sequence to be responded and is divided based on the first key-value feature of low-rank compression to obtain the first sub-key-value feature to be processed by different computing units in each computing node. Then, different computing units in each computing node rotate and embed the assigned first sub-key-value feature to obtain the second sub-key-value feature, so as to communicate based on each computing node to obtain the second key-value feature of the first key-value feature after the rotation position embedding, and based on the second key-value feature, obtain the target key feature and target value feature of the multi-head attention mechanism, and then execute the multi-head attention mechanism based on the target query feature, target key feature and target value feature to complete the MLA processing. Since the parallel aggregation operation is separated from the multi-head attention MHA, the parallel aggregation operation is placed before the multi-head attention MHA after obtaining the low-rank compressed first key-value feature. At this time, only the low-rank compressed first key-value feature can be communicated. Compared with executing the parallel aggregation operation in the multi-head attention MHA, the communication volume of the computing nodes can be greatly reduced. Therefore, the communication volume of the computing nodes can be reduced when performing sequence processing based on MLA, so as to improve the final processing efficiency of the computing nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of an embodiment of the sequence processing method based on MLA of the present application;

[0011] Figure 2a This is a process diagram of an embodiment of the sequence processing method based on MLA of the present application;

[0012] Figure 2b This is a schematic diagram of an embodiment of the present application performing dimensional segmentation on the first key-value feature;

[0013] Figure 2c This is a schematic diagram of an embodiment of CP group communication of the present application;

[0014] Figure 2d This is a schematic diagram of an embodiment of SP group communication of the present application;

[0015] Figure 2e This is a schematic diagram of the effect of an embodiment of the sequence processing method based on MLA of the present application;

[0016] Figure 3 This is a schematic diagram of the framework of an embodiment of a sequence processing device based on MLA in the present application;

[0017] Figure 4 This is a schematic diagram of the framework of an embodiment of the sequence processing system of the present application;

[0018] Figure 5It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0020] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0021] The terms "system" and "network" are often used interchangeably in this document. The term "and / or" is simply a description of an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the fragment " / " generally indicates that the related objects are in an "or" relationship. Furthermore, "multiple" in this document refers to two or more than two.

[0022] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the sequence processing method based on MLA of the present application. Specifically, it may include the following steps:

[0023] Step S11: The first query feature based on low-rank compression is decompressed and then rotated and embedded to obtain the target query feature of the multi-head attention mechanism.

[0024] In the disclosed embodiment, the first query feature can be obtained from a sequence of texts to be responded to. It should be noted that, depending on the actual application scenario, the specific content of the text to be responded to may also be different. For example, in a teaching scenario, the text to be responded to may include but is not limited to: "Please explain a certain knowledge point", "Please analyze the following test questions", "Please generate a handout on a certain knowledge point", etc.; or, in a medical scenario, the text to be responded to may include but is not limited to: "What are the indications of a certain drug", "Can two certain drugs be taken at the same time", "Is it necessary to fast before a certain medical examination", etc. Of course, the above examples are only a few possible examples of texts to be responded to in actual applications. The specific content of the text to be responded to is not limited here, and no examples are given one by one.

[0025] In one implementation scenario, after obtaining the text to be responded to, it can be embedded to obtain the embedded representation of the text to be responded to. Then, the embedded representation of the text to be responded to can be sent to the intelligent dialogue model containing MLA (i.e., multi-head latent attention), so that each network layer in the intelligent dialogue model can process it in sequence until the previous network layer of MLA is processed. The hidden layer features of the previous network layer of MLA can be obtained, and then the query low-rank projection matrix can be used to perform low-rank compression on the hidden layer features to obtain the first query feature. Please refer to Figure 2a , Figure 2a This is a process diagram of an embodiment of the sequence processing method based on MLA of this application. Figure 2a As shown, for the convenience of description, the query-related low-rank compression can be recorded as q_down, and the aforementioned hidden layer features can be recorded as h t , then the first query feature can be expressed as:

[0026]

[0027] In the above formula, w DQ represents the query low-rank projection matrix, It should be noted that the above is only a brief description of low-rank projection. The specific process can be found in the technical details of MLA, which will not be repeated here.

[0028] In one implementation scenario, after obtaining the low-rank compressed first query feature, the query restoration projection matrix can be used to project the first query feature and then perform dimensional slicing to obtain the shard query features of each attention head in the multi-head attention. After that, for each shard query feature, on the one hand, the shard query feature can be rotated and embedded, and on the other hand, the shard query feature can be fused (e.g., spliced) with its own shard query feature after the rotation and embedding to obtain the target query feature of each attention head. For ease of understanding, the aforementioned projection operation can be recorded as q_up, and the shard query feature after the projection operation and dimensional slicing can be expressed as:

[0029]

[0030] In the above formula, split represents dimensional splitting, and Q_nope and Q_rope both represent the shard query features after projection and dimensional splitting. They are essentially the same, with the main difference being that the latter also requires rotational position embedding, meaning the latter can be understood as an identical copy of the former. Based on this, for ease of description, the rotational position encoding can be denoted as Rotary_Embed, and the shard query features after rotational position embedding can be expressed as:

[0031] Q_rope=Rotary_Embed(Q_rope)

[0032] In the above formula, Q_rope on the right side of the equal sign represents the sharded query feature that needs to be embedded in the rotational position after the projection operation and dimensional splitting, and Q_rope on the left side of the equal sign represents the sharded query feature after the rotational position embedding. It should be noted that for the specific process of rotational position embedding, please refer to the technical details of Rotary Position Embedding (RoPE), which will not be repeated here. On this basis, the sharded query features after the projection operation and dimensional splitting and the sharded query features after the rotational position embedding can be fused to obtain the target query feature. For the sake of convenience of description, taking the fusion achieved through feature splicing as an example, the target query feature Q can be expressed as:

[0033] Q=concate(Q_nope,Q_rope)

[0034] In the above formula, "concate" represents concatenation, "Q_nope" represents the sharded query features after projection and dimensional slicing, and "Q_rope" represents the sharded query features after rotation and position embedding. It should be noted that the above-mentioned process operations for obtaining the target query matrix can be performed in any computing node in the sequence processing system. For example, the sequence processing system may include multiple computing nodes (e.g., two computing nodes, three computing nodes, four computing nodes, etc.), each of which may contain several computing units (e.g., GPUs, etc.). Furthermore, the sequence processing system may also include computing resources other than computing nodes, and the above-mentioned process operations for obtaining the target query matrix can be performed in computing resources other than computing nodes in the sequence processing system, without limitation here. For the specific structure of the sequence processing system, please refer to the following embodiments of the sequence processing system and will not be described in detail here. Furthermore, to facilitate subsequent multi-head attention mechanism processing, the feature dimension of the target query feature at the hidden layer scale can be expressed in the form of n*d1, where n represents the number of multi-heads and d1 represents the target dimension. The multiplication of the number of multi-heads n and the target dimension d1 equals the feature dimension of the target query feature at the hidden layer scale.

[0035] Step S12: Split the first key-value feature based on low-rank compression to obtain the first sub-key-value features to be processed by different computing units in each computing node.

[0036] In one implementation scenario, the first key-value feature of low-rank compression can be obtained with reference to the first query feature of the aforementioned low-rank compression. For example, in a multi-round conversation process with an intelligent conversation model that includes MLA (i.e., multi-head potential attention), the hidden layer features of the conversation text sequence in the previous network layer of MLA can be obtained in each round of conversation, so that the key-value low-rank projection matrix can be used to perform low-rank compression on the hidden layer features to obtain the first key-value features and cache them, thereby greatly reducing the video memory space required for the KV cache, and then the cached first key-value features can be obtained for segmentation in the current round of conversation. For ease of description, the key-value related low-rank compression can be recorded as kv_down, and the aforementioned hidden layer features can be recorded as h t , then the first key-value feature can be expressed as:

[0037]

[0038] In the above formula, represents the key-value low-rank projection matrix, It should be noted that the above is only a brief description of low-rank projection. The specific process can be found in the technical details of MLA, which will not be repeated here.

[0039] In one implementation scenario, as a possible implementation example, the low-rank compressed first key-value feature can be divided according to the total number of different computing units in each computing node. For example, the first key-value feature can be evenly divided into the above-mentioned total number of parts to be respectively allocated to different computing units in each computing node. For example, when there are M computing nodes and N computing units in each computing node, the first key-value feature can be divided into M*N first sub-key-value features. Of course, the above example is only a possible example of dividing the first key-value feature in actual application. Other possible situations are not limited here, and no examples are given one by one.

[0040] In another implementation scenario, as another possible implementation example, distinct from the aforementioned splitting method, the low-rank compressed first key-value feature can also be split sequentially based on the number of CP groups and the number of SP groups to obtain the first sub-key-value feature to be processed by different computing units in each computing node. For example, the first key-value feature is represented as a feature tensor whose feature size is the product of the batch size, sequence length, and hidden layer size. For ease of description, it can be expressed as B*S*H, or simply (B, S, H), where B represents the batch size, S represents the sequence length, and H represents the hidden layer size. After obtaining the low-rank compressed first key-value feature, the first key-value feature can be first split based on the number of CP groups in the feature dimension of the sequence length, for allocation to different computing nodes after the final split. Then, each sub-feature of the first key-value feature after the first split can be split a second time based on the feature dimension of the sequence length in the feature dimension of the sequence length, to obtain the first sub-key-value feature to be processed by different computing units in each computing node. That is, the first key feature with a feature size of (B, S, H) is split into CP groups CP_size for the first time, and CP groups CP_size of sub-features with a feature size of (B, S / CP_size, H) are obtained. After the final split, these sub-features are assigned to different computing nodes. Each sub-feature can then be split into SP groups SP_size for the second time, and SP groups of sub-features with a feature size of (B, S / CP_size / SP_size, H) are obtained, which are used as the first sub-key features assigned to different computing units. Please refer to Figure 2b , Figure 2b This is a schematic diagram of an embodiment of the present application for dimensional segmentation of the first key value feature. Figure 2b As shown, taking the CP group number CP_size and SP group number SP_size as 2 as an example, the first key value feature with feature size (B, S, H) can be first split into two sub-features with feature size (B, S / 2, H) according to the CP group number, and will be assigned to different computing nodes after the final split is completed, such as computing node 0 (i.e. Figure 2b Node0) and compute node 1 (such as Figure 2b Node1 in the middle), and then the first sub-feature is further divided into two sub-features with feature sizes of (B, S / 2 / 2, H) according to the number of SP groups as the first sub-key value features, and the calculation nodes 0 (i.e. Figure 2b Computational unit 0 (i.e. Figure 2b GPU0) and compute unit 1 (i.e. Figure 2bGPU1 in the middle), and the second sub-feature is further divided into two sub-features with feature sizes of (B, S / 2 / 2, H) according to the number of SP groups as the first sub-key value feature, and the computing nodes 1 (i.e. Figure 2b Node1) in the computing unit 0 (i.e. Figure 2b GPU0) and compute unit 1 (i.e. Figure 2b GPU1 in the example). Of course, the above example is only one possible example of how to split the first key-value feature in actual application. Other possible situations are not limited here, and no further examples are given. For ease of description, the above splitting process can be referred to as split, and the first sub-key-value feature can be expressed as:

[0041]

[0042] In the above formula, Both represent the first sub-key value characteristics, that is, the two are essentially the same. The main difference is that the latter also needs to continue to perform rotation position embedding, that is, in order to further perform rotation position embedding, the former is backed up and renamed to the latter.

[0043] Step S13: Different computing units in each computing node respectively perform rotation position embedding on the assigned first sub-key value feature to obtain a second sub-key value feature.

[0044] Specifically, the specific process of embedding the first sub-key value feature in the rotation position can refer to the specific process of embedding the first query feature in the rotation position after decompression. Figure 2b , computing node 0 (i.e. Figure 2b Node0) in computing unit 0 (i.e. Figure 2b GPU0), computing node 0 (i.e. Figure 2b Node0) in computing unit 1 (i.e. Figure 2b GPU1), computing node 1 (i.e. Figure 2b Node1) in computing unit 0 (i.e. Figure 2b GPU0), computing node 1 (i.e. Figure 2b Node1) in computing unit 1 (i.e. Figure 2b The GPU 1 in the GPU can independently process data, that is, it can perform rotation position embedding on each of its assigned first sub-key value features to obtain the corresponding second sub-key value features. The following briefly describes the specific process of each computing unit independently performing rotation position embedding. For ease of description, for any first sub-key value feature, the first sub-key value feature after performing rotation position embedding can be expressed as:

[0045] K_rope=Rotary_Embed(K_rope)

[0046] In the above formula, Rotary_Embed represents rotational position embedding, K_rope on the right side of the equal sign is the first sub-key value feature, and K_rope on the left side of the equal sign is the first sub-key value feature after the rotational position embedding is performed. It should be noted that for the specific process of rotational position embedding, please refer to the technical details of Rotary Position Embedding (RoPE), which will not be repeated here. On this basis, the first sub-key value feature before the rotational position embedding is performed can be fused with the first sub-key value feature after the rotational position embedding is performed to obtain the second sub-key value feature. Taking the fusion achieved through feature splicing as an example, the second sub-key value feature can be expressed as:

[0047]

[0048] In the above formula, Indicates the second sub-key value feature, concate indicates splicing, It represents the first sub-key value feature before the rotation position embedding is performed, and K_rope represents the first sub-key value feature after the rotation position embedding is performed.

[0049] Step S14: communicate with each computing node to obtain a second key-value feature that is embedded in the first key-value feature after the rotation position.

[0050] In one implementation scenario, as described above, the low-rank compressed first key-value feature can be divided according to the total number of different computing units in each computing node. In this case, different computing nodes in each computing node can exchange their respective calculated second sub-key-value features with each other, so that different computing units in each computing node can obtain the second sub-key-value features calculated by other computing units, and then each computing unit can combine the various second sub-key-value features to obtain the second key-value feature embedded in the first key-value feature through rotational position encoding.

[0051] In another implementation scenario, as mentioned above, different from the aforementioned situation, the first key-value feature of low-rank compression can also be divided in sequence based on the number of CP groups and the number of SP groups to obtain the first sub-key-value feature to be processed by different computing units in each computing node. In this case, CP group communication can be performed between different computing nodes and then SP group communication can be performed within the same computing node to obtain the second key-value feature embedded in the rotated position of the first key-value feature. In the above method, CP group communication is performed first and then SP group communication is performed during the execution of the parallel aggregation operation, that is, communication is performed between computing nodes first and then communication is performed within the computing node. Compared with performing SP group communication first and then CP group communication, the communication volume between computing nodes can be reduced. The following is an explanation from the two aspects of CP group communication and SP communication:

[0052] When performing CP group communication, the allgather function can be used to perform CP group communication on the second sub-key features of different compute nodes based on the CP group configuration information between different compute nodes. It should be noted that allgather is a communication primitive whose core function is to aggregate relevant data originally distributed across various compute units to all compute units, ultimately ensuring that each compute unit has a complete copy of the data. Of course, the above introduction is only a brief description of allgather. For more detailed technical details, please refer to allgather.com, which will not be elaborated here.

[0053] In a specific implementation scenario, CP group communication occurs between computing nodes, and the configuration information of the CP group may include computing unit pairs that communicate between different computing nodes. Figure 2b Taking the scenario shown as an example, the CP group configuration information may include: (Node0GPU0, Node1GPU0), (Node0GPU1, Node1GPU1). That is, when the CP group communicates, computing unit 0 in computing node 0 communicates with computing unit 0 in computing node 1, and computing unit 1 in computing node 0 communicates with computing unit 1 in computing node 1. Of course, the above example is only one possible example in actual application, and other possible scenarios will not be given one by one here.

[0054] In a specific implementation scenario, please refer to Figure 2c , Figure 2c This is a schematic diagram of an embodiment of CP group communication of the present application. Computing node 0 (ie Figure 2c Node0) contains 8 computing units (e.g., 8 GPUs). Figure 2c They are denoted as 0 to 7 respectively, computing node 1 (i.e. Figure 2c Node1) contains 8 computing units (e.g., 8 GPUs). Figure 2cIn the table, they are abbreviated as 0 to 7 respectively. In addition, the first computing unit in computing node 0 contains the second sub-key value feature e00, the second computing unit in computing node 0 contains the second sub-key value feature e01, the third computing unit in computing node 0 contains the second sub-key value feature e02, the fourth computing unit in computing node 0 contains the second sub-key value feature e03, the fifth computing unit in computing node 0 contains the second sub-key value feature e04, the sixth computing unit in computing node 0 contains the second sub-key value feature e05, the seventh computing unit in computing node 0 contains the second sub-key value feature e06, and the eighth computing unit in computing node 0 contains the second sub-key value feature e07; Similarly, the first computing unit in computing node 1 contains the second sub-key value feature e10, the second computing unit in computing node 1 contains the second sub-key value feature e11, the third computing unit in computing node 1 contains the second sub-key value feature e12, the fourth computing unit in computing node 1 contains the second sub-key value feature e13, the fifth computing unit in computing node 1 contains the second sub-key value feature e14, the sixth computing unit in computing node 1 contains the second sub-key value feature e15, the seventh computing unit in computing node 1 contains the second sub-key value feature e16, and the eighth computing unit in computing node 1 contains the second sub-key value feature e17. On this basis, if the configuration information of the CP group includes: (Node0GPU0, Node1GPU0), (Node0GPU1, Node1GPU1), (Node0GPU2, Node1GPU2), (Node0GPU3, Node1GPU3), (Node0GPU4, Node1GPU4), (Node0GPU5, Node1GPU5), (Node0GPU6, Node1GPU6), (Node0GPU7, Node1GPU7), then after executing the allgather function based on the configuration information of the above CP group, the calculation node 0 (i.e. Figure 2c The data of computing unit 0 in Node0 is the second sub-key value feature e00 and the second sub-key value feature e10. Figure 2c The data of computing unit 1 in Node0 is the second sub-key value feature e01 and the second sub-key value feature e11. Figure 2c The data of computing unit 2 in Node0 is the second sub-key value feature e02 and the second sub-key value feature e11. Figure 2c The data of computing unit 3 in Node0 is the second sub-key value feature e03 and the second sub-key value feature e13. Figure 2c The data of computing unit 4 in Node0 is the second sub-key value feature e04 and the second sub-key value feature e14. Figure 2cThe data of computing unit 5 in Node0 is the second sub-key value feature e05 and the second sub-key value feature e15. Figure 2c The data of computing unit 6 in Node0 is the second sub-key value feature e06 and the second sub-key value feature e16. Figure 2c The data of computing unit 7 in Node0 is the second sub-key value feature e07 and the second sub-key value feature e17; similarly, computing node 1 (i.e. Figure 2c The data of computing unit 0 in Node1 is the second sub-key value feature e10 and the second sub-key value feature e00. Figure 2c The data of computing unit 1 in Node1 is the second sub-key value feature e11 and the second sub-key value feature e01. Figure 2c The data of computing unit 2 in Node1 is the second sub-key value feature e12 and the second sub-key value feature e02. Figure 2c The data of computing unit 3 in Node1 is the second sub-key value feature e13 and the second sub-key value feature e03. Figure 2c The data of computing unit 4 in Node1 is the second sub-key value feature e14 and the second sub-key value feature e04. Figure 2c The data of computing unit 5 in Node1 is the second sub-key value feature e15 and the second sub-key value feature e05. Figure 2c The data of computing unit 6 in Node1 is the second sub-key value feature e16 and the second sub-key value feature e06. Figure 2c The data held by the computing unit 7 in Node1 is the second sub-key feature e17 and the second sub-key feature e07. That is, the communication volume of the CP group is 16*G (where G is the data volume of each second sub-key feature). Of course, Figure 2c The example shown is only one possible example of CP group communication in actual application. Other possible situations are not limited here and will not be given examples one by one.

[0055] When performing SP group communication, the allgather function can be used to perform SP group communication on the second sub-key feature of the same compute node based on the configuration information about the SP group within the same compute node. It should be noted that unlike the aforementioned CP group communication, where allgather is performed on different compute nodes, SP group communication is performed on the same compute node.

[0056] In a specific implementation scenario, the SP group communication occurs within the computing node, and the configuration information of the SP group may include the computing unit pairs that communicate between the same computing nodes. Figure 2b Taking the scenario shown as an example, the CP group configuration information may include: (Node0GPU0, Node0GPU1), (Node1GPU0, Node1GPU1). That is, when the CP group communicates, computing unit 0 in computing node 0 communicates with computing unit 1 in computing node 0, and computing unit 0 in computing node 1 communicates with computing unit 1 in computing node 1. Of course, the above example is only one possible example in actual application, and other possible scenarios are not listed here one by one.

[0057] In a specific implementation scenario, please refer to Figure 2d , Figure 2d This is a schematic diagram of an embodiment of the SP group communication of the present application. Computing node 0 (ie Figure 2d Node0) contains 8 computing units (e.g., 8 GPUs). Figure 2d They are denoted as 0 to 7 respectively, computing node 1 (i.e. Figure 2d Node1) contains 8 computing units (e.g., 8 GPUs). Figure 2d In addition, after executing CP group communication, the data contained in different computing units in each computing node can refer to the above description, which will not be repeated here. On this basis, if computing node 0 (i.e. Figure 2d The configuration information of the SP group of Node0 in the figure includes: (Node0GPU0, Node0GPU1), (Node0GPU1, Node0GPU2), (Node0GPU2, Node0GPU3), (Node0GPU3, Node0GPU4), (Node0GPU4, Node0GPU5), (Node0GPU5, Node0GPU6), (Node0GPU6, Node0GPU7), (Node0GPU7, Node0GPU0); similarly, the configuration information of the SP group of computing node 1 (i.e. Figure 2dThe configuration information of SP group of Node1 in Node1 includes: (Node1GPU0, Node1GPU1), (Node1GPU1, Node1GPU2), (Node1GPU2, Node1GPU3), (Node1GPU3, Node1GPU4), (Node1GPU4, Node1GPU5), (Node1GPU5, Node1GPU6), (Node1GPU6, Node1GPU7), (Node1GPU7, Node1GPU0). After executing the allgather function based on the configuration information of the above SP group, the computing node 0 (i.e. Figure 2d Node0) and computing node 1 (i.e. Figure 2d The data owned by each computing unit in Node1) are all e00~e17 (that is, the second key-value features after summarizing all the second sub-key-value features). It should be noted that, in this example, if SP group communication is performed first and then CP group communication is performed, then since after the SP communication is performed first, the data owned by each computing unit in computing node 0 and computing node 1 are all 8*G (where G is the data volume of each second sub-key-value feature), when the CP group communication is performed again, the communication volume of the CP group will reach 16*8*G (where G is the data volume of each second sub-key-value feature), which is much larger than the communication volume of the aforementioned CP group 8*G (where G is the data volume of each second sub-key-value feature). Therefore, in the embodiment of the present disclosure, performing CP group communication first and then performing SP group communication can significantly reduce the communication volume between computing nodes.

[0058] In one implementation scenario, in order to facilitate the processing of the second sub-key value feature when CP group communication is first performed and then SP group communication is performed, the second sub-key value feature can be unsqueezed first, and then CP group communication is performed on the second sub-key value feature after the unsqueeze operation is performed, and finally SP group communication is performed on the second sub-key value feature after the CP group communication is performed, so that each computing unit in different computing nodes can obtain the complete second key value feature. For ease of description, the above process can be expressed as:

[0059]

[0060]

[0061]

[0062] In the above formula, Indicates the second sub-key value feature, Indicates the second sub-key value feature after the unsqueeze operation is performed. Indicates the allgather function, CP_GROUP indicates the configuration information of the CP group, Indicates the second sub-key value feature after executing CP communication, SP_GROUP indicates the configuration information of the SP group, Indicates the key value feature after executing SP group communication. As a possible example, after executing CP group communication and SP group communication, each computing unit in different computing nodes can obtain the third sub-key value feature (as mentioned above ), which can be characterized as a feature tensor whose feature size is the product of batch size, number of CP groups, sequence length divided by number of SP groups, and hidden layer size, that is, it can be expressed as (B, CP_size, S / SP_size, H), where the specific meaning of each parameter can refer to the aforementioned related description and will not be repeated here. On this basis, the target dimension can be reshaped (reshaped) based on the third sub-key feature to obtain the second key feature. For example, the CP dimension of the third sub-key feature can be flattened on the SP dimension through a reshape operation. It should be noted that the target dimension includes the feature dimension of the number of CP groups and the feature dimension of the sequence length divided by the number of SP groups, and the second key feature is characterized as a feature tensor whose feature size is the product of batch size, sequence length, and hidden layer size (that is, B*S*H). For the convenience of description, the second key feature can be expressed as:

[0063]

[0064] In the above formula, Represents the third sub-key value feature, RESHAPE represents the reshaping operation, Indicates the second key-value feature.

[0065] Step S15: Based on the second key-value feature, obtain the target key feature and target value feature of the multi-head attention mechanism.

[0066] Specifically, after obtaining the second key-value feature, the feature dimension of the rotated position embedding can be used to split the second key-value feature into a third key-value feature representing low-rank compression and a first key-value feature representing the rotated position embedding. Decompression is then performed based on the third key-value feature to obtain the second key-value feature and the target value feature of the multi-head attention mechanism. The first and second key features are then fused to obtain the target key feature of the multi-head attention mechanism. It should be noted that the feature dimension of the rotated position embedding represents the feature dimension after the key feature vector in MLA is rotated and embedded. For example, taking the second key-value feature represented as (B, S, 576) (i.e., with a hidden layer size H of 576), if the feature dimension of the rotated position embedding is 64, then 64 of the hidden layer dimensions of the second key-value feature belong to the first key feature. Of course, the above example is only one possible scenario in practical application. Other possible scenarios are not limited here and will not be given one by one.

[0067] In a specific implementation scenario, the feature dimension of the second key-value feature in the hidden layer size can be split based on the feature dimension of the rotation position embedding to obtain the third key-value feature and the first key feature. It should be noted that the feature dimension of the rotation position embedding and the feature dimension of the first key feature in the hidden layer size are the same. Still taking the second key-value feature represented as (B, S, 576) as an example, if the feature dimension of the rotation position embedding is 64, the feature dimension of the second key-value feature in the hidden layer size can be split into a 64-dimensional first key feature (B, S, 64). For example, the last 64 dimensions of the feature dimension of the second key-value feature in the hidden layer size can be considered to belong to the first key feature. On this basis, the remaining part can be regarded as a low-rank compressed third key-value feature (B, S, 512). Of course, the above example is only one possible scenario in actual application. Other possible scenarios are not limited here, and no examples are given one by one. For the convenience of description, the third key-value feature and the first key feature can be expressed as:

[0068]

[0069] In the above formula, Represents the second key feature, split represents segmentation, K_rope represents the first key feature, Indicates the third key-value feature.

[0070] In a specific implementation scenario, after obtaining the low-rank compressed third key-value feature, it can be decompressed to separate K and V. Specifically, the third key-value feature can be processed based on the key-value decompression parameter to obtain the decompressed key-value feature, and then the key-value splitting is performed based on the decompressed key-value feature to obtain the second key feature and the target value feature. In order to facilitate the description of the decompression operation of the key-value feature, it can be recorded as kv_up, such as by the decompression parameter wUKV For details on the implementation, please refer to the technical details of MLA, which will not be repeated here. On this basis, the specific process of decompressing the third key value feature to obtain the second key feature and the target value feature can be expressed as:

[0071]

[0072] In the above formula, represents the third key-value feature, split represents the key-value split, K_nope represents the second key feature, and V represents the target value feature. In addition, to facilitate the subsequent multi-head attention mechanism processing, the feature dimension of the hidden layer size of the value feature obtained after the key-value split can be expressed as n*d2, where n represents the number of heads and d2 represents the target dimension. The multiplication of the number of heads n and the target dimension d2 can be the same as the feature dimension of the hidden layer size of the value feature obtained after the key-value split.

[0073] In a specific implementation scenario, after obtaining the first key feature and the second key feature, the two can be fused to incorporate position encoding information into the key feature information. Specifically, the feature dimension of the first key feature at the hidden layer size can be copied based on the number of heads of the multi-head attention mechanism to obtain the third key feature, and the feature dimension of the second key feature at the hidden layer size can be reshaped into the number of heads multiplied by the target dimension to obtain the fourth key feature. It should be noted that the number of heads multiplied by the target dimension is equal to the feature dimension of the second key feature at the hidden layer size. Taking the number of heads as n as an example, if the feature dimension of the first key feature at the hidden layer size is 64, the feature dimension of the third key feature at the hidden layer size is n*64. In addition, if the feature dimension of the second key feature at the hidden layer size is H, it can be reshaped into n*d3, where d3 is the target dimension, and the number of heads n multiplied by the target dimension d3 is equal to the feature dimension H of the second key feature at the hidden layer size. On this basis, the target key features of the multi-head attention mechanism can be obtained by splicing the feature dimensions of the third key feature and the fourth key feature in the hidden layer size. Taking the above example as an example, the feature dimension of the target key feature obtained by splicing the feature dimensions of the hidden layer size can be expressed as n*(d3+64) in the hidden layer size. It should be noted that the feature dimensions of other dimensions (i.e. batch size, sequence length) can still be expressed as B and S respectively. Of course, the above examples are only a few possible examples in actual application. Other possible situations are not limited here, and no examples are given one by one. For the convenience of description, the third key feature can be expressed as:

[0074] HK_rope=K_rope.repeat(num.heads)

[0075] In the above formula, num.heads represents the number of heads, repeat represents the copy operation, K_rope represents the first key feature, and HK_rope represents the third key feature. Based on this, the target key feature of the multi-head attention mechanism can be expressed as:

[0076] K=concate(K_nope,HK_rope)

[0077] In the above formula, HK_rope represents the third key feature, K_nope represents the fourth key feature, and K represents the target key feature.

[0078] Step S16: Execute a multi-head attention mechanism based on the target query feature, target key feature, and target value feature to complete the MLA process.

[0079] In an implementation scenario, after obtaining the target query features, target key features, and target value features of the multi-head attention mechanism, the multi-head attention mechanism can be executed based on them to complete the MLA processing. For ease of description, the output of the multi-head attention mechanism can be expressed as:

[0080] Atten_out =MHA(Q,K,V)

[0081] In the above formula, "Atten_out" represents the output of the multi-head attention mechanism, "MHA" stands for the multi-head attention mechanism, and "Q," "K," and "V" represent the target query feature, target key feature, and target value feature, respectively. For the specific processing process, please refer to the technical details of the multi-head attention mechanism (MHA), which will not be repeated here.

[0082] In one implementation scenario, after completing MLA, decoding can be performed based on the output features after MLA processing (such as the output result Atten_out of the aforementioned multi-head attention mechanism) to obtain the response result of the text sequence to be responded. It should be noted that the response result may include but is not limited to: at least one data type of text, image, video, audio, and table, which is not limited here. For example, when the text sequence to be responded is the text sequence "Please explain a certain knowledge point" in an educational scenario, the response result can be the text sequence "The meaning of a certain knowledge point is..., for example,..."; or, when the text sequence to be responded is the text sequence "What are the indications of a certain drug" in a medical scenario, the response result can be the text sequence "The indications of a certain drug can include..., but it is particularly noted that it does not include..." Of course, the above examples are only a few possible examples in actual application. Other possible situations are not limited here, and no examples are given one by one. It should be noted that the specific process of decoding based on output features can refer to the details of decoding technologies such as autoregressive decoding, and the decoding process will not be described in detail here. Please refer to Figure 2e , Figure 2e This is a schematic diagram of the effect of an embodiment of the sequence processing method based on MLA in this application. Figure 2e As shown, for a test sample with a sequence length of 32k under the 73B MoE (i.e., mixture of experts) model, if the parallel configuration of CP2 and SP8 and the hardware configuration of NPU910B 64 cards are adopted, the processing method of the embodiment disclosed in this application has a wps (word per second) index ( Figure 2e The blue line) is compared with the conventional method. Figure 2e The orange line shows a significant improvement.

[0083] In the above scheme, the first query feature based on low-rank compression is compressed and then rotated and embedded to obtain the target query feature of the multi-head attention mechanism. The first query feature is obtained from the text sequence to be responded and is divided based on the first key-value feature of low-rank compression to obtain the first sub-key-value feature to be processed by different computing units in each computing node. Then, different computing units in each computing node rotate and embed the assigned first sub-key-value feature to obtain the second sub-key-value feature, so as to communicate based on each computing node to obtain the second key-value feature of the first key-value feature after the rotation position embedding, and based on the second key-value feature, obtain the target key feature and target value feature of the multi-head attention mechanism, and then execute the multi-head attention mechanism based on the target query feature, target key feature and target value feature to complete the MLA processing. Since the parallel aggregation operation is separated from the multi-head attention MHA, the parallel aggregation operation is placed before the multi-head attention MHA after obtaining the low-rank compressed first key-value feature. At this time, only the low-rank compressed first key-value feature can be communicated. Compared with executing the parallel aggregation operation in the multi-head attention MHA, the communication volume of the computing nodes can be greatly reduced. Therefore, the communication volume of the computing nodes can be reduced when performing sequence processing based on MLA, so as to improve the final processing efficiency of the computing nodes.

[0084] See also Figure 3 , Figure 3It is a schematic diagram of the framework of an embodiment of the sequence processing device based on MLA in the present application. The sequence processing device 30 based on MLA includes: a position embedding module 31, a grouping and segmenting module 32, a distribution processing module 33, a sequence aggregation module 34, a feature processing module 35, and a multi-head processing module 36. The position embedding module 31 is used to perform rotation position embedding based on the first query feature of low-rank compression after decompression to obtain the target query feature of the multi-head attention mechanism; wherein the first query feature is obtained from the text sequence to be responded; the grouping and segmenting module 32 is used to perform segmentation based on the first key-value feature of low-rank compression to obtain the first sub-key-value feature to be processed by different computing units in each computing node. feature; a distribution processing module 33, which is used to rotate the position of the first sub-key-value feature assigned for processing by different computing units in each computing node to obtain the second sub-key-value feature; a sequence aggregation module 34, which is used to communicate based on each computing node to obtain the second key-value feature of the first key-value feature after the rotation position is embedded; a feature processing module 35, which is used to obtain the target key feature and target value feature of the multi-head attention mechanism based on the second key-value feature; a multi-head processing module 36, which is used to execute the multi-head attention mechanism based on the target query feature, the target key feature and the target value feature to complete the MLA processing.

[0085] In the above scheme, the sequence processing device 30 based on MLA is based on the first query feature of low-rank compression and is compressed and then rotated and embedded to obtain the target query feature of the multi-head attention mechanism, and the first query feature is obtained from the text sequence to be responded, and is segmented based on the first key-value feature of low-rank compression to obtain the first sub-key-value feature to be processed by different computing units in each computing node, and then the first sub-key-value feature assigned for processing is rotated and embedded by different computing units in each computing node to obtain the second sub-key-value feature, thereby communicating based on each computing node to obtain the first key-value feature rotated and embedded. The second key-value feature is embedded, and based on the second key-value feature, the target key feature and target value feature of the multi-head attention mechanism are obtained, and then the multi-head attention mechanism is executed based on the target query feature, target key feature and target value feature to complete the MLA processing. Since the parallel aggregation operation is separated from the multi-head attention MHA, the parallel aggregation operation is placed before the multi-head attention MHA after obtaining the low-rank compressed first key-value feature. At this time, only the low-rank compressed first key-value feature can be communicated. Compared with executing the parallel aggregation operation in the multi-head attention MHA, the communication volume of the computing nodes can be greatly reduced. Therefore, the communication volume of the computing nodes can be reduced when performing sequence processing based on MLA to improve the final processing efficiency of the computing nodes.

[0086] In some disclosed embodiments, the group segmentation module 32 is specifically used to segment the low-rank compressed first value feature in sequence based on the number of CP groups and the number of SP groups to obtain the first sub-key value feature to be processed by different computing units in each computing node. The sequence aggregation module 34 is specifically used to perform CP group communication between different computing nodes and then perform SP group communication within the same computing node to obtain the second key value feature embedded in the first key value feature after the rotation position.

[0087] In some disclosed embodiments, the first key-value feature is represented as a feature tensor whose feature size is the product of the batch size, the sequence length, and the hidden layer size. The grouping and splitting module 32 includes a first splitting submodule for performing a first split on the feature dimension of the sequence length of the first key-value feature based on the number of CP groups, so as to be distributed to different computing nodes after the final split is completed; the grouping and splitting module 32 includes a second splitting submodule for performing a second split on each sub-feature of the first key-value feature in the feature dimension of the sequence length after the first split based on the number of SP groups, so as to obtain the first sub-key-value feature to be processed by different computing units in each computing node.

[0088] In some disclosed embodiments, when performing CP group communication between different computing nodes, the sequence aggregation module 34 is specifically used to perform CP group communication on the second sub-key value features of different computing nodes based on the configuration information about the CP group between different computing nodes using the allgather function.

[0089] In some disclosed embodiments, when performing SP group communication within the same computing node, the sequence aggregation module 34 is specifically used to perform SP group communication on the second sub-key value feature of the same computing node based on configuration information about the SP group within the same computing node using the allgather function.

[0090] In some disclosed embodiments, after executing CP group communication and SP group communication successively, each computing unit in different computing nodes obtains a third sub-key-value feature, and the third sub-key-value feature is represented as a feature tensor whose feature size is the product of batch size, number of CP groups, sequence length divided by the number of SP groups, and hidden layer size. The MLA-based sequence processing device 30 also includes a feature reshaping module, which is used to reshape the target dimension based on the third sub-key-value feature to obtain a second key-value feature; wherein the target dimension includes the feature dimension of the number of CP groups and the feature dimension of the sequence length divided by the number of SP groups, and the second key-value feature is represented as a feature tensor whose feature size is the product of batch size, sequence length, and hidden layer size.

[0091] In some disclosed embodiments, the feature processing module 35 includes a slicing submodule for slicing a third key-value feature representing low-rank compression and a first key feature representing rotated position embedding from the second key-value feature based on the feature dimension of the rotated position embedding; the feature processing module 35 includes a decompression submodule for decompressing based on the third key-value feature to obtain the second key feature and the target value feature of the multi-head attention mechanism; the feature processing module 35 includes a fusion submodule for fusing based on the first key feature and the second key feature to obtain the target key feature.

[0092] In some disclosed embodiments, the sub-module is specifically used to split the feature dimension of the second key-value feature in the hidden layer size based on the feature dimension of the rotational position embedding, to obtain the third key-value feature and the first key feature; wherein, the feature dimension of the rotational position embedding and the feature dimension of the first key feature in the hidden layer size are the same.

[0093] In some disclosed embodiments, the decompression submodule includes a processing unit for processing the third key-value feature based on the key-value decompression parameter to obtain a decompressed key-value feature; the decompression submodule includes a splitting unit for performing key-value splitting based on the decompressed key-value feature to obtain a second key feature and a target value feature.

[0094] In some disclosed embodiments, the fusion submodule includes a copying unit for copying the feature dimension of the first key feature at the hidden layer size based on the number of multi-heads of the multi-head attention mechanism to obtain a third key feature; the fusion submodule includes a reshaping unit for reshaping the feature dimension of the second key feature at the hidden layer size to the number of multi-heads multiplied by the target dimension to obtain a fourth key feature; wherein the number of multi-heads multiplied by the target dimension is equal to the feature dimension of the second key feature at the hidden layer size; the fusion submodule includes a splicing unit for splicing the feature dimension of the third key feature and the fourth key feature based on the feature dimension of the hidden layer size to obtain a target key feature.

[0095] In some disclosed embodiments, the MLA-based sequence processing device 30 also includes a feature decoding module for performing a multi-head attention mechanism based on the target query feature, the target key feature, and the target value feature to complete the MLA processing, and then decoding based on the output features after the MLA processing to obtain a response result of the text sequence to be responded; wherein the response result includes: at least one data type among text, image, video, audio, and table.

[0096] See also Figure 4 , Figure 4It is a schematic diagram of the framework of an embodiment of the sequence processing system of the present application. The sequence processing system 40 includes a plurality of computing nodes 41, and each computing node 41 contains a plurality of computing units 411. The sequence processing system 40 is used to execute program instructions to implement the steps in any of the above-mentioned MLA-based sequence processing method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the computing unit 411 may include but is not limited to GPU, NPU, CPU, etc., and the specific type of the computing unit 411 is not limited here. In addition, each computing node 41 can adopt a distributed architecture; or, each computing node 41 can also adopt a centralized architecture. The deployment method of each computing node 41 is also not limited here. It should be noted that although Figure 4 Only three computing nodes 41 are schematically shown, but in actual applications, the specific number of computing nodes 41 in the sequence processing system 40 can be set according to actual application needs. For example, when the computing load is relatively light, the specific number of computing nodes 41 in the sequence processing system 40 can be relatively small, such as 2 or 3. Alternatively, when the computing load is relatively heavy, the specific number of computing nodes 41 in the sequence processing system 40 can be relatively large, such as 7 or 8. The specific number of computing nodes 40 in the sequence processing system 40 is not limited here. It is understandable that the specific number of computing units in each computing node 41 can also be set based on the aforementioned computing load and other factors such as parallel efficiency, and will not be further described here.

[0097] In the above scheme, the sequence processing system 40 performs rotation position embedding on the first query feature of low-rank compression after compression to obtain the target query feature of the multi-head attention mechanism, and the first query feature is obtained from the text sequence to be responded, and is segmented based on the first key-value feature of low-rank compression to obtain the first sub-key-value feature to be processed by different computing units in each computing node, and then the different computing units in each computing node respectively perform rotation position embedding on the assigned first sub-key-value feature to obtain the second sub-key-value feature, thereby communicating based on each computing node to obtain the first key-value feature after rotation position embedding. The second key-value feature is obtained, and based on the second key-value feature, the target key feature and target value feature of the multi-head attention mechanism are obtained, and then the multi-head attention mechanism is executed based on the target query feature, target key feature and target value feature to complete the MLA processing. Since the parallel aggregation operation is separated from the multi-head attention MHA, the parallel aggregation operation is placed before the multi-head attention MHA after obtaining the low-rank compressed first key-value feature. At this time, only the low-rank compressed first key-value feature can be communicated. Compared with executing the parallel aggregation operation in the multi-head attention MHA, the communication volume of the computing node can be greatly reduced. Therefore, the communication volume of the computing node can be reduced when performing sequence processing based on MLA to improve the final processing efficiency of the computing node.

[0098] See also Figure 5 , Figure 5 The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps of any of the above-mentioned MLA-based sequence processing method embodiments.

[0099] In the above scheme, the computer-readable storage medium 50 performs rotation position embedding on the first query feature compressed based on low rank compression to obtain the target query feature of the multi-head attention mechanism, and the first query feature is obtained from the text sequence to be responded, and is segmented based on the first key-value feature of low rank compression to obtain the first sub-key-value feature to be processed by different computing units in each computing node, and then the first sub-key-value feature to be processed is rotated and embedded by different computing units in each computing node to obtain the second sub-key-value feature, thereby communicating based on each computing node to obtain the first key-value feature after rotation position embedding. The second key-value feature of the input is obtained, and based on the second key-value feature, the target key feature and target value feature of the multi-head attention mechanism are obtained, and then the multi-head attention mechanism is executed based on the target query feature, target key feature and target value feature to complete the MLA processing. Since the parallel aggregation operation is separated from the multi-head attention MHA, the parallel aggregation operation is placed before the multi-head attention MHA after obtaining the low-rank compressed first key-value feature. At this time, only the low-rank compressed first key-value feature can be communicated. Compared with executing the parallel aggregation operation in the multi-head attention MHA, the communication volume of the computing nodes can be greatly reduced. Therefore, the communication volume of the computing nodes can be reduced when performing sequence processing based on MLA to improve the final processing efficiency of the computing nodes.

[0100] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0101] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0103] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0104] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0106] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A sequence processing method based on MLA, characterized in that: include: The first query feature based on low-rank compression is decompressed and then rotated and embedded to obtain the target query feature of the multi-head attention mechanism; wherein the first query feature is obtained from the text sequence to be responded; Splitting the first key-value feature based on low-rank compression to obtain the first sub-key-value features to be processed by different computing units in each computing node; Different computing units in the computing nodes respectively perform rotation position embedding on the assigned first sub-key value features to obtain second sub-key value features; Communicating with each of the computing nodes to obtain a second key-value feature embedded with the first key-value feature through the rotation position; Based on the second key-value feature, obtain the target key feature and target value feature of the multi-head attention mechanism; Performing a multi-head attention mechanism based on the target query feature, the target key feature, and the target value feature to complete MLA processing; The first key-value feature based on low-rank compression is segmented to obtain the first sub-key-value features to be processed by different computing units in each computing node, including: The first key-value features of the low-rank compression are divided in sequence based on the number of CP groups and the number of SP groups, and the first sub-key-value features to be processed by different computing units in each computing node are obtained; The obtaining of the second key-value feature embedded in the first key-value feature by the rotation position based on the communication between the computing nodes includes: Based on CP group communication between different computing nodes and SP group communication within the same computing node, a second key-value feature embedded with the first key-value feature through the rotation position is obtained.

2. The method according to claim 1, characterized in that The first key-value feature is characterized as a feature tensor whose feature size is the product of the batch size, the sequence length, and the hidden layer size. The low-rank compressed first key-value feature is divided in sequence based on the number of CP groups and the number of SP groups to obtain the first sub-key-value features to be processed by different computing units in each computing node, including: Performing a first segmentation on the first key-value feature in the feature dimension of the sequence length based on the number of CP groups, so as to allocate the first key-value feature to different computing nodes after the final segmentation is completed; Based on the number of SP groups, each sub-feature of the first key-value feature after the first segmentation is split for a second time in the feature dimension of the sequence length to obtain the first sub-key-value features to be processed by different computing units in each computing node.

3. The method according to claim 1, characterized in that The performing CP group communication between different computing nodes includes: Based on the configuration information about the CP group between different computing nodes, using the allgather function to perform CP group communication on the second sub-key value features of different computing nodes; And / or, the performing SP group communication within the same computing node includes: Based on the configuration information about the SP group within the same computing node, an allgather function is used to perform SP group communication on the second sub-key value feature of the same computing node.

4. The method according to claim 1, wherein After successively executing the CP group communication and the SP group communication, each of the computing units in the different computing nodes obtains a third sub-key feature, wherein the third sub-key feature is characterized as a feature tensor whose feature size is a product of a batch size, the number of CP groups, a sequence length divided by the number of SP groups, and a hidden layer size. The method further includes: The second key-value feature is obtained by reshaping the target dimension based on the third sub-key-value feature; wherein the target dimension includes the feature dimension of the number of CP groups and the feature dimension of the sequence length divided by the number of SP groups, and the second key-value feature is represented as a feature tensor whose feature size is the product of the batch size, the sequence length, and the hidden layer size.

5. The method according to claim 1, wherein The step of obtaining a target key feature and a target value feature of the multi-head attention mechanism based on the second key-value feature includes: Based on the feature dimension of the rotation position embedding, a third key feature representing low-rank compression and a first key feature representing the rotation position embedding are split from the second key feature; Decompressing the third key value feature to obtain the second key feature and the target value feature of the multi-head attention mechanism; The target key feature is obtained by fusing the first key feature and the second key feature.

6. The method according to claim 5, characterized in that The step of separating the third key-value feature representing low-rank compression and the first key-value feature representing the embedding of the rotation position from the second key-value feature based on the feature dimension of the rotation position embedding includes: Based on the feature dimension of the rotation position embedding, the feature dimension of the second key-value feature in the hidden layer size is divided to obtain the third key-value feature and the first key feature; wherein the feature dimension of the rotation position embedding and the feature dimension of the first key feature in the hidden layer size are the same.

7. The method according to claim 5, characterized in that The decompression based on the third key value feature to obtain the second key feature and the target value feature of the multi-head attention mechanism includes: Processing the third key-value feature based on the key-value decompression parameter to obtain a decompressed key-value feature; Key value splitting is performed based on the decompressed key value feature to obtain the second key feature and the target value feature.

8. The method according to claim 5, characterized in that The fusing the first key feature and the second key feature to obtain the target key feature includes: The first key feature is copied in the feature dimension of the hidden layer size based on the number of multi-heads of the multi-head attention mechanism to obtain a third key feature, and the feature dimension of the hidden layer size of the second key feature is reshaped into the number of multi-heads multiplied by the target dimension to obtain a fourth key feature; wherein the number of multi-heads multiplied by the target dimension is equal to the feature dimension of the second key feature in the feature dimension of the hidden layer size; The target key feature is obtained by splicing the third key feature and the fourth key feature in the feature dimension of the hidden layer size.

9. The method according to claim 1, characterized in that After performing a multi-head attention mechanism based on the target query feature, the target key feature, and the target value feature to complete MLA processing, the method further includes: Decoding is performed based on the output features after the MLA processing to obtain a response result of the text sequence to be responded to; wherein the response result includes: at least one data type of text, image, video, audio, and table.

10. A sequence processing device based on MLA, characterized in that: include: A position embedding module is configured to perform rotational position embedding on the first query feature after decompression based on low-rank compression to obtain a target query feature of the multi-head attention mechanism; wherein the first query feature is obtained from the text sequence to be responded; A grouping and segmentation module is used to perform segmentation based on the first key-value feature of low-rank compression to obtain the first sub-key-value features to be processed by different computing units in each computing node; a distributed processing module, configured to perform rotation position embedding on the first sub-key value feature assigned for processing by different computing units in each computing node, to obtain a second sub-key value feature; a sequence aggregation module, configured to communicate based on each of the computing nodes to obtain a second key-value feature embedded with the first key-value feature through the rotation position; a feature processing module, configured to obtain a target key feature and a target value feature of a multi-head attention mechanism based on the second key-value feature; A multi-head processing module is configured to execute a multi-head attention mechanism based on the target query feature, the target key feature, and the target value feature to complete MLA processing; wherein the first key-value feature based on low-rank compression is split to obtain the first sub-key-value features to be processed by different computing units in each computing node, including: The first key-value features of the low-rank compression are divided in sequence based on the number of CP groups and the number of SP groups, and the first sub-key-value features to be processed by different computing units in each computing node are obtained; The obtaining of the second key-value feature embedded in the first key-value feature by the rotation position based on the communication between the computing nodes includes: Based on CP group communication between different computing nodes and SP group communication within the same computing node, a second key-value feature embedded with the first key-value feature through the rotation position is obtained.

11. A sequence processing system, characterized in that: The sequence processing system includes a plurality of computing nodes, each of which contains a plurality of computing units. The sequence processing system is used to execute program instructions to implement the MLA-based sequence processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the MLA-based sequence processing method according to any one of claims 1 to 9.

Images