A data processing method, device, apparatus, storage medium, and program product

Abstract

This application provides a data processing method, apparatus, device, storage medium, and program product, relating to the field of artificial intelligence technology. The data processing system includes multiple cards, and the method is applied to each card, including: acquiring input data of the output layer of a target model when processing batch tasks, the input data including semantic information and weight information of the batch tasks; selecting sub-semantic information corresponding to a card from the semantic information and sub-weight information corresponding to a card from the weight information according to the parallel parsing rules of multiple cards; parsing based on the sub-semantic information and sub-weight information to obtain sub-parsing results of the batch tasks, and sending the sub-parsing results to each of the other cards respectively; and determining the output data of the output layer based on the sub-parsing results and the received sub-parsing results corresponding to each of the other cards. This application can improve the computational utilization of the data processing system.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Large Language Models (LLMs) typically consist of an input layer, a hidden layer, and an output layer. The output layer of an LLM is usually a linear layer, often referred to as the lm_head (language model head) layer. Its function is to map the semantic information output by the hidden layer onto logical values ​​of the entire vocabulary size, and then select generated lexical units through sampling operations.

[0003] Currently, the data processing system includes multiple cards (computing cards). For computations at the lm_head layer, a single-card execution strategy is typically employed. Specifically, only one card in the data processing system performs the following operations: acquiring the semantic information (semantic matrix) of the batch tasks output from the hidden layer and the weight information (weight matrix) of the lm_head layer; performing matrix multiplication on the semantic matrix and weight matrix to obtain the target parsing result of the batch tasks, i.e., the raw scores (logits); and sampling the logits to obtain the output data of the output layer, i.e., the sequence of token IDs (Identity Documents).

[0004] Since only one card is responsible for performing the above processing operations, the other cards are idle and waiting during this processing stage, which leads to a decrease in the overall computing utilization of the data processing system and forms a "single card bottleneck". Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, storage medium, and program product to improve the computational utilization of a data processing system.

[0006] On one hand, embodiments of this application provide a data processing method. The data processing system includes multiple cards, and the method is applied to each card, including:

[0007] Obtain the input data of the output layer of the target model when processing batch tasks, wherein the input data includes the semantic information and weight information of the batch tasks;

[0008] According to the parallel parsing rules of the multiple cards, the sub-semantic information corresponding to the card is selected from the semantic information, and the sub-weight information corresponding to the card is selected from the weight information;

[0009] Based on the sub-semantic information and the sub-weight information, the sub-parsing results of the batch tasks are obtained, and the sub-parsing results are sent to each of the other cards respectively;

[0010] Based on the sub-parsing result and the sub-parsing results received from other cards, the output data of the output layer is determined.

[0011] In this embodiment, each card in the data processing system acquires the input data of the output layer of the target model when processing batch tasks. The input data includes semantic information and weight information of the batch tasks. The semantic information of the batch tasks is used to characterize the semantic features of the batch tasks. According to the parallel parsing rules of multiple cards, the sub-semantic information corresponding to the card is selected from the semantic information, and the sub-weight information corresponding to the card is selected from the weight information, thereby reducing the amount of semantic and weight information data that the card needs to process to a certain extent. Furthermore, this application parses based on the sub-semantic information and sub-weight information to obtain the sub-parsing results of the batch tasks. The sub-parsing results corresponding to the card are used to characterize the initial scores of each candidate result of each task in the batch tasks. Since each card determines the sub-parsing results using the corresponding sub-semantic information and sub-weight information, the data is incomplete, and the sub-parsing results cannot accurately characterize the final scores of each candidate result of each task in the batch tasks. Therefore, the sub-parsing results are sent to each of the other cards respectively, and the output data of the output layer is determined based on the sub-parsing results and the received sub-parsing results corresponding to the other cards, thereby ensuring the accuracy of the output data of the output layer. This application eliminates single-card bottlenecks and improves the computational utilization of the data processing system by distributing the parsing tasks across all cards in the data processing system. Furthermore, the performance of the data processing system increases approximately linearly with the increase in the number of cards.

[0012] Optionally, determining the output data of the output layer based on the sub-parsing result and the sub-parsing results received from other cards includes:

[0013] The sub-parsing results and the sub-parsing results corresponding to each other card are fused to obtain the target parsing result of the batch task;

[0014] Based on the target parsing results of the batch tasks, the output data of the output layer is determined.

[0015] In this embodiment, since each card determines its sub-parsing result using corresponding sub-semantic and sub-weight information, the data is incomplete. This means the sub-parsing results cannot accurately represent the final score of each candidate result for each task in the batch task. Therefore, this application fuses the sub-parsing results with those of other cards to obtain the target parsing result for the batch task, thus more accurately representing the final score of each candidate result for each task in the batch task. Furthermore, this application determines the output data of the output layer based on the target parsing result of the batch task, thereby obtaining more accurate output data for the output layer.

[0016] Optionally, determining the output data of the output layer based on the target parsing results of the batch tasks includes:

[0017] Based on the first task allocation rule of the multiple cards, select the target parsing results of a portion of the tasks corresponding to the card from the target parsing results;

[0018] Based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined.

[0019] In this embodiment, according to the first task allocation rule of multiple cards, the target parsing result of the partial task corresponding to the card is selected from the target parsing result to represent the final score of each candidate result of each task in the partial task of the batch task. Based on the target parsing result of the partial task corresponding to the card, the partial output data of the output layer is determined, so that each card processes the target parsing result of its corresponding partial task, improves the utilization of computing resources, eliminates the single card bottleneck, and enables the data processing system to maintain a high load throughout the entire inference stage (including sampling), making full use of hardware computing power. Furthermore, the performance of the data processing system increases approximately linearly with the increase of the number of cards.

[0020] Optionally, sending the sub-resolution results to each of the other cards includes:

[0021] For each other card, according to the second task allocation rule of the multiple cards, the sub-parse result of the partial task corresponding to the other card is selected from the sub-parse result, and the sub-parse result of the partial task corresponding to the other card is sent to the other card.

[0022] Optionally, determining the output data of the output layer based on the sub-parsing result and the sub-parsing results received from other cards includes:

[0023] According to the second task allocation rule of the multiple cards, select the sub-parse result of the partial task corresponding to the card from the sub-parse result;

[0024] The sub-parsing results of the partial tasks corresponding to the card are fused with the sub-parsing results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing result of the partial tasks corresponding to the card.

[0025] Based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined.

[0026] In this embodiment, according to the second task allocation rule of multiple cards, the sub-parse results of the partial tasks corresponding to the card are selected from the sub-parse results to represent the initial scores of each candidate result of each task in the partial tasks of the batch task; the sub-parse results of the partial tasks corresponding to the card are fused with the sub-parse results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing results of the partial tasks corresponding to the card, to represent the final scores of each candidate result of each task in the partial tasks of the batch task; and based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined, so that each card processes the target parsing results of its corresponding partial tasks, improving the utilization of computing resources, eliminating the single-card bottleneck, and enabling the data processing system to maintain a high load throughout the entire inference stage (including sampling), making full use of hardware computing power. The performance of the data processing system increases approximately linearly with the increase of the number of cards.

[0027] On one hand, embodiments of this application provide a data processing apparatus, including:

[0028] The acquisition module is used to acquire the input data of the output layer of the target model when processing batch tasks. The input data includes the semantic information and weight information of the batch tasks.

[0029] The selection module is used to select the sub-semantic information corresponding to the card from the semantic information and select the sub-weight information corresponding to the card from the weight information according to the parallel parsing rules of multiple cards;

[0030] The parsing module is used to parse based on the sub-semantic information and the sub-weight information to obtain the sub-parsing results of the batch task, and send the sub-parsing results to each of the other cards respectively;

[0031] The determining module is used to determine the output data of the output layer based on the sub-parsing result and the sub-parsing results received from other cards.

[0032] Optionally, the determining module is specifically used for:

[0033] The sub-parsing results and the sub-parsing results corresponding to each other card are fused to obtain the target parsing result of the batch task;

[0034] Based on the target parsing results of the batch tasks, the output data of the output layer is determined.

[0035] Optionally, the determining module is specifically used for:

[0036] Based on the first task allocation rule of the multiple cards, select the target parsing results of a portion of the tasks from the target parsing results;

[0037] Based on the target parsing results of some tasks, determine some of the output data of the output layer.

[0038] Optionally, the parsing module is specifically used for:

[0039] For each other card, according to the second task allocation rule of the multiple cards, the sub-parse result of the partial task corresponding to the other card is selected from the sub-parse result, and the sub-parse result of the partial task corresponding to the other card is sent to the other card.

[0040] Optionally, the determining module is specifically used for:

[0041] According to the second task allocation rule of the multiple cards, select the sub-parse result of the partial task corresponding to the card from the sub-parse result;

[0042] The sub-parsing results of the partial tasks corresponding to the card are fused with the sub-parsing results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing result of the partial tasks corresponding to the card.

[0043] Based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined.

[0044] On the one hand, embodiments of this application provide a data processing system, which includes multiple cards, each card executing the steps of the above-described data processing method.

[0045] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip executes the program to implement the steps of the above-described data processing method.

[0046] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described data processing method.

[0047] On one hand, embodiments of this application provide a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described data processing method. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application;

[0050] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0051] Figure 3 A flowchart illustrating a method for determining output data of an output layer, provided in an embodiment of this application;

[0052] Figure 4 A flowchart illustrating a method for selecting sub-semantic information and sub-weight information corresponding to a card, provided in an embodiment of this application;

[0053] Figure 5 A detailed flowchart illustrating the first data processing method provided in this application embodiment;

[0054] Figure 6 A flowchart illustrating a method for determining output data of an output layer, provided in an embodiment of this application;

[0055] Figure 7 This is a flowchart illustrating a method for selecting a portion of the target parsing results corresponding to a card from the target parsing results, as provided in an embodiment of this application.

[0056] Figure 8 A detailed flowchart illustrating the second data processing method provided in this application embodiment;

[0057] Figure 9 A flowchart illustrating another method for determining the output data of the output layer provided in an embodiment of this application;

[0058] Figure 10 A flowchart illustrating a method for selecting a portion of the task corresponding to a card from the sub-parse results, provided in an embodiment of this application;

[0059] Figure 11 A detailed flowchart illustrating the third data processing method provided in this application embodiment;

[0060] Figure 12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0061] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0063] The data collection, dissemination, and use in this application all comply with relevant national laws and regulations.

[0064] The data processing system in this embodiment includes multiple cards. These cards can be referred to as computing cards or artificial intelligence chips. A card is a processor specifically designed for hardware acceleration of artificial intelligence algorithms (especially machine learning / deep learning). Cards can be: Graphics Processing Units (GPUs), General-purpose computing on graphics processing units (GPGPUs), Tensor Processing Units (TPUs), Neural Processing Units (NPUs), Domain Specific Architectures (DSAs), etc.

[0065] Before introducing a data processing method provided by the embodiments of this application, the technical background of the embodiments of this application will be described in detail below for ease of understanding.

[0066] LLM typically consists of an input layer, a hidden layer, and an output layer. The output layer of an LLM is usually a linear layer, often called the lm_head layer. Its function is to map the semantic information output by the hidden layer onto logical values ​​of the entire vocabulary size, and then select the generated lexical units through sampling operations.

[0067] Currently, data processing systems consist of multiple cards (computing cards), and a single-card execution strategy is typically used for computation at the lm_head layer. Specifically, only one card in the data processing system performs the following operations: acquiring the semantic information (semantic matrix) of the batch tasks output from the hidden layer and the weight information (weight matrix) of the lm_head layer; performing matrix multiplication on the semantic matrix and weight matrix to obtain the target parsing result of the batch tasks, i.e., logits; and sampling the logits to obtain the output data of the output layer, i.e., the token ID sequence. Since only one card is responsible for performing the above processing operations, the other cards are idle and waiting during this processing stage, resulting in a decrease in the overall computational utilization of the data processing system, forming a "single-card bottleneck".

[0068] To perform parallel computation on the lm_head layer, the industry commonly adopts a model parallelism strategy, which involves splitting the weight matrix along the first dimension (vocabulary dimension). Specifically, assuming the shape of the lm_head weight matrix is ​​[K, N], the data processing system includes p identical cards, and the shape of the semantic matrix output by the hidden layer is [Batch_size, K], where K is the dimension of the hidden layer, N is the vocabulary size, Batch_size is the batch size, and p is an integer greater than 1. Two common data processing schemes are as follows:

[0069] The first approach involves partitioning the weight matrix along the vocabulary dimension (the first dimension) based on the number of cards in the data processing system, resulting in a sub-weight matrix for each card. The shape of each sub-weight matrix is ​​[K, N / p]. For each card, matrix multiplication is performed on the semantic matrix output from the hidden layer and the corresponding sub-weight matrix to determine the partial logits for that card. The shape of each partial logits is [Batch_size, N / p]. Next, each card in the data processing system performs an AllGather communication operation to collect the partial logits from all other cards. For each card, based on its partial logits and the collected partial logits from other cards, the complete logits are determined, with a shape of [Batch_size, N]. Finally, all cards independently and repeatedly perform the same sampling operation on the complete logits, generating the same lexical ID sequence for use in the next round of inference. The shape of the lexical ID sequence for each card is [Batch_size, 1].

[0070] Since the amount of communication data in the AllGather communication operation is proportional to the Batch_size Given that the vocabulary size N is typically enormous (tens of thousands to hundreds of thousands), this communication becomes a severe performance bottleneck and incurs significant communication overhead when processing large batches of tasks. Furthermore, all cards in the data processing system perform identical sampling calculations on the same complete set of logits, resulting in a significant waste of computational power. While sampling operations are lighter than forward computation, their cumulative overhead is not negligible in large-scale task processing scenarios.

[0071] The second approach, based on the number of cards in the data processing system, divides the weight matrix according to the first dimension (vocabulary dimension) to obtain a sub-weight matrix corresponding to each card, where the shape of each card's sub-weight matrix is ​​[K, N / p]. For each card, matrix multiplication is performed on the semantic matrix output from the hidden layer and the sub-weight matrix corresponding to that card to determine the partial logits, the shape of which is [Batch_size, N / p]. These partial logits are aggregated to a designated master card via point-to-point communication to obtain complete logits, the shape of which is [Batch_size, N]. The point-to-point communication method can be P2P (ponit-to-ponit). The master card samples these logits to obtain a token ID sequence, the shape of which is [Batch_size, 1]. Then, the master card distributes the token ID sequence to each other card via broadcast communication for use in the next round of inference.

[0072] Because only one master card is responsible for performing the sampling operation, the other cards are idle and waiting during the sampling phase, resulting in a decrease in the overall computing utilization of the data processing system and forming a "single-card bottleneck." Furthermore, as the number of cards increases, this mode of having a single card responsible for critical sampling tasks cannot effectively utilize the additional computing resources, and the performance of the data processing system cannot be improved linearly, which is particularly unreasonable when processing large batches of tasks.

[0073] To address the aforementioned problems, this application proposes a data processing method, apparatus, device, storage medium, and program product to improve the computational utilization of a data processing system.

[0074] First, the specific flow of the data processing method provided in the embodiments of this application can be executed by an artificial intelligence chip. (See reference...) Figure 1 This is a structural diagram of an artificial intelligence chip applicable to the embodiments of this application. The artificial intelligence chip 100 includes at least: video memory 101, on-chip cache 102 and multiple computing units 103.

[0075] Video memory 101 can be high-bandwidth memory (HBM) or other types of memory. On-chip cache 102 is a temporary memory with a smaller capacity than video memory 101, but faster data read and write speeds.

[0076] The computing unit 103 can be used to execute various types of operators, including but not limited to: memory transfer operators, convolution operators, attention mechanism operators, and matrix multiplication operators.

[0077] Different computing units 103 can exchange data through video memory 101 or through on-chip cache 102.

[0078] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.

[0079] The AI ​​chip 100 can be: a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain-specific architecture (DSA), etc.

[0080] The following is for reference. Figure 2 The flowchart shown illustrates a data processing method, which clarifies the technical solution provided in the embodiments of this application.

[0081] Step 201: Obtain the input data of the output layer of the target model when processing batch tasks.

[0082] The batch task can be a sequence reasoning task, text classification task, sentiment analysis task, document task, dialogue generation task, translation task, image classification task, object detection task, object tracking task, etc. The input data includes semantic information and weight information for the batch task. Semantic information can be obtained by extracting features from the hidden layers (feature extraction layers) of the target model. The batch task includes the data to be processed in the current batch. Semantic information is used to represent the semantic features of the batch task, and these features can be represented as a semantic matrix. The semantic matrix has the shape [Batch_size, K], where Batch_size is the batch size (i.e., the number of tasks), and K is the dimension of the hidden layer. In other words, the target model uses a vector containing K components (features) to represent all the semantic, syntactic, and contextual information of each task.

[0083] In this embodiment, when the batch task is a sequence reasoning task, the final output data needs to be obtained through calculations at multiple time steps. The calculation process for the j-th time step is as follows: The data to be processed in the current batch and the output data of the previous j-1 time steps are input into the hidden layer of the target model. Feature extraction is performed using the hidden layer of the target model to obtain semantic information, where j is an integer greater than or equal to 1. The calculation process for the 0th time step is as follows: The data to be processed in the current batch is input into the hidden layer of the target model. Feature extraction is performed using the hidden layer of the target model to obtain semantic information.

[0084] Since target models typically consist of various layers, each corresponding to a weight information, the weight information in this application refers to the weight information corresponding to the output layer of the target model. The weight information can be represented as a weight matrix. The output layer of the target model is called the lm_head layer in LLM, which is the final output layer responsible for converting the model's internal representation into human-readable vocabulary (usually a probability distribution). The weight matrix has a shape of [K, N], where K is the dimension of the hidden layer, and N is the vocabulary size. The vocabulary is the set of all unique tokens used by the model, establishing a bidirectional mapping between tokens and their integer IDs. In LLM, a token typically refers to a discrete text unit, which can be a word, punctuation mark, number, or other linguistic element. These elements are used as the basic units for training and generating text.

[0085] In this embodiment, the target model can be a large language model or other models with a large parameter scale that require distributed reasoning, such as the Transformer (a deep learning model based on an attention mechanism). The model can be specifically used to implement sequence reasoning, such as text sequence reasoning, image sequence reasoning, and speech sequence reasoning; this embodiment does not specifically limit its application in this regard. Furthermore, the target model can be a model used for tasks such as text classification and sentiment analysis, or a model used for question answering systems and dialogue generation, or a model applied to computer vision tasks such as image classification, object detection, and object tracking; this embodiment does not specifically limit its application in this regard either.

[0086] For different types of target models, the types of data to be processed in batch tasks vary, and the meaning of the semantic information in batch tasks also differs. For example, for text-based models, such as large language models, the data to be processed in batch tasks can be text, and the semantic information of batch tasks is used to represent the semantic features of the text. Similarly, for speech-based models, such as speech recognition models and speech translation models, the data to be processed in batch tasks can be speech, and the semantic information of batch tasks is used to represent the semantic features of the speech. Furthermore, for image-based models, such as object detection models and image classification models, the data to be processed in batch tasks can be images, and the semantic information of batch tasks is used to represent the semantic features of the images. Finally, for video-based models, such as object tracking models and behavior recognition models, the data to be processed in batch tasks can be video, and the semantic information of batch tasks is used to represent the semantic features of the video. This application does not impose specific limitations on these aspects.

[0087] For example, if the batch task is a translation task, the data to be translated within a set time period is obtained. For example, if there are 100 data to be translated within the set time period, and the hidden layer dimension of the target model is 7168, then the 100 data to be translated are input into the hidden layer of the target model, and the hidden layer of the target model is used to extract features from the 100 data to be translated to obtain the semantic information of the batch task. The shape of the semantic information is [100, 7168].

[0088] If the batch task is a question-and-answer task, obtain the question data within a set time period. For example, if there are 200 questions within the set time period and the hidden layer dimension of the target model is 7168, then input the 200 questions into the hidden layer of the target model, use the hidden layer of the target model to extract features from the 200 questions, and obtain the semantic information of the batch task. The shape of the semantic information is [200, 7168].

[0089] Step 202: Based on the parallel parsing rules of multiple cards, select the sub-semantic information corresponding to the card from the semantic information, and select the sub-weight information corresponding to the card from the weight information.

[0090] The parallel parsing rule for multiple cards involves dividing semantic information into multiple sub-semantic information and weight information into multiple sub-weight information. This can be set according to the actual situation. The number of sub-semantic information and the number of sub-weight information are the same as the number of cards in the data processing system. Each card corresponds to one sub-semantic information and one sub-weight information.

[0091] Step 203: Based on the sub-semantic information and sub-weight information, perform parsing to obtain the sub-parsing results of the batch task, and send the sub-parsing results to each of the other cards respectively.

[0092] The data processing system comprises multiple cards, and the aforementioned data processing methods are applied to each card. For any given card, "other cards" refers to every card in the data processing system other than that given card. For example, the data processing system includes three cards: Card 1, Card 2, and Card 3, which execute in parallel. For Card 1, Card 2 and Card 3 are considered "other cards." For Card 2, Card 1 and Card 3 are considered "other cards." For Card 3, Card 1 and Card 2 are considered "other cards."

[0093] In this embodiment, the sub-parsing results of the batch tasks are obtained based on sub-semantic information and sub-weight information. This includes performing tensor multiplication operations on the sub-semantic information and sub-weight information to obtain the sub-parsing results of the batch tasks. The sub-parsing results of the batch tasks are used to characterize the preliminary scores (probabilities) of each candidate result for each task in the batch tasks. The sub-parsing results are sub-logits, where logits are the raw unnormalized scores of the deep learning model's output layer, representing the model's preference for each category.

[0094] In this embodiment of the application, the sub-resolution result corresponding to the i-th card can be determined using the following formula. : ;in, For the sub-semantic information corresponding to the i-th card, Let i be the sub-weight information corresponding to the i-th card, where i is greater than 0 and less than or equal to p, where p is the number of cards in the data processing system, and p is an integer greater than 1.

[0095] In this embodiment of the application, after parsing based on sub-semantic information and sub-weight information to obtain the sub-parsing results of the batch tasks, the sub-parsing results are sent to each of the other cards respectively, and the corresponding sub-parsing results sent by each of the other cards are received.

[0096] For example, a data processing system includes three cards: Card 1, Card 2, and Card 3, which execute in parallel. Card 1 parses the data based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 1 for the batch task. Card 2 parses the data based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 2 for the batch task. Card 3 parses the data based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 3 for the batch task. Card 1 sends sub-parsing result 1 to Card 2 and Card 3 respectively, and receives sub-parsing result 2 from Card 2 and sub-parsing result 3 from Card 3. Card 2 sends sub-parsing result 2 to Card 1 and Card 3 respectively, and receives sub-parsing result 1 from Card 1 and sub-parsing result 3 from Card 3. Card 3 sends sub-parsing result 3 to Card 1 and Card 2 respectively, and receives sub-parsing result 1 from Card 1 and sub-parsing result 2 from Card 2.

[0097] Step 204: Determine the output data of the output layer based on the sub-parsing results and the sub-parsing results received from other cards.

[0098] The output layer's output data is a sequence of token IDs, representing the processing result of each task in the batch task. The shape of the output data is [Batch_size, 1], where Batch_size is the batch size. For example, if the batch task is a question-answering task, an output data shape of [100, 1] represents the answers to the current 100 questions.

[0099] In this embodiment of the application, after parsing based on sub-semantic information and sub-weight information to obtain the sub-parsing results of the batch tasks, the method further includes: collecting the sub-parsing results corresponding to each other card, determining the output data of the output layer based on the sub-parsing results and the received sub-parsing results corresponding to each other card, and sending the output data to each other card respectively, so that all cards in the data processing system hold the same complete output data.

[0100] For example, a data processing system includes three cards: Card 1, Card 2, and Card 3, which execute in parallel. Card 1 parses the data based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 1 for the batch task. Card 2 parses the data based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 2 for the batch task. Card 3 parses the data based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 3 for the batch task. Card 1 collects sub-parsing result 2 from Card 2 and sub-parsing result 3 from Card 3, determines the output data for the output layer based on these results, and sends the output data to Cards 2 and 3 respectively. Card 2 sends sub-parsing result 2 to Card 1 and receives the output data from Card 1. Card 3 sends sub-parsing result 3 to Card 1 and receives the output data from Card 1.

[0101] In this embodiment of the application, after parsing based on sub-semantic information and sub-weight information to obtain the sub-parsing results of batch tasks, the method further includes: collecting the sub-parsing results corresponding to each other card, and sending the sub-parsing results corresponding to each other card to each other card, so that each card can determine the output data of the output layer according to the sub-parsing results corresponding to each card, thereby ensuring that all cards in the data processing system hold the same and complete output data.

[0102] In this embodiment, each card in the data processing system can execute the AllGather communication operation to collect the sub-resolution results corresponding to other cards, thereby enabling each card in the data processing system to possess the sub-resolution results corresponding to each other card. The AllGather communication operation is a many-to-many communication primitive with multiple data senders and multiple data receivers. It can collect data from multiple nodes within the cluster onto a master node, and then distribute this collected data to other nodes, effectively collecting all data within the cluster onto all nodes.

[0103] For example, a data processing system includes three cards: Card 1, Card 2, and Card 3, which execute in parallel. Card 1 parses based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 1 for the batch task. Card 2 parses based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 2 for the batch task. Card 3 parses based on its corresponding sub-semantic information and sub-weight information to obtain sub-parsing result 3 for the batch task. Card 1 collects sub-parsing result 2 from Card 2 and sub-parsing result 3 from Card 3, and determines the output data of the output layer based on sub-parsing result 1, sub-parsing result 2, and sub-parsing result 3, respectively, and sends these results to Card 2 and Card 3. Card 2 sends sub-parsing result 2 to Card 1 and receives sub-parsing result 1, sub-parsing result 2, and sub-parsing result 3 from Card 1, and determines the output data of the output layer based on these results. Card 3 sends sub-parse result 3 to Card 1 and receives sub-parse result 1, sub-parse result 2 and sub-parse result 3 from Card 1. Based on sub-parse result 1, sub-parse result 2 and sub-parse result 3, it determines the output data of the output layer.

[0104] In this embodiment of the application, after determining the output data of the output layer based on the sub-parsing results and the sub-parsing results corresponding to each of the other cards received, all cards in the data processing system hold the same complete output data, and can consistently continue to perform the embedding lookup and forward calculation in the next time step, or complete batch tasks.

[0105] In this embodiment of the application, the output data of the output layer is determined based on the sub-parsing result and the sub-parsing results corresponding to the other cards received. Figure 3 A flowchart illustrating a method for determining output data of an output layer, as provided in an embodiment of this application, is shown below. Figure 3 As shown, step 204 above includes at least the following steps 301-302:

[0106] Step 301: Merge the sub-parse results with the sub-parse results corresponding to each other card to obtain the target parsing results for the batch task.

[0107] In this embodiment of the application, the sub-parsing results and the sub-parsing results corresponding to each other card are fused to obtain the target parsing result of the batch task, including: performing tensor summation operation on the sub-parsing results and the sub-parsing results corresponding to each other card to obtain the target parsing result of the batch task.

[0108] In this embodiment of the application, the target parsing result of the batch task is determined by the following formula. :

[0109] ;

[0110] in, This is the sub-parse result corresponding to the i-th card, where i is greater than 0 and less than or equal to p, where p is the number of cards in the data processing system, and p is an integer greater than 1.

[0111] For example, a data processing system includes four cards: card 1, card 2, card 3, and card 4. If the sub-parse result corresponding to card 1 is... The sub-parse result corresponding to card 2 is The sub-parse result corresponding to card 3 is The sub-parse result corresponding to card 4 is The target parsing result of the batch task .

[0112] Step 302: Determine the output data of the output layer based on the target parsing results of the batch tasks.

[0113] Specifically, the objective resolution result is used to characterize the final score of each candidate result for each task in the batch task. For example, if the batch task is a question-answering task, the objective resolution result is used to characterize the final score of each candidate answer for each question in the batch of questions; if the batch task is a translation task, the objective resolution result is used to characterize the final score of each candidate word for each piece of data to be translated in the batch of data to be translated.

[0114] In this embodiment of the application, determining the output data of the output layer based on the target parsing results of the batch tasks includes: sampling the target parsing results of the batch tasks to determine the output data of the output layer. Specifically, sampling the target parsing results of the batch tasks to determine the output data of the output layer includes: sampling the target parsing results of the batch tasks using a set sampling strategy to determine the output data of the output layer.

[0115] The sampling strategy can be set according to the actual situation. For example, sampling strategies include Top-K sampling, Top-P sampling, temperature sampling, and joint sampling. Top-K sampling is an optimization of greedy sampling (selecting the token with the highest score from all tokens). Its parameter K represents the number of tokens to be selected. K can be set by the user and is usually a positive integer. When K=1, Top-K sampling is greedy sampling. Specifically, during sampling, the top K tokens are selected from the descending order of token scores, and then a random token is selected from these K tokens as the output. Top-P sampling selects tokens from the descending order of token scores based on a percentage. Its parameter P is equivalent to a cumulative probability threshold, where P is greater than or equal to 0 and less than or equal to 1. P is usually set to a relatively high value. In specific sampling, starting with the token with the highest probability in the descending probability sorting results, the cumulative probability of the first L tokens is calculated. Then, a token is randomly selected as the output from the smallest set whose cumulative probability is greater than or equal to probability P (i.e., the set including the first L tokens when their cumulative probability first exceeds or equals P), where L is a natural number. Temperature sampling controls the randomness of the generated text by adjusting the temperature parameter of the probability distribution. Higher temperature values ​​increase the sampling chance of low-probability tokens. Joint sampling, in practical applications, Top-K sampling, Top-P sampling, and temperature sampling are often used together to achieve a more balanced text generation effect.

[0116] In this embodiment, since each card determines its sub-parsing result using corresponding sub-semantic and sub-weight information, the data is incomplete. This means the sub-parsing results cannot accurately represent the final score of each candidate result for each task in the batch task. Therefore, this application fuses the sub-parsing results with those of other cards to obtain the target parsing result for the batch task, thus more accurately representing the final score of each candidate result for each task in the batch task. Furthermore, this application determines the output data of the output layer based on the target parsing result of the batch task, thereby obtaining more accurate output data for the output layer.

[0117] In this embodiment of the application, according to the parallel parsing rules of multiple cards, the sub-semantic information corresponding to the card is selected from the semantic information, and the sub-weight information corresponding to the card is selected from the weight information. Figure 4 A flowchart illustrating a method for selecting sub-semantic information and sub-weight information corresponding to a card, as provided in this application embodiment, is shown below. Figure 4 As shown, step 202 above includes at least the following steps 401-404:

[0118] Step 401: Based on the computing performance data of each card, the semantic information is divided to obtain a sub-semantic information sequence.

[0119] The computing performance data for each card includes the number of computing units, the average computing frequency per computing unit, memory bandwidth, and the average computing power per computing unit. In this embodiment, the computing performance data for each card is preprocessed to obtain standardized computing performance data.

[0120] In this embodiment, the sub-semantic information sequence includes multiple sub-semantic information items and a card identifier corresponding to each sub-semantic information item. The number of sub-semantic information items in the sequence is the same as the number of cards in the data processing system. Each sub-semantic information item corresponds to a card identifier, and the card identifiers corresponding to each sub-semantic information item are different. The card identifier corresponding to each sub-semantic information item is a preset identifier that can uniquely represent a certain card in the data processing system. In this embodiment, a corresponding card identifier can be set for each card in the data processing system according to the actual situation, and the card identifiers corresponding to each card are different.

[0121] In this embodiment of the application, the data volume of the sub-semantic information corresponding to each card is determined by the following method: calculating the first sum of the computing performance data of each card; and determining the data volume of the sub-semantic information of the card based on the ratio of the computing performance data of the card to the first sum and the data volume of the semantic information.

[0122] In this embodiment of the application, semantic information is divided based on the computing performance data of each card to obtain a sub-semantic information sequence, including: based on the computing performance data of each card, semantic information is divided according to the hidden layer dimension (first dimension) to obtain a sub-semantic information sequence.

[0123] In this embodiment, if the computational performance data of each card is the same, the semantic information is divided equally according to the hidden layer dimension based on the computational performance data of each card to obtain a sub-semantic information sequence. Each sub-semantic information in the sub-semantic information sequence has the same shape.

[0124] For example, a data processing system includes p cards, and the shape of the semantic information is [Batch_size, K], where Batch_size is the batch size, K is the dimension value of the hidden layer, and p is an integer greater than 1. If the computational performance data of the p cards are the same, then based on the computational performance data of each card, the semantic information is divided equally according to the hidden layer dimension to obtain a sub-semantic information sequence. The shape of each sub-semantic information in the sub-semantic information sequence is [Batch_size, K / p].

[0125] In this embodiment, if at least two cards have different computational performance data, the semantic information is divided according to the hidden layer dimension based on the computational performance data of each card to obtain a sub-semantic information sequence. The sub-semantic information sequence may have the following two cases: First, all sub-semantic information in the sub-semantic information sequence has a different shape. Second, some sub-semantic information in the sub-semantic information sequence has a different shape.

[0126] Step 402: Based on the card identifier, select the sub-semantic information corresponding to the card from the sub-semantic information sequence.

[0127] In this embodiment of the application, selecting the sub-semantic information corresponding to the card from the sub-semantic information sequence based on the card identifier includes: matching the card identifier of the card with the card identifier corresponding to each sub-semantic information in the sub-semantic information sequence, and taking the sub-semantic information corresponding to the matched card identifier as the sub-semantic information corresponding to the card.

[0128] For example, the sub-semantic information sequence includes three sub-semantic information, namely sub-semantic information 1, sub-semantic information 2 and sub-semantic information 3. The card identifier corresponding to sub-semantic information 1 is K001, the card identifier corresponding to sub-semantic information 2 is K002, and the card identifier corresponding to sub-semantic information 3 is K003. If the card identifier is K002, then the card identifier of the card is matched with the card identifier corresponding to each sub-semantic information in the sub-semantic information sequence. It is determined that the card identifier corresponding to sub-semantic information 2 is the same as the card identifier of the card. Therefore, sub-semantic information 2 is taken as the sub-semantic information corresponding to the card.

[0129] Step 403: Divide the weight information based on the computing performance data of each card to obtain a sub-weight information sequence.

[0130] The sub-weight information sequence includes multiple sub-weight information items and a corresponding card identifier for each sub-weight information item. The number of sub-weight information items in the sub-weight information sequence is the same as the number of cards in the data processing system. Each sub-weight information item corresponds to a card identifier, and the card identifiers corresponding to each sub-weight information item are different. The card identifier corresponding to each sub-weight information item is a preset identifier that can uniquely identify a certain card in the data processing system.

[0131] In this embodiment of the application, the data volume of the sub-weight information corresponding to each card is determined by the following method: calculating the second sum of the computing performance data of each card; and determining the data volume of the sub-weight information corresponding to the card based on the ratio of the computing performance data of the card to the second sum and the data volume of the weight information.

[0132] In this embodiment of the application, the weight information is divided based on the computing performance data of each card to obtain a sub-weight information sequence, including: dividing the weight information according to the hidden layer dimension (the 0th dimension) based on the computing performance data of each card to obtain a sub-weight information sequence.

[0133] In this embodiment, if the computational performance data of each card is the same, the weight information is divided equally according to the hidden layer dimension based on the computational performance data of each card to obtain a sub-weight information sequence. Each sub-weight information in the sub-weight information sequence has the same shape.

[0134] For example, a data processing system includes p cards, with semantic information of shape [K, N], where K is the dimension value of the hidden layer, N is the vocabulary size, and p is an integer greater than 1. If the computational performance data of the p cards are the same, then based on the computational performance data of each card, the weight information is divided equally according to the hidden layer dimension to obtain a sub-weight information sequence. The shape of each sub-weight information in the sub-weight information sequence is [K / p, N].

[0135] In this embodiment, if at least two cards have different computational performance data, the weight information is divided according to the hidden layer dimension based on the computational performance data of each card to obtain a sub-weight information sequence. The sub-weight information sequence may have the following two cases: First, all sub-weight information in the sub-weight information sequence has a different shape. Second, some sub-weight information in the sub-weight information sequence has a different shape.

[0136] Step 404: Based on the card identifier, select the sub-weight information corresponding to the card from the sub-weight information sequence.

[0137] In this embodiment of the application, selecting the sub-weight information corresponding to the card from the sub-weight information sequence based on the card identifier includes: matching the card identifier of the card with the card identifier corresponding to each sub-weight information in the sub-weight information sequence, and taking the sub-weight information corresponding to the matched card identifier as the sub-weight information corresponding to the card.

[0138] For example, the sub-weight information sequence includes three sub-weight information, namely sub-weight information 1, sub-weight information 2 and sub-weight information 3. The card identifier corresponding to sub-weight information 1 is K001, the card identifier corresponding to sub-weight information 2 is K002, and the card identifier corresponding to sub-weight information 3 is K003. If the card identifier of the card is K002, then the card identifier of the card is matched with the card identifier corresponding to each sub-weight information in the sub-weight information sequence. It is determined that the card identifier corresponding to sub-weight information 2 is the same as the card identifier of the card. Therefore, sub-weight information 2 is taken as the sub-weight information corresponding to the card.

[0139] In this embodiment of the application, a first data processing method is provided. In the first data processing method, each card in the data processing system selects the sub-semantic information corresponding to the card from the semantic information and selects the sub-weight information corresponding to the card from the weight information. Based on the sub-semantic information and sub-weight information, the sub-parsing results of the batch task are obtained, and the sub-parsing results are sent to each other card respectively. The sub-parsing results and the sub-parsing results corresponding to each other card are fused to obtain the target parsing result of the batch task. Based on the target parsing result of the batch task, the output data of the output layer is determined. Figure 5 A detailed flowchart illustrating the first data processing method provided in this application embodiment is shown below. Figure 5 As shown, it includes at least the following steps 501-505:

[0140] Step 501: Obtain the input data of the output layer of the target model when processing batch tasks.

[0141] Step 502: Based on the parallel parsing rules of multiple cards, select the sub-semantic information corresponding to the card from the semantic information, and select the sub-weight information corresponding to the card from the weight information.

[0142] Step 503: Based on the sub-semantic information and sub-weight information, perform parsing to obtain the sub-parsing results of the batch task, and send the sub-parsing results to each of the other cards respectively.

[0143] Step 504: Merge the sub-parsing results with the sub-parsing results corresponding to each other card to obtain the target parsing results for the batch task.

[0144] Step 505: Determine the output data of the output layer based on the target parsing results of the batch tasks.

[0145] For example, when the target model processes batch tasks, the shape of the input data to the output layer is [Batch_size, K]. The data processing system includes p identical cards. The shape of the weight information corresponding to the output layer of the target model is [K, N], where K is the dimension of the hidden layer, N is the vocabulary size, Batch_size is the batch size, and p is an integer greater than 1. For the i-th card, according to the parallel parsing rules of multiple cards, the sub-semantic information corresponding to the i-th card is selected from the semantic information, and the sub-weight information corresponding to the i-th card is selected from the weight information. Let i be an integer, where the shape of the sub-semantic information corresponding to the i-th card is [Batch_size, K / p], and the shape of the sub-weight information corresponding to the i-th card is [K / p, N]. Based on the sub-semantic information and sub-weight information corresponding to the i-th card, the sub-parse result of the batch task corresponding to the i-th card is obtained. And send the sub-resolution results to each of the other cards respectively. Sub-parse results The shape is [Batch_size, N]; the sub-parse results and the sub-parse results corresponding to the other cards are fused to obtain the target parsing result of the batch task. Target analysis results The shape is [Batch_size, N]; based on the target parsing results of the batch tasks. Determine the output data of the output layer. The output data has a shape of [Batch_size, 1].

[0146] In this embodiment of the application, the output data of the output layer is determined based on the target parsing results of the batch tasks. Figure 6 A flowchart illustrating a method for determining output data of an output layer, as provided in an embodiment of this application, is shown below. Figure 6 As shown, it includes at least the following steps 601-602:

[0147] Step 601: Based on the first task allocation rules of multiple cards, select the target parsing results of the partial tasks corresponding to the card from the target parsing results.

[0148] Among them, the first task allocation rule for multiple cards is the setting rule for dividing the target parsing result into multiple partial tasks of the target parsing result. It can be set according to the actual situation, with each card corresponding to the target parsing result of one partial task.

[0149] Step 602: Determine the partial output data of the output layer based on the target parsing results of the partial tasks corresponding to the card.

[0150] In this embodiment, the partial output data of the output layer is determined based on the target parsing results of the partial tasks corresponding to the card. This includes sampling the target parsing results of the partial tasks corresponding to the card to determine the partial output data of the output layer. Since the target parsing results of the partial tasks corresponding to the card represent the final scores of each candidate result of each task in the partial tasks of the batch task, i.e., the complete logits of the partial batch, the correct execution of the sampling operation can be guaranteed.

[0151] In this embodiment, determining partial output data of the output layer based on the target parsing results of a partial task corresponding to the card includes: sampling the target parsing results of the partial task corresponding to the card using a set sampling strategy to determine the partial output data of the output layer. The set sampling strategy is the same for each card.

[0152] In this embodiment, according to the first task allocation rule of multiple cards, the target parsing result of the partial task corresponding to the card is selected from the target parsing result to represent the final score of each candidate result of each task in the partial task of the batch task. Based on the target parsing result of the partial task corresponding to the card, the partial output data of the output layer is determined, so that each card processes the target parsing result of its corresponding partial task, improves the utilization of computing resources, eliminates the single card bottleneck, and enables the data processing system to maintain a high load throughout the entire inference stage (including sampling), making full use of hardware computing power. The performance of the data processing system increases approximately linearly with the increase of the number of cards.

[0153] In this embodiment of the application, the target parsing results of a portion of the tasks corresponding to a card are selected from the target parsing results according to the first task allocation rule of multiple cards. Figure 7 This is a flowchart illustrating a method for selecting a portion of the target parsing results corresponding to a card from the target parsing results, as provided in an embodiment of this application. Figure 7 As shown, it includes at least the following steps 701-702:

[0154] Step 701: Divide the target parsing results based on the computing performance data of each card to obtain a sequence of parsing results.

[0155] The parsing result sequence includes the target parsing results of multiple sub-tasks and the card identifier corresponding to the target parsing result of each sub-task. The number of target parsing results for each sub-task in the parsing result sequence is the same as the number of cards in the data processing system. Each target parsing result for each sub-task corresponds to a card identifier, and the card identifiers corresponding to the target parsing results of each sub-task are different. The card identifier corresponding to the target parsing result of each sub-task is a preset identifier that can uniquely identify a certain card in the data processing system. The identifier can be a string.

[0156] In this embodiment of the application, the task quantity of the target parsing result of the partial task corresponding to each card is determined by the following method: calculating the third sum of the computing performance data of each card; and determining the task quantity of the target parsing result of the partial task corresponding to the card based on the ratio of the computing performance data of the card to the third sum and the task quantity of the target parsing result.

[0157] In this embodiment of the application, the target parsing results are divided based on the computing performance data of each card to obtain a parsing result sequence, including: dividing the target parsing results according to the batch dimension based on the computing performance data of each card to obtain a parsing result sequence.

[0158] In this embodiment, if the computing performance data of each card is the same, the target parsing results are averaged according to the batch dimension based on the computing performance data of each card to obtain a parsing result sequence. The target parsing results of each task in the parsing result sequence have the same shape.

[0159] For example, a data processing system includes p cards, and the shape of the target parsing result is [Batch_size, N], where Batch_size is the batch size, N is the vocabulary size, and p is an integer greater than 1. If the computational performance data of the p cards are the same, the target parsing result is divided equally according to the batch dimension based on the computational performance data of each card, resulting in a parsing result sequence. The shape of the target parsing result for each task in the parsing result sequence is [Batch_size / p, N].

[0160] In this embodiment, if at least two cards have different computational performance data, the target parsing results are divided according to the batch dimension based on the computational performance data of each card to obtain a parsing result sequence. The parsing result sequence may have the following two scenarios: First, the shapes of the target parsing results for each task in the parsing result sequence are all different. Second, the shapes of the target parsing results for some tasks in the parsing result sequence are different.

[0161] Step 702: Based on the card identifier, select the target parsing result of the partial task corresponding to the card from the parsing result sequence.

[0162] In this embodiment of the application, selecting the target parsing result of a partial task corresponding to a card from the parsing result sequence based on the card identifier includes: matching the card identifier of the card with the card identifier corresponding to the target parsing result of each partial task in the parsing result sequence, and taking the target parsing result of the partial task corresponding to the matched card identifier as the target parsing result of the partial task corresponding to the card.

[0163] For example, the parsing result sequence includes the target parsing results of three partial tasks: the target parsing results of partial task 1, partial task 2, and partial task 3. The card identifier corresponding to the target parsing result of partial task 1 is K001, the card identifier corresponding to the target parsing result of partial task 2 is K002, and the card identifier corresponding to the target parsing result of partial task 3 is K003. If the card identifier of the card is K002, then the card identifier of the card is matched with the card identifier corresponding to the target parsing results of each partial task in the parsing result sequence. It is determined that the card identifier corresponding to the target parsing result of partial task 2 is K002, which is the same as the card identifier of the card. Therefore, the target parsing result of partial task 2 is taken as the target parsing result of the partial task corresponding to the card.

[0164] In this embodiment of the application, after determining the partial output data of the output layer based on the target parsing result of the partial task corresponding to the card in step 702, the partial output data is sent to each of the other cards respectively, and the output data of the output layer is determined based on the partial output data corresponding to the card and the partial output data corresponding to each of the other cards received.

[0165] In this embodiment of the application, the output data of the output layer is determined based on the partial output data corresponding to the card and the partial output data corresponding to each other card received, including: performing tensor concatenation on the partial output data corresponding to the card and the partial output data corresponding to each other card received according to the batch dimension to obtain the output data of the output layer.

[0166] For example, a data processing system includes three cards: Card 1, Card 2, and Card 3. The three cards execute in parallel. Each card determines partial output data for the output layer based on the target parsing result of its corresponding partial task. Card 1 corresponds to partial output data 1, Card 2 to partial output data 2, and Card 3 to partial output data 3. Card 1 sends partial output data 1 to Card 2 and Card 3 respectively, and receives partial output data 2 from Card 2 and partial output data 3 from Card 3. It then performs tensor concatenation on partial output data 1, partial output data 2, and partial output data 3 according to the batch dimension (dimension 0) to obtain the output data. Card 2 sends partial output data 2 to Card 1 and Card 3 respectively, and receives partial output data 1 from Card 1 and partial output data 3 from Card 3. It then performs tensor concatenation on partial output data 1, partial output data 2, and partial output data 3 according to the batch dimension to obtain the output data. Card 3 sends partial output data 3 to Card 1 and Card 2 respectively, and receives partial output data 1 sent by Card 1 and partial output data 2 sent by Card 2. The partial output data 1, partial output data 2 and partial output data 3 are tensor concatenated according to the batch dimension to obtain the output data.

[0167] If the shape of partial output data 1 is [Batch_size1, 1], the shape of partial output data 2 is [Batch_size2, 1], and the shape of partial output data 3 is [Batch_size3, 1], Card 1 performs tensor concatenation on partial output data 1, partial output data 2, and partial output data 3 according to the batch dimension to obtain output data with the shape [Batch_size1 + Batch_size2 + Batch_size3, 1].

[0168] In this embodiment of the application, after determining the partial output data of the output layer based on the target parsing result of the partial task corresponding to the card in step 702, the partial output data corresponding to each other card is collected. Based on the partial output data corresponding to the card and the collected partial output data corresponding to each other card, the output data of the output layer is determined, and the partial output data corresponding to each card is sent to each other card respectively, so that each other card can determine the output data of the output layer based on the partial output data corresponding to each card.

[0169] In this embodiment of the application, each card in the data processing system can perform an AllGather communication operation to collect partial output data corresponding to other cards, so that all cards can exchange their respective partial output data.

[0170] For example, a data processing system includes three cards: Card 1, Card 2, and Card 3. The three cards execute in parallel. Each card determines partial output data for the output layer based on the target parsing result of its corresponding partial task. Card 1 corresponds to partial output data 1, Card 2 to partial output data 2, and Card 3 to partial output data 3. If Card 1 is the master card, it receives partial output data 2 from Card 2 and partial output data 3 from Card 3, performs tensor concatenation on partial output data 1, 2, and 3 according to the batch dimension to obtain the output data, and sends partial output data 1, 2, and 3 back to Card 1 and Card 2 respectively. Card 2 sends partial output data 2 to Card 1 and receives partial output data 1, 2, and 3 from Card 1, performs tensor concatenation on partial output data 1, 2, and 3 according to the batch dimension to obtain the output data. Card 3 sends partial output data 3 to Card 1 and receives partial output data 1, partial output data 2 and partial output data 3 sent by Card 1. It performs tensor concatenation on partial output data 1, partial output data 2 and partial output data 3 according to the batch dimension to obtain the output data.

[0171] In this embodiment of the application, after determining the partial output data of the output layer based on the target parsing result of the partial task corresponding to the card in step 702, the partial output data corresponding to each other card is collected. Based on the partial output data corresponding to the card and the collected partial output data corresponding to each other card, the output data of the output layer is determined, and the output data is sent to each other card respectively, so that all cards in the data processing system hold the same complete output data.

[0172] For example, a data processing system includes three cards: Card 1, Card 2, and Card 3. The three cards execute in parallel. Each card determines partial output data for the output layer based on the target parsing result of its corresponding partial task. Card 1 corresponds to partial output data 1, Card 2 to partial output data 2, and Card 3 to partial output data 3. If Card 1 is the master card, it receives partial output data 2 from Card 2 and partial output data 3 from Card 3, performs tensor concatenation on partial output data 1, 2, and 3 according to the batch dimension to obtain the output data, and sends the output data to Card 1 and Card 2 respectively. Card 2 sends partial output data 2 to Card 1 and receives the output data sent by Card 1. Card 3 sends partial output data 3 to Card 1 and receives the output data sent by Card 1.

[0173] In this embodiment of the application, a second data processing method is provided. In this second data processing method, each card in the data processing system selects the sub-semantic information corresponding to the card from the semantic information and selects the sub-weight information corresponding to the card from the weight information. Based on the sub-semantic information and sub-weight information, the sub-parsing results of the batch tasks are obtained and sent to each other card respectively. The sub-parsing results and the sub-parsing results corresponding to each other card are fused to obtain the target parsing results of the batch tasks. The target parsing results of the partial tasks corresponding to the card are selected from the target parsing results. Based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined. Figure 8 A detailed flowchart illustrating the second data processing method provided in this application embodiment is shown below. Figure 8 As shown, it includes at least the following steps 801-807:

[0174] Step 801: Obtain the input data of the output layer of the target model when processing batch tasks.

[0175] Step 802: Based on the parallel parsing rules of multiple cards, select the sub-semantic information corresponding to the card from the semantic information, and select the sub-weight information corresponding to the card from the weight information.

[0176] Step 803: Based on the sub-semantic information and sub-weight information, perform parsing to obtain the sub-parsing results of the batch task, and send the sub-parsing results to each of the other cards respectively.

[0177] Step 804: Merge the sub-parsing results with the sub-parsing results corresponding to the other cards received to obtain the target parsing results for the batch task.

[0178] Step 805: Based on the first task allocation rules of multiple cards, select the target parsing results of the partial tasks corresponding to the card from the target parsing results.

[0179] Step 806: Based on the target parsing results of the partial tasks corresponding to the card, determine the partial output data of the output layer, and send the partial output data to each of the other cards respectively.

[0180] Step 807: Determine the output data of the output layer based on the partial output data corresponding to the card and the partial output data received from other cards.

[0181] For example, when the target model processes batch tasks, the shape of the input data to the output layer is [Batch_size, K]. The data processing system includes p identical cards. The shape of the weight information corresponding to the output layer of the target model is [K, N], where K is the dimension of the hidden layer, N is the vocabulary size, Batch_size is the batch size, and p is an integer greater than 1. For the i-th card, according to the parallel parsing rules of multiple cards, the sub-semantic information corresponding to the i-th card is selected from the semantic information, and the sub-weight information corresponding to the i-th card is selected from the weight information. Let i be an integer, where the shape of the sub-semantic information corresponding to the i-th card is [Batch_size, K / p], and the shape of the sub-weight information corresponding to the i-th card is [K / p, N]. Based on the sub-semantic information and sub-weight information corresponding to the i-th card, the sub-parse result of the batch task corresponding to the i-th card is obtained. And send the sub-resolution results to each of the other cards respectively. Sub-parse results The shape is [Batch_size, N]; the sub-parse results and the sub-parse results corresponding to the other cards are fused to obtain the target parsing result of the batch task. Target analysis results The shape is [Batch_size, N]; based on the task allocation rules of multiple cards, select the target parsing result of the partial task corresponding to the i-th card from the target parsing result. The target parsing result of the partial task corresponding to the i-th card The shape is [Batch_size / p, N]; based on the target parsing result of the partial task corresponding to the i-th card. Determine a portion of the output data from the output layer. And send partial output results to each of the other cards respectively. Partial output data The shape is [Batch_size / p, 1]; based on the partial output data corresponding to the i-th card and the partial output data received from other cards, the output data of the output layer is determined. Output data The shape is [Batch_size, 1].

[0182] In this embodiment of the application, if the card is the primary card, then after step 802, the method further includes: parsing based on sub-semantic information and sub-weight information to obtain the sub-parsing results of the batch task; collecting the sub-parsing results corresponding to each other card; fusing the sub-parsing results corresponding to the primary card and the received sub-parsing results corresponding to each other card to obtain the target parsing result of the batch task; dividing the target parsing result into the target parsing results of the partial tasks corresponding to each card according to the first task allocation rule of multiple cards, and sending the target parsing results of the partial tasks corresponding to each card to each other card respectively; and determining the partial output data of the output layer according to the target parsing results of the partial tasks corresponding to the card.

[0183] For example, a data processing system includes three cards: Card 1, Card 2, and Card 3. The three cards execute in parallel. Card 1 parses the data based on its corresponding sub-semantic information and sub-weight information, obtaining sub-parsing result 1 for the batch task. Card 2 parses the data based on its corresponding sub-semantic information and sub-weight information, obtaining sub-parsing result 2 for the batch task. Card 3 parses the data based on its corresponding sub-semantic information and sub-weight information, obtaining sub-parsing result 3 for the batch task. Card 1 receives sub-parsing result 2 from Card 2 and sub-parsing result 3 from Card 3, and merges sub-parsing result 1, sub-parsing result 2, and sub-parsing result 3 to obtain the target parsing result. Card 1, according to the first task allocation rule for multiple cards, divides the target parsing result into target parsing results for each card's corresponding partial task, and sends its respective partial task's target parsing result to Card 2 and Card 3. Card 2 sends sub-parsing result 2 to Card 1 and receives the target parsing result for its corresponding partial task from Card 1. Card 3 sends sub-parse result 3 to Card 1 and receives the target parsing result of the part of the task corresponding to Card 3 sent by Card 1.

[0184] In this embodiment of the application, in step 203 above, sending sub-parsing results to each other card includes: for each other card, according to the second task allocation rule of multiple cards, selecting the sub-parsing results of the corresponding partial tasks of the other card from the sub-parsing results, and sending the sub-parsing results of the corresponding partial tasks of the other card to the other card.

[0185] The second task allocation rule for multiple cards is a rule for dividing the sub-parsing result into multiple sub-parsing results of partial tasks. It can be set according to the actual situation. Each card corresponds to a sub-parsing result of a partial task.

[0186] For example, a data processing system includes two cards, Card 1 and Card 2, which execute in parallel. Card 1 parses the data based on its corresponding sub-semantic information and sub-weight information to obtain the sub-parsing results for the batch tasks corresponding to Card 1. It then divides these sub-parsing results into partial sub-parsing results for Card 1 and partial sub-parsing results for Card 2, and sends these partial sub-parsing results to Card 2. Similarly, Card 2 parses the data based on its corresponding sub-semantic information and sub-weight information to obtain the sub-parsing results for the batch tasks corresponding to Card 2. It then divides these sub-parsing results into partial sub-parsing results for Card 1 and partial sub-parsing results for Card 2, and sends these partial sub-parsing results to Card 1.

[0187] In this embodiment of the application, after sending the sub-parsing results to each of the other cards, the output data of the output layer is determined based on the sub-parsing results and the received sub-parsing results corresponding to each of the other cards. Figure 9 A flowchart illustrating another method for determining the output data of the output layer provided in this application embodiment is shown below. Figure 9 As shown, step 204 above includes at least the following steps 901-903:

[0188] Step 901: Based on the second task allocation rules for multiple cards, select the sub-parse results of the partial tasks corresponding to the card from the sub-parse results.

[0189] Among them, the second task allocation rule for multiple cards is a setting rule for dividing the sub-parsing result into multiple sub-parsing results of partial tasks. It can be set according to the actual situation, with each card corresponding to a sub-parsing result of a partial task.

[0190] Step 902: Merge the sub-parsing results of the partial tasks corresponding to the card with the sub-parsing results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing result of the partial tasks corresponding to the card.

[0191] In this embodiment of the application, the sub-parsing results of the partial tasks corresponding to the card and the sub-parsing results of the partial tasks corresponding to the card sent by other cards are fused to obtain the target parsing result of the partial tasks corresponding to the card. This includes: performing tensor summation operation on the sub-parsing results of the partial tasks corresponding to the card and the sub-parsing results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing result of the partial tasks corresponding to the card.

[0192] In this embodiment of the application, the target parsing result of the partial task corresponding to the i-th card is determined by the following formula. :

[0193] ;

[0194] in, This is the sub-parse result of the partial task corresponding to the i-th card sent by the j-th card, where i is greater than 0 and less than or equal to p, j is greater than 0 and less than or equal to p, p is the number of cards in the data processing system, and p is an integer greater than 1.

[0195] For example, a data processing system includes three cards: Card 1, Card 2, and Card 3. These three cards execute in parallel, and the sub-parsing result corresponding to Card 1 is... The sub-parse result corresponding to card 2 is The sub-parse result corresponding to card 3 is Card 1 pair parsing results The data is divided to obtain the sub-parse results of the partial tasks corresponding to Card 1. Sub-parse results of some tasks corresponding to card 2 Sub-parse results of some tasks corresponding to card 3 And send the sub-parse results of the corresponding task of card 2 to card 2. Send the sub-parse result of the partial task corresponding to card 3 to card 3. Card 2 pair parsing results The data is divided to obtain the sub-parse results of the partial tasks corresponding to Card 1. Sub-parse results of some tasks corresponding to card 2 Sub-parse results of some tasks corresponding to card 3 And send the sub-parse result of the corresponding task of card 1 to card 1. Send the sub-parse result of the partial task corresponding to card 3 to card 3. Card 3 pair parsing results The data is divided to obtain the sub-parse results of the partial tasks corresponding to Card 1. Sub-parse results of some tasks corresponding to card 2 Sub-parse results of some tasks corresponding to card 3 And send the sub-parse result of the corresponding task of card 1 to card 1. Send the sub-parse result of the partial task corresponding to card 2 to card 2. .

[0196] Card 1 is based on the sub-parse results of the partial tasks corresponding to Card 1, which it determines itself. The sub-parsing result of the task corresponding to the part sent by card 1 from card 2. And the sub-parsing results of the partial task corresponding to card 1 sent by card 3. Determine the target parsing results for part of the tasks corresponding to Card 1. Card 2 is based on the sub-parse results of the partial tasks corresponding to Card 2, which it determines itself. The sub-parsing result of the partial task corresponding to card 2 sent by card 1. And the sub-parsing results of the partial task corresponding to card 2 sent by card 3. Determine the target parsing results for part of the tasks corresponding to Card 2. Card 3 is based on the sub-parse results of the partial tasks corresponding to Card 3, which it determines itself. The sub-parsing result of the task corresponding to the part sent by card 3 from card 1. And the sub-parsing results of the partial task corresponding to the received card 2 and sent card 3. Determine the target parsing results for part of the tasks corresponding to card 3. .

[0197] Step 903: Determine the partial output data of the output layer based on the target parsing results of the partial tasks corresponding to the card.

[0198] In this embodiment of the application, after determining the partial output data of the output layer based on the target parsing result of the partial task corresponding to the card, the output data is sent to each of the other cards respectively, so that all cards in the data processing system hold the same and complete output data.

[0199] In this embodiment, the partial output data of the output layer is determined based on the target parsing results of the partial tasks corresponding to the card. This includes sampling the target parsing results of the partial tasks corresponding to the card to determine the partial output data of the output layer. Since the target parsing results of the partial tasks corresponding to the card represent the final scores of each candidate result of each task in the partial tasks of the batch task, i.e., the complete logits of the partial batch, the correct execution of the sampling operation can be guaranteed.

[0200] In this embodiment, determining partial output data of the output layer based on the target parsing results of a partial task corresponding to the card includes: sampling the target parsing results of the partial task corresponding to the card using a set sampling strategy to determine the partial output data of the output layer. The set sampling strategy is the same for each card.

[0201] In this embodiment, according to the second task allocation rule of multiple cards, the sub-parse results of the partial tasks corresponding to the card are selected from the sub-parse results to represent the initial scores of each candidate result of each task in the partial tasks of the batch task; the sub-parse results of the partial tasks corresponding to the card are fused with the sub-parse results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing results of the partial tasks corresponding to the card, to represent the final scores of each candidate result of each task in the partial tasks of the batch task; and based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined, so that each card processes the target parsing results of its corresponding partial tasks, improving the utilization of computing resources, eliminating the single-card bottleneck, and enabling the data processing system to maintain a high load throughout the entire inference stage (including sampling), making full use of hardware computing power. The performance of the data processing system increases approximately linearly with the increase of the number of cards.

[0202] In this embodiment of the application, the sub-parsing results of a portion of the tasks corresponding to a card are selected from the sub-parsing results according to the second task allocation rules of multiple cards. Figure 10 This application provides a flowchart illustrating a method for selecting a portion of the task's sub-parse result from the sub-parse result, as shown in the embodiment of the present application. Figure 10 As shown, it includes at least the following steps 1001-1002:

[0203] Step 1001: Divide the sub-parsing results based on the computing performance data of each card to obtain a sequence of sub-parsing results.

[0204] The sub-parse result sequence includes the sub-parse results of multiple sub-tasks and the card identifier corresponding to each sub-task's sub-parse result. The number of sub-parse results for each sub-task in the sub-parse result sequence is the same as the number of cards in the data processing system. Each sub-parse result for a sub-task corresponds to a card identifier, and the card identifiers corresponding to the sub-parse results of each sub-task are different. The card identifier corresponding to each sub-parse result of a sub-task is a preset identifier that can uniquely identify a specific card in the data processing system. The identifier can be a string.

[0205] In this embodiment of the application, the task load of the sub-parsing result of the partial task corresponding to each card is determined by the following method: calculating the fourth sum of the computing performance data of each card; and determining the task load of the sub-parsing result of the partial task corresponding to the card based on the ratio of the computing performance data of the card to the fourth sum and the task load of the sub-parsing result.

[0206] In this embodiment of the application, the sub-parsing results are divided based on the computing performance data of each card to obtain a sub-parsing result sequence, including: dividing the sub-parsing results according to the batch dimension based on the computing performance data of each card to obtain a sub-parsing result sequence.

[0207] In this embodiment, if the computing performance data of each card is the same, the sub-parsing results are averaged according to the batch dimension based on the computing performance data of each card to obtain a sub-parsing result sequence. The sub-parsing results of each task in the sub-parsing result sequence have the same shape.

[0208] For example, a data processing system includes p cards. The shape of the sub-parsing result is [Batch_size, N], where Batch_size is the batch size, N is the vocabulary size, and p is an integer greater than 1. If the computational performance data of the p cards are the same, the sub-parsing results are averaged according to the batch dimension based on the computational performance data of each card, resulting in a sub-parsing result sequence. The shape of the sub-parsing result of each task in the sub-parsing result sequence is [Batch_size / p, N].

[0209] In this embodiment, if at least two cards have different computational performance data, the sub-parsing results are divided according to the batch dimension based on the computational performance data of each card to obtain a sub-parsing result sequence. The sub-parsing result sequence may have the following two scenarios: First, the shapes of the sub-parsing results for each task in the sub-parsing result sequence are all different. Second, the shapes of the sub-parsing results for some tasks in the sub-parsing result sequence are different.

[0210] Step 1002: Based on the card identifier, select the sub-parse results of the partial tasks corresponding to the card from the sub-parse result sequence.

[0211] In this embodiment of the application, selecting the sub-parse result of a partial task corresponding to the card from the sub-parse result sequence based on the card identifier includes: matching the card identifier of the card with the card identifier corresponding to the sub-parse result of each partial task in the sub-parse result sequence, and taking the sub-parse result of the partial task corresponding to the matched card identifier as the sub-parse result of the partial task corresponding to the card.

[0212] For example, the sub-parse result sequence includes sub-parse results of three partial tasks: sub-parse results of partial task 1, partial task 2, and partial task 3. The card identifier corresponding to the sub-parse result of partial task 1 is K001, the card identifier corresponding to the sub-parse result of partial task 2 is K002, and the card identifier corresponding to the sub-parse result of partial task 3 is K003. If the card identifier is K003, then the card identifier of the card is matched with the card identifiers corresponding to the sub-parse results of each partial task in the sub-parse result sequence. It is determined that the card identifier corresponding to the sub-parse result of partial task 3 is K003, which is the same as the card identifier of the card. Therefore, the sub-parse result of partial task 3 is taken as the sub-parse result of the partial task corresponding to the card.

[0213] In this embodiment of the application, each card in the data processing system can perform a ReduceScatter communication operation to divide the sub-parsing result into sub-parsing results of the partial tasks corresponding to each card, send the sub-parsing results of the partial tasks corresponding to other cards to other cards, select the sub-parsing results of the partial tasks corresponding to the card from the sub-parsing results according to the second task allocation rule of multiple cards, and receive the sub-parsing results of the partial tasks corresponding to the card sent by each other card, and fuse the sub-parsing results of the partial tasks corresponding to the card and the received sub-parsing results of the partial tasks corresponding to the card sent by each other card to obtain the target parsing result of the partial tasks corresponding to the card.

[0214] ReduceScatter is a many-to-many communication primitive with multiple data senders and multiple data receivers. It performs the same Reduce operation on all nodes in the cluster according to the dimension, and then distributes the results to all nodes in the cluster.

[0215] In this embodiment of the application, after determining the partial output data of the output layer based on the target parsing result of the partial task corresponding to the card in step 903, the partial output data is sent to each of the other cards respectively, and the output data of the output layer is determined based on the partial output data corresponding to the card and the partial output data corresponding to each of the other cards received.

[0216] In this embodiment of the application, the output data of the output layer is determined based on the partial output data corresponding to the card and the partial output data corresponding to each other card received, including: performing tensor concatenation on the partial output data corresponding to the card and the partial output data corresponding to each other card received according to the batch dimension to obtain the output data of the output layer.

[0217] In this embodiment of the application, after determining the partial output data of the output layer based on the target parsing result of the partial task corresponding to the card in step 903, the partial output data corresponding to each other card is collected. Based on the partial output data corresponding to the card and the collected partial output data corresponding to each other card, the output data of the output layer is determined, and the partial output data corresponding to each card is sent to each other card respectively, so that each other card can determine the output data of the output layer based on the partial output data corresponding to each card.

[0218] In this embodiment of the application, each card in the data processing system can perform an AllGather communication operation to collect partial output data corresponding to other cards, so that all cards can exchange their respective partial output data.

[0219] In this embodiment of the application, after determining the partial output data of the output layer based on the target parsing result of the partial task corresponding to the card in step 903, the partial output data corresponding to each other card is collected. Based on the partial output data of the card and the collected partial output data corresponding to each other card, the output data of the output layer is determined, and the output data is sent to each other card respectively, so that all cards in the data processing system hold the same complete output data.

[0220] In this embodiment of the application, a third data processing method is provided. In this third data processing method, each card in the data processing system selects sub-semantic information corresponding to the card from the semantic information and sub-weight information corresponding to the card from the weight information. Based on the sub-semantic information and sub-weight information, the data is parsed to obtain the sub-parsing results of the batch tasks, and the sub-parsing results are sent to each of the other cards respectively. According to the first task allocation rule of multiple cards, the sub-parsing results of the partial tasks corresponding to the card are selected from the sub-parsing results. The sub-parsing results of the partial tasks corresponding to the card are fused with the sub-parsing results of the partial tasks corresponding to the card sent by each of the other cards to obtain the target parsing results of the partial tasks corresponding to the card. Based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined. Figure 11 A detailed flowchart illustrating the third data processing method provided in this application embodiment is shown below. Figure 11 As shown, it includes at least the following steps 111-117:

[0221] Step 111: Obtain the input data of the output layer of the target model when processing batch tasks.

[0222] Step 112: Based on the parallel parsing rules of multiple cards, select the sub-semantic information corresponding to the card from the semantic information, and select the sub-weight information corresponding to the card from the weight information.

[0223] Step 113: Based on the sub-semantic information and sub-weight information, perform parsing to obtain the sub-parsing results of the batch task.

[0224] Step 114: For each other card, according to the second task allocation rule for multiple cards, select the sub-parse result of the corresponding partial task of the other card from the sub-parse result, and send the sub-parse result of the corresponding partial task of the other card to the other card.

[0225] Step 115: Based on the second task allocation rules for multiple cards, select the sub-parse results of the partial tasks corresponding to the card from the sub-parse results.

[0226] Step 116: Merge the sub-parsing results of the partial tasks corresponding to the card with the sub-parsing results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing result of the partial tasks corresponding to the card.

[0227] Step 117: Based on the target parsing results of the partial tasks corresponding to the card, determine the partial output data of the output layer, and send the partial output data to each of the other cards respectively.

[0228] Step 118: Determine the output data of the output layer based on the partial output data corresponding to the card and the partial output data received from other cards.

[0229] For example, when the target model processes batch tasks, the shape of the input data to the output layer is [Batch_size, K]. The data processing system includes p identical cards. The shape of the weight information corresponding to the output layer of the target model is [K, N], where K is the dimension of the hidden layer, N is the vocabulary size, Batch_size is the batch size, and p is an integer greater than 1. For the i-th card, according to the parallel parsing rules of multiple cards, the sub-semantic information corresponding to the i-th card is selected from the semantic information, and the sub-weight information corresponding to the i-th card is selected from the weight information. Let i be an integer, where the shape of the sub-semantic information corresponding to the i-th card is [Batch_size, K / p], and the shape of the sub-weight information corresponding to the i-th card is [K / p, N]. Based on the sub-semantic information and sub-weight information corresponding to the i-th card, the sub-parse result of the batch task corresponding to the i-th card is obtained. Based on the second task allocation principle for multiple cards, from the sub-parse results Select the sub-parse result of the partial task corresponding to the j-th card. And send it to the corresponding j-th card, where, And j is not equal to i. According to the second task allocation principle for multiple cards, from the sub-parse result... Select the sub-parse result of the partial task corresponding to the i-th card. Wherein, the sub-parse result of the partial task corresponding to the i-th card. Sub-parse results of the partial task corresponding to the j-th card The shapes are all [Batch_size / p, N]. The sub-parse results for the partial task corresponding to the i-th card. The sub-parse results of the i-th card's corresponding partial results sent by the other cards are fused together to obtain the target parsing result of the partial task corresponding to the i-th card. The target parsing result of the partial task corresponding to the i-th card The shape is [Batch_size / p, N]; based on the target parsing result of the partial task corresponding to the i-th card. Determine a portion of the output data from the output layer. And send partial output results to each of the other cards respectively. Partial output data The shape is [Batch_size / p, 1]; based on the partial output data corresponding to the i-th card and the partial output data received from other cards, the output data of the output layer is determined. Output data The shape is [Batch_size, 1].

[0230] In this embodiment, the parallel segmentation strategy of the lm_head layer is changed from the traditional segmentation output dimension (vocabulary dimension N) to the segmentation input dimension (hidden layer dimension K). Through this fundamental change, combined with corresponding communication primitives, the sampling computation task, originally concentrated in a single point or repeated execution, is efficiently and load-balancedly distributed across all cards in the data processing system. Specifically, each card holds a different sub-weight information of the weight information and a different sub-weight information of the semantic information, and is responsible for parsing the batch task's sub-parsing results based on its corresponding sub-weight information. Through a single ReduceScatter communication, the dispersed sub-parsing results are reduced and distributed to each card, allowing each card to obtain a portion of the target parsing results for the task—that is, the final correct logits of a portion of the tokens in the entire batch. Subsequently, each card can independently sample the target parsing results assigned to its portion of the task. Finally, through a lightweight AllGather communication, the small-volume partial outputs sampled by all cards are aggregated, thus completing the entire parallel sampling process.

[0231] In this embodiment, by evenly distributing the sampling computation task across all computing cards, the single-card bottleneck is completely eliminated. This allows the data processing system to maintain a high load throughout the entire inference phase (including sampling), fully utilizing hardware computing power, improving computing resource utilization, and reducing redundant computation. Furthermore, this application replaces the massive AllGather communication in existing solutions with a single ReduceScatter communication. Although the data volume may be similar, this operation is more favorable for distributed communication. Moreover, the performance of the data processing system in this application increases approximately linearly with the number of computing cards. Whether it's logits computation or sampling computation, the load is evenly distributed, enabling the system to efficiently handle extremely large batches of inference scenarios.

[0232] Based on the same inventive concept, this application provides a schematic diagram of the structure of a data processing device, such as... Figure 12 As shown, the data processing device 120 includes:

[0233] The acquisition module 1201 is used to acquire the input data of the output layer of the target model when processing batch tasks, the input data including the semantic information and weight information of the batch tasks;

[0234] Selection module 1202 is used to select sub-semantic information corresponding to the card from the semantic information and select sub-weight information corresponding to the card from the weight information according to the parallel parsing rules of multiple cards;

[0235] The parsing module 1203 is used to parse based on the sub-semantic information and the sub-weight information to obtain the sub-parsing results of the batch task, and send the sub-parsing results to each of the other cards respectively;

[0236] The determining module 1204 is used to determine the output data of the output layer based on the sub-parsing result and the sub-parsing results corresponding to the other cards received.

[0237] Optionally, the determining module 1204 is specifically used for:

[0238] The sub-parsing results and the sub-parsing results corresponding to each other card are fused to obtain the target parsing result of the batch task;

[0239] Based on the target parsing results of the batch tasks, the output data of the output layer is determined.

[0240] Optionally, the determining module 1204 is specifically used for:

[0241] Based on the first task allocation rule of the multiple cards, select the target parsing results of a portion of the tasks from the target parsing results;

[0242] Based on the target parsing results of some tasks, determine some of the output data of the output layer.

[0243] Optionally, the parsing module 1203 is specifically used for:

[0244] For each other card, according to the second task allocation rule of the multiple cards, the sub-parse result of the partial task corresponding to the other card is selected from the sub-parse result, and the sub-parse result of the partial task corresponding to the other card is sent to the other card.

[0245] Optionally, the determining module 1204 is specifically used for:

[0246] According to the second task allocation rule of the multiple cards, select the sub-parse result of the partial task corresponding to the card from the sub-parse result;

[0247] The sub-parsing results of the partial tasks corresponding to the card are fused with the sub-parsing results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing result of the partial tasks corresponding to the card.

[0248] Based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined.

[0249] Based on the same inventive concept, embodiments of this application provide a computer device, such as... Figure 13 As shown, it includes at least one artificial intelligence chip 100 and a memory 1101 connected to at least one artificial intelligence chip. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 1101 is not limited. Figure 10 Taking the connection between the AI ​​chip 100 and the memory 1101 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0250] In this embodiment of the application, the memory 1101 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 1101, at least one artificial intelligence chip 100 can perform the steps of the above-described data processing method.

[0251] The artificial intelligence chip 100 serves as the control center of the computer device, connecting to various parts of the device via various interfaces and lines. It implements the tensor splicing method described above by running or executing instructions stored in the memory 1101 and accessing data stored in the memory 1101. Optionally, the artificial intelligence chip 100 may include one or more processing units. It may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 1101 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0252] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the tensor splicing method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0253] Memory 1101, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1101 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 1101 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. In the embodiments of this application, memory 1101 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0254] Based on the same inventive concept, this application provides a data processing system, which includes multiple cards, each card executing the steps of the above-described data processing method.

[0255] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described data processing method.

[0256] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described data processing method.

[0257] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0258] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0259] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0260] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0261] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0262] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A data processing method, characterized in that, The data processing system includes multiple cards, and the method is applied to each card, including: Obtain the input data of the output layer of the target model when processing batch tasks, wherein the input data includes the semantic information and weight information of the batch tasks; According to the parallel parsing rules of the multiple cards, the sub-semantic information corresponding to the card is selected from the semantic information, and the sub-weight information corresponding to the card is selected from the weight information; Based on the sub-semantic information and the sub-weight information, the sub-parsing results of the batch tasks are obtained, and the sub-parsing results are sent to each of the other cards respectively; Based on the sub-parsing result and the sub-parsing results received from other cards, the output data of the output layer is determined.

2. The method as described in claim 1, characterized in that, The step of determining the output data of the output layer based on the sub-parsing result and the sub-parsing results corresponding to each of the other cards received includes: The sub-parsing results and the sub-parsing results corresponding to each other card are fused to obtain the target parsing result of the batch task; Based on the target parsing results of the batch tasks, the output data of the output layer is determined.

3. The method as described in claim 2, characterized in that, The step of determining the output data of the output layer based on the target parsing results of the batch tasks includes: Based on the first task allocation rule of the multiple cards, select the target parsing results of a portion of the tasks corresponding to the card from the target parsing results; Based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined.

4. The method as described in claim 1, characterized in that, Sending the sub-parsing results to each of the other cards respectively includes: For each other card, according to the second task allocation rule of the multiple cards, the sub-parse result of the partial task corresponding to the other card is selected from the sub-parse result, and the sub-parse result of the partial task corresponding to the other card is sent to the other card.

5. The method as described in claim 4, characterized in that, The step of determining the output data of the output layer based on the sub-parsing result and the sub-parsing results corresponding to each of the other cards received includes: According to the second task allocation rule of the multiple cards, select the sub-parse result of the partial task corresponding to the card from the sub-parse result; The sub-parsing results of the partial tasks corresponding to the card are fused with the sub-parsing results of the partial tasks corresponding to the card sent by other cards to obtain the target parsing result of the partial tasks corresponding to the card. Based on the target parsing results of the partial tasks corresponding to the card, the partial output data of the output layer is determined.

6. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the input data of the output layer of the target model when processing batch tasks. The input data includes the semantic information and weight information of the batch tasks. The selection module is used to select the sub-semantic information corresponding to the card from the semantic information and select the sub-weight information corresponding to the card from the weight information according to the parallel parsing rules of multiple cards; The parsing module is used to parse based on the sub-semantic information and the sub-weight information to obtain the sub-parsing results of the batch task, and send the sub-parsing results to each of the other cards respectively; The determining module is used to determine the output data of the output layer based on the sub-parsing result and the sub-parsing results received from other cards.

7. A data processing system, characterized in that, The data processing system includes multiple cards, each card performing the steps of the method according to any one of claims 1 to 5.

8. A computer device, comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method according to any one of claims 1 to 5.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method according to any one of claims 1 to 5.

Images