Model training system, acceleration chip, processing chip and model training method

Abstract

The invention discloses a model training system, an acceleration chip, a processing chip and a model training method, and relates to the technical field of AI. By utilizing the characteristic that the transmission bandwidth of a communication link between core particles is greater than that of a communication link between chips, key value vectors transmitted between different acceleration chips are transmitted through different core particles in the acceleration chips; according to the technical scheme, the communication traffic needing to be transmitted by a single pair of core particles is halved, the communication time of the other half of key value vectors transmitted through the communication link between the core particles is shortened, the total communication time needed in the model training process is shortened, the time needed by model training is shortened, and the model training efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), and in particular to a model training system, an acceleration chip, a processing chip, and a model training method. BACKGROUND

[0002] With the development of the Internet and big data technology, AI models have made progress in information processing and knowledge extraction. The AI model (hereinafter referred to as "model" in the language of the following part) can include but is not limited to a large language model (LLM) or a model that can be used to implement an interactive function. With the continuous evolution of the large language model, the sequence length used by the large language model is getting longer and longer, resulting in an increasing training time of the large language model. A single server cannot store all the features in the large language model, therefore, a distributed Flash attention (FA) method is used to split the parameters in the attention and place them in different chips in the distributed system, thereby reducing the load of a single chip and reducing the problem of low training efficiency caused by the large number of parameters in the model training process due to long sequences. Generally, different chips communicate different features in the FA method to transfer parameters in the model training process. However, the transmission bandwidth between different chips is limited, and the efficiency of model training is affected. SUMMARY

[0003] The present application provides a model training system, an acceleration chip, a processing chip, and a model training method, which solves the problem that the transmission bandwidth between chips is limited, affecting the efficiency of model processing. In the case that the total amount of data to be transmitted between different chips is unchanged, the model training process fully utilizes the characteristic that the bandwidth between chips is greater than the bandwidth between chips, so that the data communication efficiency between different chips is improved, thereby improving the efficiency of model training.

[0004] The present application adopts the following technical solutions.

[0005] In a first aspect, the present application provides a model training system. The model training system includes a plurality of acceleration chips, such as a first acceleration chip and a second acceleration chip. The first acceleration chip includes a first corelet and a second corelet, and the first corelet is configured with a first processing layer of a model. The first corelet is configured with a plurality of first query vectors corresponding to a first set of words in a query statement, and the first set of words includes a plurality of first words, wherein each first word corresponds to a first query vector. The second corelet is configured with a plurality of second query vectors corresponding to a second set of words in the query statement, and the second set of words includes a plurality of second words, wherein each second word corresponds to a second query vector. The second acceleration chip is configured with a key-value vector corresponding to a third set of words in the query statement and a key-value vector corresponding to a fourth set of words in the query statement. In the present application, the second acceleration chip is configured to send the first set of key-value vectors to the first corelet and the second set of key-value vectors to the second corelet, wherein the first set of key-value vectors includes part of the key-value vector corresponding to the third set of words, and the second set of key-value vectors includes part of the key-value vector corresponding to the fourth set of words. The second corelet is configured to send the second set of key-value vectors to the first corelet. The first corelet is configured to use the plurality of first query vectors, the first set of key-value vectors, and the second set of key-value vectors as inputs of the first processing layer, and execute training of the model.

[0006] In the first aspect of the present application, the key-value vectors transmitted between different acceleration chips are transmitted via different pairs of corelets in the acceleration chips, reducing the communication volume required to be transmitted by a single pair of corelets in the inter-chip communication link. Since the transmission bandwidth of the inter-corelet communication link is greater than that of the inter-chip communication link, the total communication volume between different acceleration chips remains unchanged, the communication volume required to be transmitted by a single pair of corelets is halved, and the communication time of the other half of the key-value vectors transmitted via the inter-corelet communication link is reduced. The total communication time required in the model training process is reduced, the time required for model training is reduced, and the efficiency of model training is improved.

[0007] In an optional implementation of the model training system provided in the first aspect, the first core particle and the second core particle are connected by a small input / output (SIO) channel. In addition, the first accelerator chip and the second accelerator chip are connected by one or more of the following: a high-speed custom communication system (HCCS) interface, a high-speed graphics processing unit (GPU) interconnection bandwidth interface, an inter-integrated circuit (I2C) interface, a controller area network (CAN) bus, a serial peripheral interface (SPI), a queued serial peripheral interface (QSPI), a full-duplex asynchronous serial interface, a half-duplex differential serial interface, and the like. In the embodiments of the present application, the transmission bandwidth of the communication link (SIO channel) between the core particles is greater than the transmission bandwidth of the communication link between the chips. The key-value vector does not need to be transmitted through a separate pair of core particles between the chips, but is transmitted through the SIO channel. The communication time of the key-value vector required for a single round of computation of the core particle is reduced, the communication time consumption in the model training process is reduced, and the model training efficiency is improved.

[0008] In an optional implementation of the model training system provided in the first aspect, the number of words in the query statement is N, the number of core particles in the model training system is T, and the number of first query vectors in the first core particle is determined according to the ratio between N and T. The query vectors corresponding to different words in the query statement are assigned to different core particles, which helps to distribute the computation required for model training to different core particles, thereby reducing the time required for a single core particle to execute the model training process, thereby reducing the overall time consumption of model training, and improving the efficiency of model training.

[0009] In an optional implementation of the model training system provided in the first aspect, the first processing layer is an attention layer of a transformer module, and the plurality of first query vectors, the first set of key-value vectors, and the second set of key-value vectors are input data of the transformer module. For example, the attention layer can include but is not limited to a self-attention layer, a multi-head self-attention layer.

[0010] In an optional implementation of the model training system provided in the first aspect, the device serial number of the first acceleration chip is i, i being a natural number.

[0011] In an optional example, the plurality of first words include: words in the query statement whose local word serial numbers match i, and words whose local word serial numbers match T-i+1.

[0012] In another optional example, the plurality of first words include: words in the query statement whose local word serial numbers match i+1, and words whose local word serial numbers match T-i.

[0013] In an optional implementation of the model training system provided in the first aspect, the second acceleration chip includes a third core particle and a fourth core particle. The third core particle is configured to send the first group of key-value vectors to the first core particle. The fourth core particle is configured to send the second group of key-value vectors to the second core particle. In the first aspect of the present application, different core particle pairs in different acceleration chips are used to transmit different key-value vectors, the communication volume required by a single core particle pair is reduced, and the transmission bandwidth of the communication link between the core particles in the same acceleration chip is greater than the transmission bandwidth between different acceleration chips. Therefore, the communication time required by the core particle is reduced under the condition that the total amount of data required by the core particle is unchanged, the model training time is reduced, and the model training efficiency is improved.

[0014] In an optional implementation of the model training system provided in the first aspect, the second core particle is configured to deploy a second processing layer of the model therein, and the second core particle is configured to deploy a plurality of second query vectors of a second group of words in the query statement; the second group of words includes a plurality of second words, and one second word corresponds to one second query vector. The first core particle is further configured to send the first group of key-value vectors to the second core particle. The second core particle is further configured to use the plurality of second query vectors, the first group of key-value vectors, and the second group of key-value vectors as inputs of the second processing layer, and perform training of the model. In the first aspect of the present application, different core particles are used to train different processing layers in the model, which is beneficial to reduce the load of a single core particle, thereby reducing the training time required by the single core particle, reducing the total time required by the model training system for training the model, and improving the efficiency of the model training.

[0015] In an optional implementation of the model training system provided in the first aspect, the number of first query vectors in the first core particle is consistent with the number of second query vectors in the second core particle.

[0016] In an optional implementation of the model training system provided in the first aspect, the sum of the global word sequence numbers of the plurality of first words is consistent with the sum of the global word sequence numbers of the plurality of second words.

[0017] In an optional implementation of the model training system provided in the first aspect, the communication amount of the first acceleration chip and the second acceleration chip in the same communication stage is consistent, and the sum of the global word sequence numbers corresponding to the key-value vectors transmitted by the first acceleration chip is consistent with the sum of the global word sequence numbers corresponding to the key-value vectors transmitted by the second acceleration chip. The communication stage is the process of transmitting the key-value vectors between different acceleration chips.

[0018] In an optional implementation of the model training system provided in the first aspect, the first core particle is specifically configured to input the plurality of first query vectors, the first group of key-value vectors, and the second group of key-value vectors into the first processing layer to obtain updated plurality of first query vectors. The second acceleration chip is further configured to send a third group of key-value vectors to the first core particle and a fourth group of key-value vectors to the second core particle. The third group of key-value vectors includes part of the key-value vectors in the key-value vectors corresponding to the third group of words, and the third group of key-value vectors is different from the first group of key-value vectors. The fourth group of key-value vectors includes part of the key-value vectors in the key-value vectors corresponding to the fourth group of words, and the fourth group of key-value vectors is different from the second group of key-value vectors. The second core particle is further configured to send the fourth group of key-value vectors to the first core particle. The first core particle is further configured to train the first processing layer according to the updated plurality of first query vectors, the third group of key-value vectors, and the fourth group of key-value vectors.

[0019] In an optional implementation of the model training system provided in the first aspect, the first core particle is further configured to obtain a query statement, and determine the plurality of first words corresponding to the first core particle in the query statement according to the number of words in the query statement, the number of core particles of the acceleration chips in the model training system, and the device sequence number of the first acceleration chip. The first core particle is further configured to obtain the plurality of first query vectors according to the plurality of first words.

[0020] In an optional implementation of the model training system provided in the first aspect, the model training system provided in the first aspect further includes a processing chip. The processing chip is configured to receive a query statement, and determine the plurality of first words corresponding to the first core particle in the query statement according to the number of words in the query statement, the number of core particles of the acceleration chips in the model training system, and the device sequence number of the first acceleration chip. The processing chip is further configured to send the plurality of first query vectors corresponding to the plurality of first words to the first core particle.

[0021] In a second aspect, the present application provides an acceleration chip. The acceleration chip comprises: a first core particle and a second core particle. The first core particle is configured with a first processing layer of a model, and the first core particle is configured with a plurality of first query vectors of a first group of words in a query statement, the first group of words comprising a plurality of first words, wherein each first word corresponds to a first query vector. The first core particle is configured to: receive a first group of key-value vectors sent by another acceleration chip; the first group of key-value vectors comprising: part of the key-value vectors corresponding to a third group of words. The second core particle is configured to: receive a second group of key-value vectors sent by another acceleration chip, and send the second group of key-value vectors to the first core particle; the second group of key-value vectors comprising: part of the key-value vectors corresponding to the third group of words. The first core particle is further configured to: take the plurality of first query vectors, the first group of key-value vectors and the second group of key-value vectors as inputs of the first processing layer, and execute training of the model.

[0022] In the second aspect of the present application, since the transmission bandwidth of the inter-core particle communication link is greater than the transmission bandwidth of the inter-chip communication link, the total communication volume between different acceleration chips remains unchanged, the communication volume required for transmission by a single pair of core particles is halved, the communication time of the other half of the key-value vectors transmitted through the inter-core particle communication link is reduced, the total communication time required during model training is reduced, the time required for model training is reduced, and the efficiency of model training is improved.

[0023] For example, the first processing layer is an attention layer of a transformer module, and the plurality of first query vectors, the first group of key-value vectors and the second group of key-value vectors are input data of the transformer module.

[0024] In a third aspect, the present application provides a processing chip. The processing chip comprises: a transceiver and a processor. The transceiver is configured to: receive a query statement. The processor is configured to: determine a plurality of first words corresponding to a first core particle in a first acceleration chip according to a number of words in the query statement and a number of core particles in an acceleration chip in a model training system, the first acceleration chip being any acceleration chip in the model training system, the plurality of first words matching a device serial number of the first acceleration chip. The transceiver is further configured to: send a plurality of first query vectors corresponding to the plurality of first words to the first core particle in the first acceleration chip, wherein each first word corresponds to a first query vector.

[0025] In the third aspect of the present application, the processing chip matches different words for the acceleration chip from the query statement according to the device sequence number of the acceleration chip in the model training system, so that the acceleration chip can communicate with other acceleration chips to balance the communication amount in the model training process, avoid the problem of long model training time caused by long communication time of a single acceleration chip, and improve the model training efficiency. Specifically, under the condition that the total communication amount between different acceleration chips is unchanged, the communication amount required for transmission by a single pair of chiplets is halved, the communication time of the key-value vector transmitted through the inter-chiplet communication link is reduced, the total communication time required in the model training process is reduced, the time required for model training is reduced, and the efficiency of model training is improved.

[0026] In a fourth aspect, the present application provides a model training method. The model training method is applied to the model training system provided in the first aspect or any one of the optional implementation manners of the first aspect. The model training method provided in the fourth aspect of the present application includes: a first chiplet obtaining a plurality of first query vectors of a first group of words in a query statement; the first group of words includes a plurality of first words, wherein each first word corresponds to a first query vector. The first chiplet receives a first group of key-value vectors of a second acceleration chip in the model training system, and a second chiplet receives a second group of key-value vectors of the second acceleration chip; the first group of key-value vectors includes part of the key-value vectors corresponding to the third group of words, and the second group of key-value vectors includes part of the key-value vectors corresponding to the fourth group of words. In addition, the first chiplet receives the second group of key-value vectors sent by the second chiplet, and takes the plurality of first query vectors, the first group of key-value vectors and the second group of key-value vectors as inputs of a first processing layer to perform training of the model.

[0027] In the fourth aspect of the present application, the key-value vectors transmitted between different acceleration chips are transmitted through different pairs of chiplets in the acceleration chip, which reduces the communication amount required for transmission by a single pair of chiplets. Since the transmission bandwidth of the inter-chiplet communication link is greater than that of the inter-chip communication link, under the condition that the total communication amount between different acceleration chips is unchanged, the communication amount required for transmission by a single pair of chiplets is halved, the communication time of the other half of the key-value vectors transmitted through the inter-chiplet communication link is reduced, the total communication time required in the model training process is reduced, the time required for model training is reduced, and the efficiency of model training is improved.

[0028] For example, the first processing layer is an attention layer of a transformer module, and the plurality of first query vectors, the first group of key-value vectors and the second group of key-value vectors are input data of the transformer module.

[0029] In an optional implementation manner of the model training method provided in the fourth aspect, the second core particle in the foregoing description is deployed with a second processing layer of the model, and the second core particle is configured with a plurality of second query vectors of a second group of words in the query statement. The second group of words includes a plurality of second words, and one second word corresponds to one second query vector. The model training method provided in the fourth aspect of the present application further includes: the second core particle receives the first group of key-value vectors sent by the first core particle, and inputs the plurality of second query vectors, the first group of key-value vectors, and the second group of key-value vectors as inputs of the second processing layer, and performs training of the model.

[0030] In an optional implementation manner of the model training method provided in the fourth aspect, the first core particle inputs the plurality of first query vectors, the first group of key-value vectors, and the second group of key-value vectors as inputs of the first processing layer, and performs training of the model, including: the first core particle inputs the plurality of first query vectors, the first group of key-value vectors, and the second group of key-value vectors to the first processing layer to obtain updated plurality of first query vectors. The first core particle receives a third group of key-value vectors sent by a third acceleration chip in the model training system, and the second core particle receives a fourth group of key-value vectors sent by the third acceleration chip; the third group of key-value vectors include part of the key-value vectors in the key-value vectors corresponding to the third group of words, and the third group of key-value vectors are different from the first group of key-value vectors; the fourth group of key-value vectors include part of the key-value vectors in the key-value vectors corresponding to the fourth group of words, and the fourth group of key-value vectors are different from the second group of key-value vectors. The first core particle receives the fourth group of key-value vectors sent by the second core particle, and inputs the updated plurality of first query vectors, the third group of key-value vectors, and the fourth group of key-value vectors to the first processing layer to perform training of the model.

[0031] In the fifth aspect, the present application provides a computer program product. When the computer program product is run in a computing device, the computing device performs the operation steps of the method provided in the fourth aspect or any optional implementation manner of the fourth aspect. The computing device can be, for example, the acceleration chip, the model training system, or the data processing system containing the model training system.

[0032] The beneficial effects of the fifth aspect can be referred to the description of any optional implementation manner of the first aspect to the fourth aspect, which will not be repeated here. Based on the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A structural schematic diagram of a data processing system provided by the present application;

[0034] Figure 2 A structural schematic diagram of a model training system 120 provided by the present application;

[0035] Figure 3 A schematic diagram of distributed training of a model provided for the present application;

[0036] Figure 4 A schematic diagram of vector rearrangement provided for the present application;

[0037] Figure 5 A flowchart of a model training method provided for the present application Figure 1 ;

[0038] Figure 6 A flowchart of a model training method provided for the present application Figure 2 ;

[0039] Figure 7 A flowchart of a model training method provided for the present application Figure 3 ;

[0040] Figure 8 A flowchart of a model training method provided for the present application Figure 4 ;

[0041] Figure 9 A communication schematic diagram of a model training process provided for the present application;

[0042] Figure 10 Effect comparison of different schemes provided for the present application Figure 1 ;

[0043] Figure 11 Effect comparison of different schemes provided for the present application Figure 2 ;

[0044] Figure 12 A structural schematic diagram of a processing chip provided for the present application. DETAILED DESCRIPTION

[0045] In the model training system provided by the embodiments of the present application, the key-value vectors transmitted between different acceleration chips are transmitted via different pairs of core particles in the acceleration chips, thereby reducing the communication amount required to be transmitted by a single pair of core particles in the inter-chip communication link. Since the transmission bandwidth of the inter-core communication link is greater than that of the inter-chip communication link, in the case that the total communication amount between different acceleration chips remains unchanged, the communication amount required to be transmitted by a single pair of core particles is halved, and the communication time of the other half of the key-value vectors transmitted through the inter-core communication link is reduced. The total communication time required in the model training process is reduced, the time required for model training is reduced, and the efficiency of model training is improved.

[0046] The technical solutions provided in this application can be applied not only to current model training or distributed training scenarios, but also to future model training or distributed training technologies. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. The following, in conjunction with the accompanying drawings, provides exemplary descriptions of some application scenarios and system architectures that may be involved in this application.

[0047] Figure 1 A schematic diagram of the structure of a data processing system provided in this application is shown below. Figure 1 As shown, the data processing system includes a computing device 110, a model training system 120, a database 130, a terminal device 140, a data storage system 150, and a data acquisition device 160.

[0048] The computing device 110 can be a terminal, such as a computer, mobile terminal, tablet computer, laptop computer, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, extended reality (ER) device, camera, or in-vehicle computer, etc., or it can be an edge device (e.g., a box carrying a chip with processing capabilities). In this application, the computing device 110 can be a computing device connected to a base station, or a computing device deployed at a base station, such as a server or cloud device.

[0049] The model training system 120 can be a terminal, or other computing devices that support integer or floating-point calculations, such as servers, cloud devices, or chip systems. The model training system 120 includes a processing chip 120a and multiple acceleration chips, such as a first acceleration chip 121, a second acceleration chip 122, etc.

[0050] exist Figure 1 In the illustrated model training system 120, processing chip 120a is used to deploy different processing layers in the model to different acceleration chips. These acceleration chips train the processing layers deployed locally on each acceleration chip based on different input data, thereby improving the training efficiency of the model. Processing chip 120a can be a central processing unit (CPU) for receiving query statements and sending different feature vectors corresponding to the query statements to different acceleration chips, etc. In terms of hardware implementation, processing chip 120a may include one or more processors, which is not limited in this application.

[0051] The acceleration chip provided in the embodiments of the present application includes one or more dies, such as the first acceleration chip 121 including die 1 and die 2, and the second acceleration chip 122 including die 3 and die 4. The first acceleration chip 121 can be, but is not limited to, a GPU chip, a neural network processing unit (NPU) chip, a tensor processing unit (TPU) chip, a micro processing chip, an application-specific integrated circuit (ASIC) chip, or one or more integrated circuit chips for controlling the execution of the computer program product provided in the present application, and the like. Each acceleration chip includes a GPU in a GPU chip, an NPU in an NPU chip, or a TPU in a TPU chip, and the like. For example, if the die is a GPU in a GPU chip, the die can be used to perform mathematical and geometric calculations to achieve image rendering and the like.

[0052] In some possible implementations, different acceleration chips are connected in a full mesh mode (Full Mesh), that is, each acceleration chip is directly connected to each other. In some cases, the communication link between different acceleration chips can also be referred to as a chip-to-chip communication link, that is, chip to chip, such as the connection between different acceleration chips in one or more of the following ways: a high-speed custom communication system (HCCS) interface, a high-speed GPU interconnection bandwidth interface, an integrated circuit interface, a controller area network bus, a serial peripheral interface, a queue serial peripheral interface, a full-duplex asynchronous serial interface, or a half-duplex differential serial interface, and the like.

[0053] The HCCS interface is a high-speed connection channel between dies, which is used to promote data and calculation acceleration to obtain executable results. For example, in the acceleration chip, different dies are connected to each other in pairs using the HCCS technology.

[0054] The high-speed GPU interconnection bandwidth interface is a high-speed interconnection technology between GPUs, which is usually implemented by a plurality of pairs of wires printed on a computer board, and the two ends of a pair of wires are connected to different GPUs.

[0055] An inter-integrated circuit (I2C) bus is a source synchronous serial bus for short distance communication between different integrated circuits. I2C uses two lines for data transmission: a serial data line (SDL) and a serial clock line (SCL). A controller area network (CAN) bus is a serial communication protocol bus for real-time applications, which can use twisted pair wires to transmit signals. A serial peripheral interface (SPI) bus is a 3-wire synchronous serial full-duplex communication interface, which has the advantages of simple circuit, high speed, reliable communication, etc. A queued serial peripheral interface (QSPI) bus is based on SPI and adds a queue transmission mechanism. QSPI uses a dedicated communication interface to connect single, double, or four data lines. A full-duplex asynchronous serial interface is also called a universal asynchronous receiver / transmitter (UART) interface. The UART interface is a general-purpose serial data bus for asynchronous communication. The UART bus can be a bidirectional communication bus. The UART interface converts data to be transmitted between serial communication and parallel communication. For example, the UART interface refers to an RS-232 interface. A half-duplex differential serial interface is a serial communication bus interface. The half-duplex differential serial interface uses two-wire system, differential transmission, and half-duplex mode, such as an RS-485 interface.

[0056] It is worth noting that, unlike the chip to chip communication link, in an acceleration chip, the die to die communication link between the dies can be connected using the SIO interface.

[0057] In the embodiments of the present application, the transmission bandwidth (SIO bandwidth) of the die to die communication link between the dies is greater than the transmission bandwidth of the chip to chip communication link. The transmission bandwidth refers to the maximum amount of data that can be transmitted by different devices in a unit of time. For example, the transmission bandwidth of the chip to chip communication link is 1 Gbps, and the transmission bandwidth of the die to die communication link is 2 Gbps. Figure 1As shown, the SIO bandwidth between the first acceleration chip 121 and the second acceleration chip 122 is greater than the transmission bandwidth between the first acceleration chip 121 and the second acceleration chip 122. For example, the SIO bandwidth between the first acceleration chip 121 and the second acceleration chip 122 is 392 gigabytes per second (GB / s), and the transmission bandwidth between the first acceleration chip 121 and the second acceleration chip 122 is 100 GB / s.

[0058] In a possible embodiment, the computing device 110 and the model training system 120 are deployed on different physical devices (e.g., servers or servers in a cluster), or the computing device 110 and the model training system 120 are different physical devices. For example, the computing device 110 and the model training system 120 are chip systems deployed on different physical devices. For example, the computing device 110 can include a GPU, a central processor, another general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or another programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like. The general-purpose processor can be a microprocessor or any conventional processor, and the like.

[0059] In another possible embodiment, the computing device 110 and the model training system 120 are deployed on the same physical device, or the computing device 110 and the model training system 120 are the same physical device.

[0060] The data collection device 160 is configured to collect training data and store the training data in the database 130. The data collection device 160 can be the same or different from the computing device 110 and the model training system 120. In an embodiment of the present application, the data collection device 160 can be a camera, a camera, a mobile phone, a tablet computer, or a computing device or network device with data collection function.

[0061] The model training system 120 is configured to train the neural network using the training data until the loss function in the neural network converges and the loss function value is less than a certain threshold, and the neural network training is completed, so that the neural network reaches a certain accuracy. Further, the model training system 120 configures the trained neural network to the model training system 120, such as the neural network 101. The computing device 110 is configured to implement reasoning and calculation on a plurality of tokens containing query sentences according to the trained neural network 101.

[0062] In the embodiment, the above-mentioned neural network can be referred to as an AI model or a model, which can be a large language model or other models, and the like.

[0063] In an optional example, the neural network refers to a large language model (LLM). The LLM utilizes its powerful computing capability and complex algorithm to effectively process these massive data and provide efficient and accurate information processing and analysis services for users. The LLM not only performs outstandingly in understanding and generating human language, but also exhibits great potential in solving complex problems and tasks. For example, the LLM is widely used in automated question answering systems, text summarization generation, machine translation, and language generation, etc., greatly improving efficiency and accuracy. In particular, when processing large-scale data sets, the LLM can mine deep patterns and correlations from them to support decision-making. In addition, the self-learning ability of the LLM enables the LLM to continuously evolve, and through continuous learning of new data, the performance and intelligence level of the LLM are continuously improved. Commonly used LLMs include the Bidirectional Encoder Representations from Transformers (BERT) model, which can be pre-trained using unlabelled text to train deep bidirectional representations. In some optional cases, the LLM can also be referred to as a large model. The large model provided by the embodiments of the present application can not only refer to the LLM, but also refer to a model whose model parameters reach a certain degree. For example, according to the field to which the large model is applied, the large model can also refer to a model that contains various functions such as image processing function, human-computer interaction function, semantic search function, semantic query function, or dialogue function, etc. The present application does not limit the field to which the large model can be applied and the specific name. In this paper, in order to facilitate description, the large model is named, but this should not be understood as a limitation of the present application, which will not be described in detail hereinafter.

[0064] Because the LLM contains a large number of processing layers, it takes a long time to train the LLM using a single device. Therefore, the distributed training method is used to train the LLM. For details, please refer to the following Figures 3 to 11 The embodiments provided in the present application will not be described here.

[0065] In an optional implementation, the neural network refers to other types of networks. For example, the neural network 101 is a convolutional neural network (CNN), a recurrent neural network (RNN), or a graph neural network (GNN), etc. For more implementations of CNN, RNN, or GNN, please refer to the description of the general technology, which will not be described here.

[0066] In some embodiments, if the computing device 110 and the model training system 120 are the same physical device, the model training system 120 can configure the trained neural network 101 to itself, and use the trained neural network 101 to achieve the target function of the model, such as identification or document verification in a business district, school, park, or sports venue in a city, or other functions, such as target detection, object recognition, or classification operations on data, or functions such as face payment or object classification (such as product classification).

[0067] In other embodiments, the model training system 120 can configure the trained neural network 101 to multiple computing devices, so that each computing device can achieve the target function of the model described above. In addition to the target functions of the above embodiments, the neural network 101 can also achieve some functions that can be achieved by a large language model (LLM), and the like.

[0068] It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the data collection device 160, but may also be received from other devices. In addition, the computing device 110 may not train the neural network based entirely on the training data maintained in the database 130, but may also obtain training data from the cloud or other places to train the neural network. The above description should not be considered as a limitation of the embodiments of the present application.

[0069] Further, according to the functions performed by the computing device 110, the computing device 110 can be further divided into an architecture as shown in Figure 1 As shown in Figure 1 The computing device 110 is configured with an inference module 111, an I / O interface 112, and a preprocessing module 113.

[0070] Taking the computing device 110 as an example connected to an access station such as a base station, a wireless access point, etc., the inference module 111 can be a GPU, a CPU, other general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. on a vehicle-mounted computer.

[0071] For example, the inference module 111 is used to run the neural network 101 to implement a semantic query function, etc., to obtain a query result corresponding to a query statement of a user.

[0072] The I / O interface 112 is used for data interaction with external devices. A user can input data to the I / O interface 112 through a terminal device 140, for example, an instruction for instructing the computing device 110 to start feature extraction or model training on training data. In addition, the input data can also come from the database 130.

[0073] The preprocessing module 113 is configured to perform preprocessing on the input data received by the I / O interface 112. In the embodiments of the present application, the preprocessing module 113 can be configured to generate training data, such as a training set, a validation set and a test set, from the input data received by the I / O interface 112. Optionally, the preprocessing module 113 can also perform preprocessing operations such as denoising and eliminating irrelevant information on the input data, such as a feature vector corresponding to a query statement, to recover useful real information.

[0074] During the preprocessing of the input data by the computing device 110 or the execution of the inference module 111 of the computing device 110 to perform the related processing, the computing device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data and instructions obtained by the corresponding processing in the data storage system 150.

[0075] Finally, the I / O interface 112 returns the processing result to the terminal device 140, so as to provide the user with the processing result for viewing. In the embodiments of the present application, the processing result can be an updated or optimized model, a query result corresponding to a query statement, etc.

[0076] The terminal device 140 can be configured to collect the input data input to the I / O interface 112 and the processing result output from the I / O interface 112 as new sample data, and store the new sample data in the database 130. Of course, the sample data can also be collected without the terminal device 140, and the input data input to the I / O interface 112 and the processing result output from the I / O interface 112 can be stored as new sample data in the database 130 by the I / O interface 112.

[0077] Figure 1 The figure is only a schematic diagram of a system architecture provided by the embodiments of the present application, Figure 1 The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation, and the model training system and the computing device can include more or fewer hardware components according to the user's demand for model training, which is not limited in the present application. For example, in the Figure 1 In the figure, the data storage system 150 is an external memory of the computing device 110, and in other cases, the data storage system 150 can also be placed in the computing device 110.

[0078] The scenarios to which the present application can be applied include but are not limited to the following: a semantic query scenario of human-computer interaction, a model training scenario, a model optimization scenario, and a scenario involving distributed training of a model, etc.

[0079] The structure of the model training system 120 will be introduced below based on the model training system 120 shown in the figure, Figure 1 The structure of the model training system 120 will be introduced below based on the model training system 120 shown in the figure,Figure 2 A structural diagram of a model training system 120 provided by the present application is shown. Please refer to Figure 2 The model training system 120 includes a processing chip 120a and a plurality of acceleration chips, such as a first acceleration chip 121, a second acceleration chip 122, a third acceleration chip 123, and a fourth acceleration chip 124, etc.

[0080] The processing chip 120a and the plurality of acceleration chips are connected in a full mesh mode, i.e., any two chips communicate with each other in a direct connection mode. For example, an acceleration chip communicates with any other chip in a direct connection mode. The interface used for direct connection between different chips can refer to the related description, which will not be repeated here. Figure 1

[0081] In the model training system 120 shown in Figure 2 Each acceleration chip includes one or more dies, such as the first acceleration chip 121 including die 1 and die 2, the second acceleration chip 122 including die 3 and die 4, the third acceleration chip 123 including die 5 and die 6, and the fourth acceleration chip 124 including die 7 and die 8. For example, different dies within the same acceleration chip are connected by an SIO interface, and the transmission bandwidth of the communication link (or channel) connected by the SIO is greater than the transmission bandwidth of the communication link between different chips.

[0082] Based on the model training system 120 shown in Figure 2 , it is assumed that the model training system 120 of the present application is used to train an AI model based on a transformer architecture. A possible example is shown in Figure 3 Figure 3 A structural diagram of distributed training of a model provided by the present application is shown. Please refer to Figure 3 The model includes a plurality of transformer modules, such as transformer module 1, transformer module 2, and transformer module n, etc., where n is a positive integer; and the plurality of transformer modules are connected in a full mesh mode.

[0083] Different transformer modules depend on Figure 1 or Figure 2 ​​The model training system 120 is shown to implement, each transformer module corresponding to an attention layer can be trained by one or more cores (or acceleration chips), thereby improving the training efficiency of the model. For example, different transformer modules in the model are configured to different devices for training. For example, the control chip 120a configures transformer module 1 to core 1, transformer module 2 to core 2, and transformer module n to core 5. Other transformer modules can also be configured to other cores in the model training system 120, which will not be described here.

[0084] During the training of the model, the control chip 120a obtains a sequence and inputs the sequence into the model. Different cores are used to train the transformer module according to a part of the words in the sequence, and output the query result corresponding to the part of the words. It can be understood that different transformer modules in the model are trained by different cores or acceleration chips, which realizes the parallel training process of the model.

[0085] In some optional examples, the sequence is variable, and part or all of the elements can be modified; and the sequence can be iterated to traverse the elements in the sequence, and the device can also obtain a sub-sequence in the sequence by slicing. During the use of the sequence, one or more operations can be performed on the sequence as needed, such as adding elements, deleting elements, sorting, searching, etc. For example, the elements in the sequence can include any type of elements, such as numbers, strings, objects, arrays, tuples, linked lists (or lists), tokens, stacks, and queues, etc. A string is a sequence composed of characters, commonly used in text processing, cryptography, image processing, etc. An array is a sequence composed of elements of the same type, commonly used to store a large amount of data and perform numerical calculations. A linked list is a sequence composed of nodes, commonly used to implement dynamic data structures and efficient insertion and deletion operations. A tuple is an immutable sequence, commonly used to package multiple values into a whole. A list is a variable sequence, commonly used to store and manipulate data. Stacks and queues are special sequences, commonly used to implement data structures and algorithms.

[0086] Referring to Figure 3In the embodiment of the present application, the sequence can refer to a query statement, which is semantic information generated by the control chip 120a according to a sequence of text or audio information, and the text or audio information is input by a user, transmitted by another device, or generated by a model training system 120 according to a locally configured training data generation program. Please refer to Figure 3 The query statement includes a plurality of tokens, such as token 1 to token m corresponding to the black pattern. In this article, the token can include but is not limited to: word, character, punctuation, special symbol (such as calculation symbol, etc.), etc. In some optional cases, the word can also be referred to as token, which is not limited in the present application.

[0087] As the user's demand for the processing capacity of the model gradually increases, the number of tokens in the query statement increases, and the processing efficiency of the device is low when deploying the model on a single device to process long sequences. In this example, the long sequence refers to a query statement containing a large number of tokens. The following takes core particle 1 as an example to briefly describe the training process of the model. The control chip 120a randomly or sequentially allocates the query (Q) vector, key (K) vector and value (V) vector corresponding to different tokens in the query statement to different core particles, such as the control chip 120a allocating the Q, K and V corresponding to the token 1 in the query statement to the core particle 1, and the core particle 1 performing multiple calculations on the key vector and value vector stored locally by the core particle 1, the key vector and value vector transmitted by other core particles or acceleration chips according to the query vector stored locally by the core particle 1, and executing the training process of the transformer module 1 in the model. For example, the core particle 1 obtains a first result after calculating the query vector stored locally by the core particle 1 with all key-value vectors associated with the query vector in the model training system 120, determines the loss function between the first result and the annotation of the token corresponding to the query vector stored locally by the core particle 1, and finally optimizes and updates the model parameters of the attention layer 1 deployed in the core particle 1 according to the loss function.

[0088] For possible implementation of the query vector (Q), the key vector (K) and the value vector (V), the following takes a token (such as token 1) in the query statement as an example: the control chip 120a converts the token 1 into a vector to obtain the Q, K and V corresponding to the token 1. For example, the process of converting the token 1 into a vector satisfies the following formulas (1) to (3).

[0089] Q=KW Q Formula (1)

[0090] K = XW K Equation (2)

[0091] V = XW V Equation (3)

[0092] wherein X is an embedding representation corresponding to token 1, i.e., an abstract representation of token 1 mapped to a low-dimensional space; Q is a query vector, W Q is a weight matrix corresponding to the query vector, K is a key vector, W K is a weight matrix corresponding to the key vector, V is a value vector, W V is a weight matrix corresponding to the value vector.

[0093] In this article, for the sake of simplicity, the key vector and the value vector can be referred to as a key-value vector.

[0094] Please continue to refer to Figure 3 Different cores are used for different transformer modules in the model training, and the input data used for optimizing and updating the model parameters of different transformer modules are not exactly the same, such as the query vectors and key-value vectors stored in different cores. That is, the multiple cores of full interconnection are used to train different data at the same time, realizing the data parallel training process of the model. Since the input data of the model is a query statement, and the query statement is a sequence, the data parallelism can also be referred to as sequence parallelism, which refers to: dividing the input data of the model into multiple sub-data, and distributing different sub-data to different devices for calculation, so as to reduce the amount of calculation to be performed by a single device and improve the calculation efficiency of these devices on the input data of the model.

[0095] For example, the transformer module 1 includes an attention layer 1 and a multilayer perceptron (MLP) 1. The attention layer 1 is used to calculate the query vector, the key vector and the value vector corresponding to the input data 1 (such as token 1 and token 2 in the query statement), and finally outputs the result of the attention layer 1. The MLP 1 includes two linear layers, which are used to linearly process the result output by the attention layer 1 to obtain the output result (output data 1) of the transformer module 1, and update the model parameters of the transformer module 1 based on the output result. Figure 3 The contents and training methods of other transformer modules shown in FIG. 1 can refer to the description of the transformer module 1. For the specific network structure of the transformer module deployed in the core, the following will be combinedFigure 3 An exemplary illustration is made.

[0096] See Figure 3 The transformer module includes an encoder and a decoder.

[0097] The encoder is used to process the input data and convert it into an internal representation (e.g., an encoding matrix) that the model can understand. The encoder is stacked with multiple identical layers, and each layer has a network structure containing two main substructures: a multi-head self-attention mechanism and a feed-forward neural network. As shown in Figure 3 The encoder includes a multi-head self-attention layer 1, a residual connection, and an Add&Norm (1) operation layer, a feed-forward layer (1), and an Add&Norm (2). The multi-head self-attention layer 1 is a network processing layer determined based on a multi-head self-attention mechanism, which is used to calculate the dot product between the Q corresponding to the token and the other K and V associated with the Q to obtain the correlation score between each token and other tokens. The feed-forward layer (1) is a network processing layer determined based on a feed-forward neural network, which is used to perform forward feedback on the model parameters. Add&Norm (1) and Add&Norm (2) are used to prevent the network layers in the encoder from degenerating and to normalize the activation values of different network layers, such as weighting and summing the correlation scores determined by the multi-head self-attention layer 1 to obtain a new representation of each token, such as an encoding matrix.

[0098] The decoder generates output data (e.g., output data 1) according to the encoding matrix generated by the encoder. The encoder includes a mask multi-head self-attention layer, Add&Norm (3), a multi-head self-attention layer 2, Add&Norm (4), a forward feedback layer (2), Add&Norm (5), a linear processing layer, and a normalization processing layer. The functions of Add&Norm (3) to Add&Norm (5) can refer to the description of the foregoing encoder, and will not be described here. The difference between the mask multi-head self-attention layer and the multi-head self-attention layer 2 is that the mask multi-head self-attention layer adopts a masked operation, the KV of the multi-head self-attention layer 2 uses the encoding matrix of the encoder to calculate, and the Q uses the output of the previous decoding block to calculate. The previous decoding block of the multi-head self-attention layer 2 refers to the decoder corresponding to another word before predicting the token 1. After the linear processing layer and the normalization processing layer perform linear processing and normalization operations, the predicted result corresponding to the token 1 is output, such as output data 1.

[0099] Figure 3 The structure of the transformer model shown enables the model to process sequence-to-sequence (seq2seq) tasks, such as machine translation.

[0100] In combination Figure 3 According to the related embodiments, because the control chip 120a randomly or sequentially allocates the vectors (Q, KV) of different words to different core particles, the communication amount of different core particles is greatly different, and in a round of model training, there is a problem of large communication tail caused by the fact that the communication time of some core particles is short and the communication time of some core particles is long, and the model training efficiency is reduced. Moreover, in the training stage, because the communication between different core particles is through the chip-to-chip channel of full interconnection, the transmission bandwidth of the chip-to-chip channel is low, which further increases the communication time in the model training process, and the model training efficiency is affected.

[0101] For the training process of different processing layers (such as the attention layer or MLP described above) in the model, the following will be described in combination with Figure 1 and Figure 2Based on the model training system 120 shown, taking a large language model as an example, the model training method provided in this application embodiment is introduced. This model training method includes a vector rearrangement stage and a training stage. The vector rearrangement stage is used to configure the feature vectors (query vector, key vector, and value vector) corresponding to different words (tokens) in the query statement input to the large language model to different cores, so as to maintain a balanced amount of communication transmitted between different cores. The training stage is used to communicate the feature vectors corresponding to words configured in different cores, so that each core calculates the locally stored query vector with all key vectors and value vectors to obtain the model parameters of the processing layer deployed in that core.

[0102] The vector rearrangement stage of the model training method will be illustrated below. The vector rearrangement stage can be performed by a chip in the acceleration chip, such as chip 1 in the first acceleration chip 121; the vector rearrangement stage can also be performed by a processing chip, such as processing chip 120a.

[0103] The following example illustrates the vector rearrangement stage performed by processing chip 120a. Assuming the number of words in the query statement is N and the number of kernels in the model training system is T, processing chip 120a determines the number of query vectors (Q) configured in each kernel of the model training system 120 based on the ratio between N and T. For example, the number of Qs in a kernel is N / T. When N cannot be evenly distributed across T kernels, the query vectors (Q) or key-value vectors (KV) required to be stored in the last kernel or a specific kernel are reduced.

[0104] The following explanation uses a model training system 120 that includes only the first accelerator chip 121 and the second accelerator chip 122 as an example. Figure 4 As shown, Figure 4 This is a schematic diagram of a vector rearrangement provided in this application. The query statement includes 16 words (N=16, i.e., token 1 to token 16). The acceleration chip in the model training system contains 4 chips (T=4). Therefore, each chip needs to be allocated 4 query vectors (Q) and 4 sets of key-value vectors (or: key-value vector sets).

[0105] The terms in the query statement can be, but are not limited to, words, characters, punctuation marks, and special symbols (such as calculation symbols). All terms in the query statement have a global rank, such as 1 to 16; and after being assigned to different tokens, each term also has a local rank, such as 1 to 4. For ease of description, this paper uses the global rank of each term in the query statement to describe the different terms, i.e., token 1 to token 16.

[0106] The processing chip 120a receives a query statement (including tokens 1 to 16) and sorts the tokens in the query statement according to the order in which they were received. Furthermore, the processing chip 120a divides the 16 tokens into four sequence groups, with each sequence group corresponding to one chip.

[0107] exist Figure 4 In the Q sorting shown in (a), the processing chip 120a may sort the query vectors (Q) corresponding to tokens 1 to 16 in a manner that includes, but is not limited to: rearranging tokens 1 to 16 if the sum of the local word indices corresponding to Q is consistent across different chips. The global word indices and local word indices corresponding to Q stored in each chip after rearrangement are shown in Table 1 below.

[0108] Table 1

[0109]

[0110] For the words corresponding to different chips, in this embodiment of the application, the set of words corresponding to chip 1 is called the first set of words. The first set of words includes token 1, token 8, token 9, and token 16, which are called the first words, and the query vector corresponding to the first words is the first query vector.

[0111] As an optional implementation, the token corresponding to the query vector (Q) in each chip is determined based on the device number (i) of the acceleration chip to which the chip belongs in the model training system 120.

[0112] In an optional example, the first set of words mentioned above includes: the words whose local word index matches i and the words whose local word index matches T-i+1 among all the words in the query statement. Here, T is the total number of chips in the acceleration chip of the model training system 120 (T=4), i is the device index of the first acceleration chip 121 (i=1), and the device indices in the model training system 120 are sorted starting from 1. Then, the processing chip 120a selects the words (tokens) corresponding to local word indices 1 and 4 as the first words, namely token 1, token 8, token 9, and token 16, and sends the first query vectors corresponding to token 1, token 8, token 9, and token 16 to chip 1, so that chip 1 stores the first query vectors corresponding to token 1, token 8, token 9, and token 16.

[0113] In another optional example, the plurality of first words includes the words whose local word sequence numbers match i+1 and T-i among all the words of the query statement. Wherein, T is the total number of cores included in the acceleration chip in the model training system 120 (T=4), i is the device sequence number of the first acceleration chip 121 (i=0), and the device sequence number in the model training system 120 is ordered from 0. Then the processing chip 120a selects the words corresponding to the local word sequence numbers 1 and 4 as the first words, i.e., token 1, token 8, token 9 and token 16, and sends the first query vectors corresponding to token 1, token 8, token 9 and token 16 to the core 1, so that the core 1 stores the first query vectors corresponding to token 1, token 8, token 9 and token 16.

[0114] The above two optional examples are only optional ways of configuring the query vectors corresponding to different words in the query statement in the cores provided by the embodiments of the present application, and should not be understood as a limitation of the present application.

[0115] Please continue to refer to Figure 4 and the content shown in Table 1, the group of words corresponding to the core 2 is called the second group of words, the token 4, token 5, token 12 and token 13 included in the second group of words are called the second words, and the query vector corresponding to the second words is the second query vector. As shown in Figure 4 , the query vector configured in the core 2 is the second query vector corresponding to token 4, token 5, token 12 and token 13.

[0116] The group of words corresponding to the core 3 is called the third group of words, the token 2, token 7, token 10 and token 15 included in the third group of words are called the third words, and the query vector corresponding to the third words is the third query vector. As shown in Figure 4 , the query vector configured in the core 3 is the third query vector corresponding to token 2, token 7, token 10 and token 15.

[0117] The group of words corresponding to the core 4 is called the fourth group of words, the token 3, token 6, token 11 and token 14 included in the fourth group of words are called the fourth words, and the query vector corresponding to the fourth words is the fourth query vector. As shown in Figure 4 , the query vector configured in the core 4 is the fourth query vector corresponding to token 3, token 6, token 11 and token 14.

[0118] According to the above Figure 4 and Table 1, it can be seen that the sum of the local token sequence numbers corresponding to Q in different core particles is consistent (1+4+1+4=2+3+2+3=10), and the sum of the global token sequence numbers corresponding to Q in different core particles is consistent (1+8+9+16=34). In the embodiment of the present application, by configuring tokens with different global token sequence numbers in different core particles, the data amount of the key vector and the value vector required for transmission by different core particles in the training processing layer tends to be consistent, avoiding the problem of longer communication time caused by the large data amount required for transmission by a single core particle. Please refer to Figure 4 The (ii) KV sorting shown in the figure.

[0119] In Figure 4 In the (ii) KV sorting shown in the figure, the manner in which the processing chip 120a sorts the key vector and the value vector (KV) corresponding to the token 1 to the token 16 can include but is not limited to: making the first acceleration chip 121 and the second acceleration chip 122 consistent in the amount of communication in the same communication stage, and making the sum of the global token sequence numbers corresponding to the key value vector transmitted by the first acceleration chip 121 consistent with the sum of the global token sequence numbers corresponding to the key value vector transmitted by the second acceleration chip 122, and sorting the key vector and the value vector (KV) stored in each core particle. Wherein, the communication stage is the process of transmitting the key value vector between different acceleration chips, such as the process of transmitting the key value vector including sending the key value vector and receiving the key value vector.

[0120] The global token sequence number and the local token sequence number corresponding to the KV stored in each core particle after rearrangement are shown in Table 2.

[0121] Table 2

[0122]

[0123] Among them, the key value vector stored in the core particle 1 includes the KV corresponding to the token 1, the token 16, the token 9, and the token 8.

[0124] The key value vector stored in the core particle 2 includes the KV corresponding to the token 4, the token 13, the token 12, and the token 5.

[0125] The key value vector stored in the core particle 3 includes the KV corresponding to the token 2, the token 15, the token 10, and the token 7.

[0126] The key value vector stored in the core particle 4 includes the KV corresponding to the token 3, the token 14, the token 11, and the token 6.

[0127] In combination Figure 4 As can be seen from the contents of Table 1 and Table 2, the same thing between the KV ordering and the Q ordering is that the global token sequence numbers of the tokens corresponding to each group of key value vectors (KV) in the corelet are consistent, and the sum of the global token sequence numbers of the KVs in each corelet is consistent.

[0128] The difference between the KV ordering and the Q ordering is that the KVs corresponding to the second token and the fourth token in the local ordering of the corelet are exchanged, so that the single calculation amount of a single corelet in the model training process is consistent, that is, the effective calculation amount is 8.

[0129] The above Figure 4 The provided embodiments are described by taking the execution of the processing chip 120a as an example, but in some optional cases, the process of vector rearrangement can also be executed by any acceleration chip or corelet in the model training system 120, such as the first acceleration chip 121 or the corelet 1, and the present application does not limit this.

[0130] Taking the corelet 1 executing the process of vector rearrangement as an example, the corelet 1 obtains a query statement, and determines a plurality of first tokens in the query statement corresponding to the first corelet (corelet 1) according to the number of tokens (N) in the query statement, the total number of corelets (T) of the acceleration chips in the model training system, and the device sequence number (i) of the first acceleration chip 121. And the corelet 1 obtains a plurality of first query vectors according to the plurality of first tokens. For the specific implementation manner of the corelet 1 executing the vector rearrangement, reference can be made to the foregoing Figure 2 related embodiments, which will not be described here. The difference between the corelet 1 executing the vector rearrangement and the processing chip 120a executing the vector rearrangement is that the plurality of first query vectors locally required to be configured by the corelet 1 do not need to be sent through the inter-chip communication link, reducing the communication amount required for model training.

[0131] According to Figure 4 and Figures 1 to 4 As shown in the model training system 120, different acceleration chips are used to train different processing layers in the model during the training of the model. The processing layers here can include but are not limited to: an input layer, a sampling layer, a convolution layer, a pooling layer, one or more hidden layers in a neural network layer, an output layer, etc. More description of different processing layers in the model can refer to the description of the prior art, which will not be described here.

[0132] It can be understood that training different processing layers in the model by different acceleration chips in the model training system 120 is beneficial to improve the training efficiency of the model. However, due to the small transmission bandwidth between different acceleration chips, in the case that the communication amount of transmitting data between different acceleration chips is large, the acceleration chip needs to wait for a long time to receive new input data, and the input data is calculated by the acceleration chip before a round of training is completed, so the efficiency of model training is low.

[0133] In the following, taking the first processing layer in which the model is deployed in the first corelet (corelet 1) and the second processing layer in which the model is deployed in the second corelet (corelet 2) as an example, on the basis of Figure 5 , the embodiment of the present application provides a model training method, as shown in Figure 5 , the embodiment of the present application provides a model training method, as shown in Figure 1 , the embodiment of the present application provides a model training method, as shown in Figure 5 . The hardware implementation of the first acceleration chip 121 and the second acceleration chip 122 can refer to the description of the foregoing embodiments, which will not be repeated here.

[0134] Please refer to Figure 4 , the first corelet 1 is configured with a plurality of first query vectors (Q) corresponding to a first group of tokens (token 1, token 8, token 9, token 16), and a plurality of groups of key-value vectors (KV) corresponding to the first group of tokens are locally stored; the second corelet 2 is configured with a plurality of second query vectors (Q) corresponding to a second group of tokens (token 4, token 5, token 12, token 13), and a plurality of groups of key-value vectors (KV) corresponding to the second group of tokens are locally stored. The key-value vectors stored in other corelets can refer to the description of Figure 3 , which will not be repeated here. In the embodiment of the present application, the number of first query vectors in the first corelet 1 is consistent with the number of second query vectors in the second corelet 2.

[0135] Optionally, the first processing layer is an attention layer (attention layer) of a transformer module. Illustratively, the attention layer can include but is not limited to a self-attention layer or a multi-head self-attention layer. The self-attention layer is used to calculate the Q and KV of the token to obtain the calculation result corresponding to the token. In the self-attention layer, there will be an association between the tokens, assuming that a query sentence has 50 tokens, then it can be roughly imagined that in the self-attention calculation process, a 50x 50 association matrix will be constructed and calculated. The specific implementation of the multi-head self-attention layer can refer to the description of Figure 5 , which will not be repeated here. In addition, the second processing layer is also similar to the first processing layer described above, which will not be repeated here.

[0136] The model training method provided by the embodiment of the application includes the following S510 to S530.

[0137] S510, the second acceleration chip 122 sends the first group of key-value vectors to the corelet 1 and the second group of key-value vectors to the corelet 2.

[0138] Corresponding to the process of S510, the corelet 1 receives the first group of key-value vectors sent by the second acceleration chip 122, and the corelet 2 receives the second group of key-value vectors sent by the second acceleration chip 122.

[0139] The first group of key-value vectors includes part of the key-value vectors in the key-value vectors corresponding to the third group of words in the query statement, such as the KV corresponding to token 2 and token 15. The second group of key-value vectors includes part of the key-value vectors in the key-value vectors corresponding to the fourth group of words in the query statement, such as the KV corresponding to token 3 and token 14.

[0140] For example, the corelet 3 sends the first group of key-value vectors to the corelet 1, and the corelet 4 sends the second group of key-value vectors to the corelet 2. In the case of communication between different acceleration chips, different pairs of corelets are used to transmit the key-value vectors between different acceleration chips, thereby reducing the communication amount required for transmission of the inter-chip communication link by a single pair of corelets.

[0141] S520, the corelet 2 sends the second group of key-value vectors to the corelet 1.

[0142] Corresponding to the process of S520, the corelet 1 receives the second group of key-value vectors sent by the corelet 2.

[0143] It is worth noting that the key-value vectors sent by the second acceleration chip 122 are transmitted through the inter-chip communication link, and the second group of key-value vectors sent by the corelet 2 are transmitted through the inter-core communication link.

[0144] S530, the corelet 1 takes the plurality of first query vectors, the first group of key-value vectors and the second group of key-value vectors as inputs of a first processing layer, and performs training of the model.

[0145] Referring to Figure 5 , the plurality of first query vectors in the corelet 1 include Q corresponding to token 1, token 8, token 9 and token 16; the first group of key-value vectors include KV corresponding to token 2 and token 15; and the second group of key-value vectors include KV corresponding to token 3 and token 14.

[0146] In Figure 5In the provided model training method, since the transmission bandwidth of the inter-corelet communication link is greater than the transmission bandwidth of the inter-chip communication link, the total communication volume between different acceleration chips remains unchanged, the communication volume required for transmission by a single pair of corelets is halved, and the communication time of the other half of the key-value vectors transmitted through the inter-corelet communication link is reduced, the total communication time required in the model training process is reduced, the time required for model training is reduced, and the efficiency of model training is improved.

[0147] Specifically, the transmission bandwidth of the SIO channel is greater than the transmission bandwidth of the inter-chip communication link, and the inter-chip does not need to transmit the key-value vector through a single pair of corelets, but uses the SIO channel to transmit the key-value vector, the communication time of the key-value vector required by the corelet to perform a single round of calculation is reduced, the communication time-consuming in the model training process is reduced, and the efficiency of model training is improved.

[0148] In the model training system 120, different corelets in the same acceleration chip can also be used to train different processing layers of the model, and in Figure 6 the prior art, the present application provides a feasible example, as Figure 6 shown, Figure 2 a flowchart of a model training method provided by the present application Figure 5 . For the specific implementation of each acceleration chip and the vector configured in the corelet, refer to the description of the foregoing embodiments, S510 to S530 have been described in Figure 6 the related embodiments, and will not be repeated here.

[0149] Please refer to Figure 6 , the model training method provided by the embodiments of the present application further comprises the following S540 and S550.

[0150] S540, corelet 1 sends the first group of key-value vectors to corelet 2.

[0151] Corresponding to the process of S540, corelet 2 receives the first group of key-value vectors sent by corelet 1.

[0152] Illustratively, the first group of key-value vectors includes the key-value vectors corresponding to token 2 and token 15.

[0153] S550, corelet 2 takes the plurality of second query vectors, the first group of key-value vectors, and the second group of key-value vectors as inputs of the second processing layer, and performs training of the model.

[0154] Please refer to Figure 6The multiple second query vectors in core 2 include: Q corresponding to token 4, token 5, token 12, and token 13; the first set of key-value vectors includes KV corresponding to token 2 and token 15; and the second set of key-value vectors includes KV corresponding to token 3 and token 14.

[0155] exist Figure 5 In the provided model training method, the model training system uses different cores to train different processing layers in the model, which helps to reduce the load on a single core and thus reduce the training time required for a single core, thereby reducing the total time required for the model training system to train the model and improving the efficiency of model training.

[0156] To improve the accuracy of model training, in Figure 6 and Figure 7 Based on this, the embodiments of this application also provide a feasible example, such as Figure 7 As shown, Figure 3 A flowchart illustrating a model training method provided in this application. Figure 5 The specific implementation of each acceleration chip and the vectors configured in the chip can be found in the description of the foregoing embodiments. S510 to S520 have already been described. Figure 7 The relevant embodiments are described, and will not be repeated here.

[0157] Please see Figure 7 The above-mentioned S530 may include the following S531.

[0158] S531, Core 1 inputs multiple first query vectors, a first set of key-value vectors and a second set of key-value vectors into the first processing layer to obtain multiple updated first query vectors.

[0159] exist Figures 5 to 7 In the middle, the updated first query vectors are black square patterns, which are the updated query vectors corresponding to token 1, token 8, token 9, and token 16, respectively.

[0160] Following S531, the model training method provided in this application embodiment further includes the following S561 and S563.

[0161] S561, the second acceleration chip 122 sends the third set of key value vectors to chip 1 and the fourth set of key value vectors to chip 2.

[0162] Corresponding to the process in S561, chip 1 receives the third set of key value vectors sent by the second acceleration chip 122, and chip 2 receives the fourth set of key value vectors sent by the second acceleration chip 122.

[0163] For example, the corelet 3 sends the third group of key-value vectors to the corelet 1, and the corelet 4 sends the fourth group of key-value vectors to the corelet 2.

[0164] The third group of key-value vectors includes part of the key-value vectors in the third group of key-value vectors in the query statement, such as the KV corresponding to the token 10 and the token 7. The fourth group of key-value vectors includes part of the key-value vectors in the fourth group of key-value vectors in the query statement, such as the KV corresponding to the token 11 and the token 6. In the case of communication between different acceleration chips, different pairs of corelets are used to transmit the key-value vectors between different acceleration chips, thereby reducing the communication amount required to be transmitted by a single pair of corelets through the inter-chip communication link.

[0165] S562, the corelet 2 sends the fourth group of key-value vectors to the corelet 1.

[0166] It is worth noting that the fourth group of key-value vectors sent by the second acceleration chip 122 is transmitted through the inter-chip communication link, and the fourth group of key-value vectors sent by the corelet 2 is transmitted through the inter-corelet communication link.

[0167] S563, the corelet 1 trains the first processing layer according to the updated plurality of first query vectors, the third group of key-value vectors, and the fourth group of key-value vectors.

[0168] In the embodiments of the present application, the key-value vectors configured in different corelets are calculated with the first query vectors stored locally in the corelet 1, thereby avoiding the problem of low accuracy caused by training the first processing layer with only part of the training data by the corelet 1, and facilitating to improve the accuracy of model training. Moreover, the key-value vectors are transmitted based on the characteristic that the transmission bandwidth of the inter-corelet communication link is greater than that of the inter-chip communication link. In the case that the total communication amount between different acceleration chips is unchanged, the communication amount required to be transmitted by a single pair of corelets is halved, and the communication time of the other half of the key-value vectors transmitted through the inter-corelet communication link is reduced. The total communication time required in the model training process is reduced, the time required for model training is reduced, and the efficiency of model training is improved.

[0169] In the above Figures 5 to 7 In the embodiments provided, the execution subject of the model training method is the first acceleration chip 121. The following will be exemplarily described with reference to the content of Figure 8 , taking the first acceleration chip 121 and the second acceleration chip 122 in the model training system 120 jointly performing the model training method as an example, as shown in Figure 8 . Figure 4 The flowchart of a model training method provided by the present application is shown in Figure 8 . The specific implementation of each acceleration chip and the vectors configured in the corelets can be referred to the description of the foregoing embodiments, which will not be described here. Please refer to Figure 8The model training method provided in the embodiments of the present application includes the following (1) to (6).

[0170] (1) Inter-chip communication.

[0171] The corelet 1 sends the key value vector corresponding to the token 1 and the token 16 to the corelet 3, the corelet 2 sends the key value vector corresponding to the token 4 and the token 13 to the corelet 4, the corelet 3 sends the key value vector corresponding to the token 2 and the token 15 to the corelet 1, and the corelet 4 sends the key value vector corresponding to the token 3 and the token 14 to the corelet 2.

[0172] (2) Inter-corelet communication.

[0173] In the first acceleration chip 121, the corelet 1 sends the key value vector corresponding to the token 2 and the token 15 to the corelet 2, and the corelet 2 sends the key value vector corresponding to the token 3 and the token 14 to the corelet 1.

[0174] In the second acceleration chip 122, the corelet 3 sends the key value vector corresponding to the token 1 and the token 16 to the corelet 4, and the corelet 4 sends the key value vector corresponding to the token 4 and the token 13 to the corelet 3.

[0175] (3) Training.

[0176] The corelet 1 calculates the query vector corresponding to the token 1, the token 8, the token 9 and the token 16 with the key value vector corresponding to the token 2, the token 3, the token 14 and the token 15 to obtain a calculation result. The total calculation amount of the corelet 1 is 16, and the effective calculation amount is 8. Figure 8 In the figure, the white background and the block with the mark of “√” represent the effective calculation amount. In addition, Figure 8 In the figure, the black background and the block with the mark of “×” represent the invalid calculation amount.

[0177] The corelet 2 calculates the query vector corresponding to the token 4, the token 5, the token 12 and the token 13 with the key value vector corresponding to the token 2, the token 3, the token 14 and the token 15 to obtain a calculation result. The total calculation amount of the corelet 2 is 16, and the effective calculation amount is 8.

[0178] The core particle 3 calculates the query vector corresponding to token 2, token 7, token 10 and token 15 with the key value vector corresponding to token 1, token 4, token 13 and token 16 to obtain a calculation result. The total calculation amount of the core particle 3 is 16, and the effective calculation amount is 8.

[0179] The core particle 4 calculates the query vector corresponding to token 3, token 6, token 11 and token 14 with the key value vector corresponding to token 3, token 14, token 11 and token 6 to obtain a calculation result. The total calculation amount of the core particle 4 is 16, and the effective calculation amount is 8.

[0180] For the communication process of (1) and (2) in the above Figure 9 , an example is provided as shown in Figure 9 . Figure 9 The communication schematic diagram of the model training process is provided in the present application. Please refer to Figure 8 , the communication of the model training process includes (1) inter-chip communication and (2) inter-die communication.

[0181] (1) Inter-chip communication: transmitting the partial key value vector (D1 part) of the core particle 1 and the partial key value vector (D3 part) of the core particle 3 between the core particle 1 and the core particle 3, and transmitting the partial key value vector (D2 part) of the core particle 2 and the partial key value vector (D4 part) of the core particle 4 between the core particle 2 and the core particle 4.

[0182] (2) Inter-die communication: transmitting D3 part-D4 part between the core particle 1 and the core particle 2, and transmitting D1 part-D2 part between the core particle 3 and the core particle 4.

[0183] In the embodiments of the present application, the characteristics that the SIO bandwidth between the dies is higher than the bandwidth between the chips are utilized to perform pipelining in parallel, which reduces the communication amount between the chips by half and improves the model training performance in a distributed scenario.

[0184] Please continue to refer to Figure 8 , the model training method provided in the embodiments of the present application further includes (4) to (6) as follows.

[0185] (4) Inter-chip communication.

[0186] The core grain 1 sends the key value vector corresponding to the token 9 and the token 8 to the core grain 3, the core grain 2 sends the key value vector corresponding to the token 12 and the token 5 to the core grain 4, the core grain 3 sends the key value vector corresponding to the token 10 and the token 7 to the core grain 1, and the core grain 4 sends the key value vector corresponding to the token 11 and the token 6 to the core grain 2.

[0187] (5) Inter-core communication.

[0188] In the first acceleration chip 121, the core grain 1 sends the key value vector corresponding to the token 10 and the token 7 to the core grain 2, and the core grain 2 sends the key value vector corresponding to the token 11 and the token 6 to the core grain 1.

[0189] In the second acceleration chip 122, the core grain 3 sends the key value vector corresponding to the token 9 and the token 8 to the core grain 4, and the core grain 4 sends the key value vector corresponding to the token 12 and the token 5 to the core grain 3.

[0190] (6) Training.

[0191] The core grain 1 calculates the query vector corresponding to the token 1, the token 8, the token 9 and the token 16 with the key value vector corresponding to the token 6, the token 7, the token 10 and the token 11 to obtain a calculation result. The total calculation amount of the core grain 1 is 16, and the effective calculation amount is 8.

[0192] The core grain 2 calculates the query vector corresponding to the token 4, the token 5, the token 12 and the token 13 with the key value vector corresponding to the token 6, the token 7, the token 10 and the token 11 to obtain a calculation result. The total calculation amount of the core grain 2 is 16, and the effective calculation amount is 8.

[0193] The core grain 3 calculates the query vector corresponding to the token 2, the token 7, the token 10 and the token 15 with the key value vector corresponding to the token 5, the token 8, the token 9 and the token 12 to obtain a calculation result. The total calculation amount of the core grain 3 is 16, and the effective calculation amount is 8.

[0194] The core grain 4 calculates the query vector corresponding to the token 3, the token 6, the token 11 and the token 14 with the key value vector corresponding to the token 5, the token 8, the token 9 and the token 12 to obtain a calculation result. The total calculation amount of the core grain 4 is 16, and the effective calculation amount is 8.

[0195] above Figure 9 and Figure 10 The query vector and key-value vector calculated by different core particles during the model training process are illustrated below. Figure 11 and Figure 10 To illustrate the beneficial effects of the model training system and model training method provided in the embodiments of this application, such as... Figure 10 As shown, Figure 1 Comparison of the effects of different solutions provided for this application Figure 10 .about Figure 2 The description of the model training system 120 in the above can be found in the foregoing. Figure 10 The description of the relevant embodiments is omitted here.

[0196] Please see Figure 10 The bandwidth between different chips belonging to the same acceleration chip is greater than the bandwidth between chips in the model training system 120.

[0197] In conventional technology, for different accelerator chips, after performing one inter-chip communication and completing 8 units of communication, chip 1 needs to perform 10 units of data computation. After the first accelerator chip 121 to which chip 1 belongs performs the next inter-chip communication and completes 8 units of communication, chip 1 performs 6 units of data computation.

[0198] In this embodiment, if chip 1 is to achieve 16 units of data computation, the communication time that chip 1 waits for includes: 8 units of communication between chips and 2 units of communication between chips, which is significantly less than the time required in conventional technology to wait for 16 units of communication between chips. The communication time during model training is significantly reduced, and the data computation time can be masked by the communication time, thus reducing the model training time. Figure 8 In this context, the communication volume of two units in inter-core communication can be, for example, as described above. Figure 11 The key-value vectors corresponding to token 2 and token 15 are shown.

[0199] Depending on the processing layers in the model, some processing layers require longer data computation times. However, the model training system and model training method provided in this application can still achieve good optimization results in such scenarios. Figure 11 As shown, Figure 2 Comparison of the effects of different solutions provided for this application Figure 11 .

[0200] Please see Figure 11In case one, the corelet 1 can start to perform the 8-unit data computation after transmitting 4 units of inter-chip communication and 2 units of intra-chip communication. Since the data computation time of the corelet 1 is short, the corelet 1 can start to perform the 8-unit data computation before the corelet 1 completes three times of 4-unit inter-chip communication. In case one, the computation time of 16 units of data computation is completely covered by the time of inter-chip communication.

[0201] Please refer to Figure 1 In case two, the corelet 1 can start to perform the 8-unit data computation after transmitting 4 units of inter-chip communication and 2 units of intra-chip communication. Since the data computation time of the corelet 1 is long, the corelet 1 can start to perform the second 8-unit data computation after the corelet 1 completes two times of 4-unit inter-chip communication and 2-unit intra-chip communication. In case two, the computation time of 16 units of data computation is partially covered by the time of inter-chip communication.

[0202] It can be known from the above embodiments that, by rearranging the query vectors and key-value vectors stored in different corelets in the model training system, a more efficient computation and communication covering scheme is realized based on the inter-die and inter-chip interconnection architecture of different communication links under the premise of ensuring load balancing, thereby improving the model training performance.

[0203] It can be understood that the above embodiments are described by taking model training as an example, but the technical solutions provided in the present application can also be used in a model use scenario or a model inference scenario. For example, in the model inference scenario, the processing chip 120a receives a query statement, and distributes the query vectors and key-value vectors corresponding to each word in the query statement to different corelets, and outputs an inference result corresponding to the query statement by the model training system 120. In some optional cases, the model obtained by the embodiments of the present application can not only be applied to the model training system 120 described above, but also be applied to a model use scenario or a model inference scenario. Figure 8 The computing device 110 or other devices shown can be used, and the present application is not limited thereto.

[0204] The difference between the model inference scenario and the model training scenario is that, since the model inference scenario does not need to focus on the computation imbalance problem caused by the mask, the query vectors and key-value vectors corresponding to each word in the query statement do not need to be rearranged, and the chip to chip and die to die flow covering scheme (i.e., the above Figure 9 and Figure 12 communication scheme between different corelets) can be directly used to reduce the communication time in the model inference process, improve the communication efficiency, and be beneficial to reducing the time consumption required by the model inference.

[0205] It can be understood that, in order to realize the functions in the above embodiments, the model training system includes the hardware structure and / or software module corresponding to the functions. Those skilled in the art should easily realize that, in combination with the units and method steps of the examples described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0206] The acceleration chip and the model training system provided by the embodiments of the present application can refer to the description of the foregoing embodiments, and will not be repeated here. Next, the processing chip provided by the embodiments of the present application will be introduced in combination with the drawings, Figure 1 The structure schematic diagram of a processing chip provided by the present application. The processing chip 1200 can be used to realize the functions of the processing chip in the above method embodiments, and thus can also realize the beneficial effects possessed by the above method embodiments. In the present embodiment, the processing chip can be a processing chip 120a as shown in Figure 12 , and can also be a processing chip in subsequent embodiments, or a module (such as a chip) applied to a processing chip.

[0207] As shown in Figure 12 , the processing chip 1200 can include a processor 1220. Optionally, the processing chip 1200 can also include a memory 1230 and / or a transceiver 1210. The processor 1220 is coupled with the memory 1230 and the transceiver 1210, which can be connected through a communication bus. The communication bus can include but is not limited to: a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0208] Next, the various constituent components of the processing chip 1200 will be specifically introduced: Figure 12

[0209] ​The processor 1220 is the control center of the processing chip 1200, and can be one processor or a collective term of multiple processing elements. For example, the processor 1220 is one or more CPUs, or is an ASIC, or is one or more integrated circuits configured to implement the embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGA).

[0210] Optionally, the processor 1220 can execute various functions of the processing chip 1200 by running or executing software programs stored in the memory 1230, and calling data stored in the memory 1230. In a specific implementation, as an embodiment, the processor 1220 can include one or more CPUs.

[0211] Optionally, the processing chip 1200 can also include multiple processors. Each of the processors can be a single-CPU or a multi-CPU. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (for example, computer program instructions).

[0212] The memory 1230 is configured to store software programs for performing the vector rearrangement in the embodiments of the present application, and the execution is controlled by the processor 1220. The specific implementation can refer to the vector rearrangement stage in the above method embodiments, and details are not described herein. For example, the memory 1230 can be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage medium or other magnetic storage devices, or any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory 1230 can be integrated with the processor 1220, or can exist independently, and is coupled with the processor 1220 through an interface circuit (not shown) of the processing chip 1200. The embodiments of the present application do not make a specific limitation here. Figure 12

[0213] ​The transceiver 1210 is configured to communicate with other devices. For example, the processing chip 1200 is a client or an application server, and the transceiver 1210 can be configured to communicate with an acceleration chip or another processing chip. For another example, the processing chip 1200 is a multi-core chip, and the transceiver 1210 can be configured to communicate with another multi-core chip.

[0214] Optionally, the transceiver 1210 can include a receiver and a transmitter (not shown separately in the figures). The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function. Optionally, the transceiver 1210 can be integrated with the processor 1220, or can exist independently and be coupled to the processor 1220 through an interface circuit (not shown in the figures) of the processing chip 1200. The embodiments of the present application do not make any specific limitation in this regard. Figure 12 ​ Optionally, the transceiver 1210 can include a receiver and a transmitter (not shown separately in the figures). The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function. Optionally, the transceiver 1210 can be integrated with the processor 1220, or can exist independently and be coupled to the processor 1220 through an interface circuit (not shown in the figures) of the processing chip 1200. The embodiments of the present application do not make any specific limitation in this regard.

[0215] In this embodiment, the transceiver 1210 is configured to obtain a query statement. The processor 1220 is configured to determine, according to a number of words in the query statement and a number of corelets of an acceleration chip in a model training system, a plurality of first words corresponding to a first corelet in a first acceleration chip, the first acceleration chip being any one of the acceleration chips in the model training system, the plurality of first words matching a device serial number of the first acceleration chip. The transceiver 1210 is further configured to send, to the first corelet, a plurality of first query vectors corresponding to the plurality of first words, each first word corresponding to a first query vector. The processing chip 1200 can be configured to implement the functions of the processing chip 120a in the foregoing embodiments and achieve the corresponding beneficial effects, which are not described herein again.

[0216] The method steps in the embodiments can be implemented by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disks, mobile disks, CD-ROMs, or any other forms of storage mediums well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0217] ​In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable apparatus. The computer programs or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer programs or instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired or wireless means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; an optical medium, such as a digital video disc (digital video disc, DVD); and a semiconductor medium, such as a solid state drive (solid state drive, SSD).

[0218] Notably, if the computer program product can be provided to users for use through a cloud service subscription mode, users can choose different subscription levels according to their needs; for example, the computer program product can also provide enterprise-level customization services with professional domain customization, interface personalization and expansion functions according to the needs of users or enterprises. In addition, the computer program product provided by the present application which can implement the above model training method can also be made into value-added services for users, which is not limited by the present application. Alternatively, the computer program product can also be embedded in eDataMate TM or other large language model (large model) tool chain systems, such as operator acceleration library, etc.

[0219] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A model training system, characterized in that, include: First acceleration chip and second acceleration chip; The first acceleration chip includes a first chip and a second chip, wherein a first processing layer of the model is deployed in the first chip; The first core is configured with multiple first query vectors corresponding to the first group of words in the query statement; the first group of words includes multiple first words, wherein each first word corresponds to a first query vector; The second core is configured with multiple second query vectors corresponding to the second group of words in the query statement; the second group of words includes multiple second words, wherein each second word corresponds to a second query vector; The second acceleration chip is configured with: key value vectors corresponding to the third group of words in the query statement and key value vectors corresponding to the fourth group of words; The second acceleration chip is configured to: send a first set of key-value vectors to the first chip and send a second set of key-value vectors to the second chip; the first set of key-value vectors includes a portion of the key-value vectors corresponding to the third set of words; the second set of key-value vectors includes a portion of the key-value vectors corresponding to the third set of words. The second core is used to send the second set of key value vectors to the first core; The first core is used to take the plurality of first query vectors, the first set of key-value vectors and the second set of key-value vectors as input to the first processing layer to perform training of the model.

2. The model training system according to claim 1, characterized in that, The first chip and the second chip are connected via a small input / output (SIO) channel; The first acceleration chip and the second acceleration chip are connected in one or more of the following ways: HCCS interface for aggregated communication bandwidth, GPU interconnect bandwidth interface for high-speed graphics processors, integrated circuit interface, controller area network bus, serial peripheral device interface, queued serial peripheral device interface, full-duplex asynchronous serial interface, and half-duplex differential serial interface.

3. The model training system according to claim 1 or 2, characterized in that, The number of words in the query statement is N, the number of cores in the model training system is T, and the number of first query vectors in the first core is determined based on the ratio between N and T.

4. The model training system according to claim 3, characterized in that, The device number of the first acceleration chip is i; The plurality of first words include: among all the words in the query statement, the word whose local word index matches i, and the word whose local word index matches T-i+1; Alternatively, the plurality of first words may include: among all the words in the query statement, the word whose local word index matches i+1, and the word whose local word index matches Ti.

5. The model training system according to any one of claims 1-4, characterized in that, The first processing layer is the attention layer of the transformer module, and the plurality of first query vectors, the first set of key-value vectors and the second set of key-value vectors are the input data of the transformer module.

6. The model training system according to any one of claims 1-5, characterized in that, The second acceleration chip includes a third chip and a fourth chip; The third core is used to send the first set of key value vectors to the first core; The fourth core is used to send the second set of key value vectors to the second core.

7. The model training system according to claim 6, characterized in that, The second core contains the second processing layer of the model; The first core is also used to send the first set of key-value vectors to the second core; The second core is also used to take the plurality of second query vectors, the first set of key-value vectors, and the second set of key-value vectors as inputs to the second processing layer to perform training of the model.

8. The model training system according to claim 7, characterized in that, The number of first query vectors in the first core is the same as the number of second query vectors in the second core.

9. The model training system according to claim 8, characterized in that, The sum of the global word indices of the plurality of first words is consistent with the sum of the global word indices of the plurality of second words.

10. The model training system according to any one of claims 1-8, characterized in that, The first acceleration chip and the second acceleration chip have the same amount of communication in the same communication phase, and the sum of the global word indices corresponding to the key value vectors transmitted by the first acceleration chip is the same as the sum of the global word indices corresponding to the key value vectors transmitted by the second acceleration chip. The communication phase is the process of transmitting key value vectors between different acceleration chips.

11. The model training system according to any one of claims 1-10, characterized in that, The first core is specifically used to: input the plurality of first query vectors, the first set of key value vectors and the second set of key value vectors into the first processing layer to obtain the updated plurality of first query vectors; The second acceleration chip is further configured to: send a third set of key value vectors to the first chip and send a fourth set of key value vectors to the second chip; the third set of key value vectors includes: a portion of the key value vectors corresponding to the third set of words, and the third set of key value vectors is different from the first set of key value vectors; the fourth set of key value vectors includes: a portion of the key value vectors corresponding to the fourth set of words, and the fourth set of key value vectors is different from the second set of key value vectors; The second core is further configured to: send the fourth set of key-value vectors to the first core; The first core is further used to: train the first processing layer based on the updated multiple first query vectors, the third set of key-value vectors, and the fourth set of key-value vectors.

12. The model training system according to any one of claims 1-11, characterized in that, The first chip is further configured to: obtain a query statement; and determine multiple first words in the query statement corresponding to the first chip based on the number of words in the query statement, the number of chips in the acceleration chip in the model training system, and the device serial number of the first acceleration chip. The first core is further configured to: obtain the plurality of first query vectors based on the plurality of first words.

13. The model training system according to any one of claims 1-11, characterized in that, Also includes: The processing chip is used to: receive a query statement and determine multiple first words in the query statement that correspond to the first chip based on the number of words in the query statement, the number of chips in the acceleration chip in the model training system, and the device serial number of the first acceleration chip; The processing chip is further configured to: send multiple first query vectors corresponding to the multiple first words to the first chip.

14. An accelerator chip, characterized in that, include: First core and second core; The first core is equipped with a first processing layer of the model and multiple first query vectors of the first group of words in the query statement. The first group of words includes multiple first words, wherein each first word corresponds to a first query vector. The first chip is configured to: receive a first set of key-value vectors sent by other acceleration chips; the first set of key-value vectors includes: partial key-value vectors corresponding to the third set of words in the query statement; The second chip is configured to: receive a second set of key-value vectors sent by the other acceleration chips, and send the second set of key-value vectors to the first chip; the second set of key-value vectors includes: a portion of the key-value vectors corresponding to the fourth set of words in the query statement; The first core is further configured to: use the plurality of first query vectors, the first set of key-value vectors and the second set of key-value vectors as input to the first processing layer to perform training of the model.

15. A processing chip, characterized in that, include: A transceiver is used to receive query statements. The processor is configured to: determine multiple first words corresponding to a first chip in a first accelerator chip based on the number of words in the query statement and the number of chips in the accelerator chip in the model training system, wherein the first accelerator chip is any accelerator chip in the model training system and the multiple first words are matched with the device serial number of the first accelerator chip; The transceiver is further configured to: send multiple first query vectors corresponding to the multiple first words to the first chip, wherein each first word corresponds to one first query vector.

16. A model training method, characterized in that, The method is applied to the model training system according to any one of claims 1-13, wherein the first acceleration chip in the model training system includes a first chip and a second chip, and the first chip has a first processing layer of the model deployed therein, and the method includes: The first core obtains multiple first query vectors of the first group of words in the query statement; the first group of words includes multiple first words, wherein each first word corresponds to a first query vector; The first chip receives a first set of key-value vectors from the second acceleration chip in the model training system, and the second chip receives a second set of key-value vectors from the second acceleration chip; the first set of key-value vectors includes: a portion of the key-value vectors corresponding to the third set of words in the query statement, and the second set of key-value vectors includes: a portion of the key-value vectors corresponding to the fourth set of words in the query statement; The first core receives the second set of key-value vectors sent by the second core; The first core takes the plurality of first query vectors, the first set of key-value vectors and the second set of key-value vectors as input to the first processing layer to train the model.

17. The method according to claim 16, characterized in that, The second core contains a second processing layer of the model and multiple second query vectors configured in the second group of words in the query statement; the second group of words includes multiple second words, and one second word corresponds to one second query vector; The method further includes: The second core receives the first set of key-value vectors sent by the first core; The second core takes the plurality of second query vectors, the first set of key-value vectors, and the second set of key-value vectors as input to the second processing layer to train the model.

18. The method according to claim 16 or 17, characterized in that, The first core takes the plurality of first query vectors, the first set of key-value vectors, and the second set of key-value vectors as input to the first processing layer to perform training of the model, including: The first core inputs the plurality of first query vectors, the first set of key value vectors and the second set of key value vectors to the first processing layer to obtain the updated plurality of first query vectors; The first chip receives a third set of key-value vectors sent by the second acceleration chip, and the second chip receives a fourth set of key-value vectors sent by the second acceleration chip; the third set of key-value vectors includes: partial key-value vectors corresponding to the third set of words, and the third set of key-value vectors is different from the first set of key-value vectors; the fourth set of key-value vectors includes: partial key-value vectors corresponding to the fourth set of words, and the fourth set of key-value vectors is different from the second set of key-value vectors. The first core receives the fourth set of key-value vectors sent by the second core, and inputs the updated multiple first query vectors, the third set of key-value vectors, and the fourth set of key-value vectors into the first processing layer to perform training of the model.

19. A computer program product, characterized in that, When the computer program product is run in a computing device, the computing device performs the method of any one of claims 16-18.

Images