Ai server, decoding node, data distribution, and data aggregation method

Abstract

The present disclosure provides an AI server, a decoding node, a data distribution method and a data aggregation method, and relates to the technical field of artificial intelligence such as chips, large models and the like. The AI server comprises a CPU, a first AI chip, a second AI chip, a communication network card and a switching chip. The first AI chip is configured to perform attention mechanism calculation to obtain distributed data. The second AI chip is configured to perform a forward network calculation of an expert to obtain aggregated data. The switching chip is configured to connect the CPU, the first AI chip and the communication network card. The communication network card is configured to realize communication between the first AI chips in different AI servers. The decoding node comprises a plurality of AI servers and a switching network. The switching network is configured to realize communication between the plurality of AI servers. The present disclosure also provides a data distribution method and a data aggregation method based on the decoding node. The present disclosure can mix and deploy heterogeneous first AI chips and second AI chips in the AI server, and achieve the purpose of multiplexing expert parallel communication mode.

Description

Technical Field

[0001] This disclosure relates to the field of internet technology, and more particularly to the field of artificial intelligence technology such as chips and large-scale models. It provides an AI server, a decoding node, a data distribution and data aggregation method, an electronic device, and a readable storage medium. Background Technology

[0002] In existing technologies, in the Decode node of MoE (Mixture of Experts), the attention mechanism module and the feedforward network module are deployed independently. After completing the attention mechanism calculation and the selection of the routed expert, the attention mechanism module sends the dispatch data to the feedforward network module. After completing the feedforward network card calculation, the feedforward network module returns the aggregated data to the attention mechanism module.

[0003] However, in the existing technology, the attention mechanism module and the feedforward network module of the Decode node usually correspond to the same architecture, which has great limitations. If the two are heterogeneous, the communication between the attention mechanism module and the feedforward network module will have problems, and the expert parallel communication method cannot be reused, resulting in low communication efficiency. Summary of the Invention

[0004] According to a first aspect of this disclosure, an AI server is provided, comprising: a CPU, a first AI chip, a second AI chip, a communication network interface card (NIC), and a switching chip; wherein the first AI chip is used to perform attention mechanism calculations to generate distributed data; the second AI chip is used to perform expert feedforward network calculations to generate aggregated data; the switching chip is used to connect the CPU, the first AI chip, and the communication NIC; and the communication NIC is used to enable communication between the first AI chips in different AI servers.

[0005] According to a second aspect of this disclosure, a decoding node for a hybrid expert model is provided, comprising: a plurality of AI servers and a switching network; wherein the switching network is used to enable communication between the plurality of AI servers; and the AI ​​servers are AI servers corresponding to the first aspect.

[0006] According to a third aspect of this disclosure, a data distribution method is provided, executed by a first AI server in a decoding node, comprising: acquiring distribution data generated by a first AI chip; determining a target expert based on the distribution data; and sending the distribution data to a second AI server corresponding to the target expert through a switching network of the decoding node, so that the second AI server generates aggregated data based on the distribution data.

[0007] According to a fourth aspect of this disclosure, a data distribution method is provided, executed by a second AI server in a decoding node, comprising: receiving distribution data sent by a first AI server through a switching network of the decoding node; writing the distribution data into a first AI chip; reading the distribution data from the first AI chip using a CPU; and then writing the distribution data into a second AI chip using the CPU, so that the second AI chip generates aggregated data based on the distribution data.

[0008] According to a fifth aspect of this disclosure, a data aggregation method is provided, executed by a second AI server in a decoding node, comprising: reading aggregated data from a second AI chip using a CPU, and then writing the aggregated data into a first AI chip using the CPU; and sending the aggregated data in the first AI chip to the first AI server through the switching network of the decoding node, so that the first AI server can obtain final generated data based on the aggregated data.

[0009] According to a sixth aspect of this disclosure, a data aggregation method is provided, executed by a first AI server in a decoding node, comprising: receiving aggregated data sent by a second AI server through a switching network of the decoding node, writing the aggregated data into a first AI chip, so that the first AI chip obtains final generated data based on the aggregated data.

[0010] According to a seventh aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0011] According to an eighth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0012] According to a ninth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0016] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0017] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0018] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0019] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0020] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure;

[0021] Figure 7 This is a schematic diagram according to the seventh embodiment of the present disclosure;

[0022] Figure 8 This is a schematic diagram according to the eighth embodiment of the present disclosure;

[0023] Figure 9 This is a block diagram of an electronic device used to implement the data distribution or data aggregation method of the embodiments of this disclosure. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and mechanisms are omitted in the following description.

[0025] Figure 1 This is a schematic diagram according to the first embodiment of the present disclosure. Figure 1 A structural diagram of the AI ​​server in this embodiment is shown.

[0026] The AI ​​server in this embodiment includes a CPU, a first AI chip, a second AI chip, a communication network card, and a switching chip.

[0027] In this embodiment, the first AI chip and the second AI chip in the AI ​​server can be heterogeneous chips (different chips from the same manufacturer or chips from different manufacturers), thereby achieving the purpose of hybrid deployment of first AI chips and second AI chips with different architectures in the AI ​​server. This can reduce the limitations of the AI ​​server on chip architecture, thereby improving the construction efficiency of the AI ​​server and reducing the construction cost of the AI ​​server.

[0028] In this embodiment, the number of CPUs can be one or more; wherein, if the AI ​​server includes one CPU, the AI ​​server is a single-processor AI server, and if the AI ​​server includes two or more CPUs, the AI ​​server is a multi-processor AI server.

[0029] In this embodiment, the CPU is used to initiate a device-to-host (D2H) transmission to the first AI chip to read distribution data from the first AI chip; and to initiate a host-to-device (H2D) transmission to the second AI chip to write the distribution data to the second AI chip.

[0030] In other words, in this embodiment, the CPU reads the distribution data from the first AI chip and writes the distribution data into the second AI chip, thereby enabling the second AI chip to obtain the distribution data sent by other AI servers. Then, the second AI chip performs expert feedforward network calculations based on the distribution data to generate aggregated data corresponding to the distribution data.

[0031] In this embodiment, the CPU is also used to initiate D2H transmission to the second AI chip to read aggregated data from the second AI chip; and to initiate H2D transmission to the first AI chip to write aggregated data to the first AI chip.

[0032] In other words, in this embodiment, the CPU reads the aggregated data generated by the second AI chip and writes the aggregated data into the first AI chip, thereby enabling the first AI chip to obtain the aggregated data generated by the second AI chip. Then, the first AI chip sends the aggregated data to other AI servers (i.e., AI servers that send distribution data to the current AI server), so that other AI servers can obtain the final generated data based on the aggregated data.

[0033] In this embodiment, the first AI chip is an attention mechanism chip, used to perform attention mechanism calculations to generate and distribute data; the number of first AI chips can be one or more.

[0034] In this embodiment, the distribution data generated by the first AI chip is used to send to other AI servers (i.e., the AI ​​server corresponding to the target expert) for expert feedforward network calculation.

[0035] In addition, the first AI chip in this embodiment can also be used to obtain the final generated data based on the aggregated data sent by other AI servers. The aggregated data sent by other AI servers is obtained from the distribution data generated by the first AI chip.

[0036] In this embodiment, the first AI chip is connected to the CPU and the communication network card via a switch chip.

[0037] In other words, after generating distribution data, the first AI chip in this embodiment can transmit the distribution data to the communication network card through the switching chip, and then send the distribution data to other AI servers (specifically, the first AI chip of other AI servers) through the communication network card.

[0038] Similarly, the first AI chip in this embodiment can also receive aggregated data sent by other AI servers through a communication network card and a switching chip.

[0039] In this embodiment, the second AI chip is a feedforward network (FFN) chip, which is used to perform expert feedforward network calculations to generate aggregated data.

[0040] In this embodiment, the number of second AI chips can be one or more; each second AI chip in the AI ​​server corresponds to an expert in the MoE (Mixture of Experts) model, and different second AI chips are used to perform feedforward network calculations for different experts.

[0041] In other words, the second AI chip in this embodiment is used to perform expert feedforward network calculations based on the distribution data sent by other AI servers, thereby generating aggregated data corresponding to the received distribution data.

[0042] In this embodiment, the second AI chip is connected to the CPU via PCIe (Peripheral Component Interconnectexpress, a high-speed serial expansion bus standard), enabling data to be transferred between the CPU and the second AI chip. For example, the CPU can write distributed data to the second AI chip or read aggregated data from the second AI chip.

[0043] In this embodiment, the communication network card is used to enable communication between the first AI chips in different AI servers. It is connected to the CPU and the first AI chip through a switching chip.

[0044] In this embodiment, the communication network card can send the distribution data generated by the first AI chip in the current AI server to the first AI chip in other AI servers, and can also send the aggregated data written to the first AI chip in the current AI server to the first AI chip in other AI servers.

[0045] In addition, the communication network card in this embodiment can transmit distribution data sent by other AI servers to the first AI chip, and can also transmit aggregation data sent by other AI servers to the first AI chip.

[0046] In this embodiment, the communication network card can be a scale-out network card, and the number of communication network cards can be one or more.

[0047] It is understandable that when multiple first AI chips are connected to the same switching chip and communicate through the same communication network card, they will share the bandwidth of that communication network card.

[0048] In this embodiment, the switching chip is used to connect the CPU, the first AI chip, and the communication network card, so that data can be transmitted between the CPU, the first AI chip, and the communication network card through the switching chip.

[0049] In this embodiment, the switching chip can be a PCIe Switch chip. There can be one or more switching chips, and each switching chip is connected to the CPU through one or more ports.

[0050] Therefore, the AI ​​server in this embodiment is not limited by the architecture of the first AI chip and the second AI chip when it is deployed. Moreover, by switching chips and communication network cards, it achieves the purpose of communication between different AI servers through expert parallel communication, which can improve communication efficiency.

[0051] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 The diagram shows the structure of the decoding node for the hybrid expert model in this embodiment.

[0052] The decoding node for hybrid expert models in this embodiment includes multiple AI servers and a switching network.

[0053] The AI ​​server in this embodiment includes a CPU, a first AI chip, a second AI chip, a communication network card, and a switching chip.

[0054] The switching network in this embodiment is used to enable communication between different AI servers; wherein, the switching network can be a scale-out switching network.

[0055] In this embodiment, the number of AI servers included in the decoding node can be flexibly set according to actual needs.

[0056] In this embodiment, the AI ​​server used for distributing data in the decoding node can be designated as the first AI server, and the AI ​​server receiving the distributed data can be designated as the second AI server. In this embodiment, there can be multiple second AI servers.

[0057] In this embodiment, the decoding node first generates distribution data (e.g., intermediate activation results) from the first AI chip in the first AI server; then, it performs expert routing based on the generated distribution data to determine the experts that need to be activated; next, it distributes the distribution data to the second AI server corresponding to the activated expert through a switching network; the second AI chip in the second AI server generates aggregated data based on the distribution data and returns the aggregated data to the first AI server through the switching network; finally, the first AI chip in the first AI server obtains the final generated data based on the returned aggregated data, which can be a token.

[0058] Therefore, in this embodiment, the construction of decoding nodes based on AI servers and switching networks with hybrid deployment of the first AI chip and the second AI chip, and the data distribution and data aggregation completed by the corresponding AI server in the decoding node, can improve the communication efficiency between different AI servers in the decoding node, thereby improving the decoding efficiency of the decoding node.

[0059] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure. Figure 3 This diagram illustrates the decoding node in this embodiment during data distribution (Dispatch).

[0060] In this embodiment, the first AI server is AI server 1, and the second AI server is AI server 2. AI server 1 is the source AI server for data distribution, and AI server 2 is the destination AI server for data distribution. The second AI server in this embodiment is the AI ​​server corresponding to the activation expert. There can be multiple second AI servers, but this embodiment uses one as an example for explanation.

[0061] Specifically, after generating the distribution data, the first AI chip in AI server 1 first transmits the distribution data to the communication network card in AI server 1 through the switching chip in AI server 1, and then the communication network card sends the distribution data to AI server 2 through the switching network.

[0062] After receiving the distribution data sent by AI Server 1 through the switching network, AI Server 2 first transmits the distribution data to the first AI chip in AI Server 2 via the communication network card and switching chip. Then, the CPU of AI Server 2 initiates a D2H (Device to Host) transmission to the first AI chip to read the distribution data from the first AI chip. Finally, the CPU of AI Server 2 initiates an H2D (Host to Device) transmission to the second AI chip to write the distribution data to the second AI chip. The second AI chip then performs corresponding expert feedforward network calculations on the distribution data based on the written data to generate aggregated data.

[0063] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 This diagram illustrates the decoding node in this embodiment performing data aggregation.

[0064] In this embodiment, the first AI server is AI server 1, and the second AI server is AI server 2. AI server 2 is the source AI server for data aggregation, and AI server 1 is the destination AI server for data aggregation. The second AI server in this embodiment is the AI ​​server corresponding to the activated expert.

[0065] Specifically, after the second AI chip of AI server 2 generates aggregated data, the CPU of AI server 2 first initiates D2H transmission to the second AI chip to read the aggregated data from the second AI chip. Then, the CPU of AI server 2 initiates H2D transmission to the first AI chip to write the aggregated data into the first AI chip. Finally, the aggregated data in the first AI chip is transmitted to the communication network card through the switching chip, and then the communication network card sends the aggregated data to AI server 1 through the switching network.

[0066] After receiving the aggregated data sent by AI Server 2 through the switching network, AI Server 1 transmits the aggregated data to the first AI chip of AI Server 1 through the communication network card and the switching chip. The first AI chip then obtains the final generated data based on the aggregated data.

[0067] Figure 5 This is a schematic diagram according to the fifth embodiment of this disclosure. (See diagram below.) Figure 5 As shown in the figure, this embodiment illustrates a data distribution method executed by the first AI server in the decoding node, specifically including the following steps:

[0068] S501. Obtain the distribution data generated by the first AI chip, and determine the target expert based on the distribution data;

[0069] S502. The distributed data is sent to the second AI server corresponding to the target expert through the exchange network of the decoding node, so that the second AI server can generate aggregated data based on the distributed data.

[0070] The distribution data obtained in S501 of this embodiment is the calculation result obtained by the first AI chip through the attention mechanism calculation of the input data.

[0071] In this embodiment, different experts correspond to different second AI servers, or more specifically, to the second AI chips within the second AI servers.

[0072] In this embodiment, when executing S502, the AI ​​server corresponding to the target expert can be determined as the second AI server according to the preset correspondence. Then, the distribution data generated by the first AI chip in the first AI server is sent to the second AI server through the switching network of the switching chip, communication network card and decoding node.

[0073] Figure 6 This is a schematic diagram according to the sixth embodiment of this disclosure. (See diagram below.) Figure 6 As shown in the figure, this embodiment illustrates a data distribution method executed by a second AI server in the decoding node, specifically including the following steps:

[0074] S601. Receive distribution data sent by the first AI server through the switching network of the decoding node, and write the distribution data into the first AI chip;

[0075] S602. After using the CPU to read the distribution data from the first AI chip, the CPU then writes the distribution data into the second AI chip, so that the second AI chip generates aggregated data based on the distribution data.

[0076] In this embodiment, when S601 is executed, the second AI server completes the transmission of distribution data with the switching chip through the communication network card, thereby writing the distribution data sent by the first AI server through the switching network into the first AI chip.

[0077] In this embodiment, when S602 is executed, the CPU initiates a D2H transmission to the first AI chip to read the distributed data from the first AI chip, and then the CPU initiates an H2D transmission to the second AI chip to write the distributed data to the second AI chip.

[0078] In this embodiment, after executing S602 to write the distributed data into the second AI chip, the second AI chip can perform feedforward network calculations for the corresponding experts based on the distributed data, thereby generating aggregated data.

[0079] Figure 7This is a schematic diagram according to the seventh embodiment of this disclosure. (See diagram below.) Figure 7 As shown in the figure, this embodiment illustrates a data aggregation method executed by a second AI server in the decoding node, specifically including the following steps:

[0080] S701. After using the CPU to read aggregated data from the second AI chip, the CPU then writes the aggregated data into the first AI chip.

[0081] S702. The aggregated data in the first AI chip is sent to the first AI server through the switching network of the decoding node, so that the first AI server can obtain the final generated data based on the aggregated data.

[0082] In this embodiment, when S701 is executed, the CPU initiates a D2H transmission to the second AI chip to perform the operation of reading aggregated data from the second AI chip, and then the CPU initiates an H2D transmission to the first AI chip to perform the operation of writing aggregated data to the first AI chip.

[0083] In this embodiment, when S702 is executed, the second AI server sends the aggregated data to the first AI server through the switching network of the first switching chip, the communication network card and the decoding node, so that the first AI chip in the first AI server obtains the final generated data based on the aggregated data.

[0084] Figure 8 This is a schematic diagram according to the eighth embodiment of this disclosure. (See diagram below.) Figure 8 As shown in the figure, this embodiment illustrates a data aggregation method executed by the first AI server in the decoding node, specifically including the following steps:

[0085] S801: Receive aggregated data sent by the second AI server through the switching network of the decoding node, write the aggregated data into the first AI chip, so that the first AI chip can obtain the final generated data based on the aggregated data.

[0086] In this embodiment, when S801 is executed, after the first AI server receives the aggregated data sent by the second AI server through the switching network, it completes the transmission of the aggregated data with the switching chip through the communication network card, writes the aggregated data into the first AI chip, so that the first AI chip obtains the final generated data based on the written aggregated data.

[0087] The acquisition, storage, and application of various types of information, such as user personal information, involved in the technical solutions disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0088] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0089] like Figure 9 The diagram shown is a block diagram of an electronic device for a data distribution or data aggregation method according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0090] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0091] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0092] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as data distribution or data aggregation methods. For example, in some embodiments, the data distribution or data aggregation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908.

[0093] In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by computing unit 901, one or more steps of the data distribution or data aggregation method described above may be performed. Alternatively, in other embodiments, computing unit 901 may be configured to perform the data distribution or data aggregation method by any other suitable means (e.g., by means of firmware).

[0094] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data distribution or data aggregation device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0096] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0098] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0099] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0100] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0101] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An AI server, comprising, CPU, first AI chip, second AI chip, communication network card and switching chip; in, The first AI chip is used to perform attention mechanism calculations to generate and distribute data; The second AI chip is used to perform expert feedforward network calculations to generate aggregated data; The switching chip is used to connect the CPU, the first AI chip, and the communication network card; The communication network card is used to enable communication between the first AI chips in different AI servers.

2. The AI ​​server according to claim 1, wherein, The CPU is used for: Initiate a device-to-host (D2H) transmission to the first AI chip to read distribution data sent by other AI servers from the first AI chip; and Initiate a host-to-device (H2D) transmission to the second AI chip to write the distributed data into the second AI chip.

3. The AI ​​server according to claim 1, wherein, The CPU is also used for: Initiate a D2H transmission to the second AI chip to read the aggregated data from the second AI chip; and An H2D transmission is initiated to the first AI chip to write the aggregated data into the first AI chip.

4. The AI ​​server according to claim 1, wherein, The first AI chip is also used to obtain the final generated data based on the aggregated data sent by other AI servers.

5. A decoding node for a hybrid expert model, comprising: Multiple AI servers and switching networks; in, The switching network is used to enable communication between the plurality of AI servers; the AI ​​server is the AI ​​server according to any one of claims 1-4.

6. The decoding node according to claim 5, wherein, The decoding node includes a first AI server and a second AI server; The first AI server is used to send distribution data to the second AI server through the exchange network; The second AI server is used to receive the distributed data through the exchange network and generate aggregated data based on the distributed data.

7. The decoding node according to claim 6, wherein, The second AI server is also configured to send the aggregated data to the first AI server via the exchange network; The first AI server is also configured to receive the aggregated data through the exchange network and obtain the final generated data based on the aggregated data.

8. A data distribution method, executed by a first AI server in a decoding node, comprising: Obtain the distribution data generated by the first AI chip, and determine the target expert based on the distribution data; The distributed data is sent to the second AI server corresponding to the target expert through the exchange network of the decoding node, so that the second AI server can generate aggregated data based on the distributed data; in, The decoding node is the decoding node described in any one of claims 5-7.

9. A data distribution method, executed by a second AI server in a decoding node, comprising: Receive distribution data sent by the first AI server through the switching network of the decoding node, and write the distribution data into the first AI chip; After the CPU reads the distribution data from the first AI chip, it writes the distribution data to the second AI chip, so that the second AI chip can generate aggregated data based on the distribution data. in, The decoding node is the decoding node described in any one of claims 5-7.

10. A data aggregation method, executed by a second AI server in a decoding node, comprising: After using the CPU to read aggregated data from the second AI chip, the CPU then writes the aggregated data into the first AI chip. The aggregated data in the first AI chip is sent to the first AI server through the switching network of the decoding node, so that the first AI server can obtain the final generated data based on the aggregated data; in, The decoding node is the decoding node described in any one of claims 5-7.

11. A data aggregation method, executed by a first AI server in a decoding node, comprising: The system receives aggregated data sent by the second AI server through the switching network of the decoding node, writes the aggregated data into the first AI chip, and then the first AI chip obtains the final generated data based on the aggregated data. in, The decoding node is the decoding node described in any one of claims 5-7.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 8-11.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 8-11.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 8-11.

Images