Ensemble communication method, system and equipment based on hybrid expert model

Abstract

The invention provides an aggregate communication method, system and device based on a hybrid expert model, and relates to the technical field of communication, in particular to the technical fields of chip processors, computing power clusters, aggregate communication operation, large models and the like. The set communication method is applied to a communication receiving end and comprises the steps that a data writing instruction is received, and the data writing instruction comprises to-be-processed data and first address information; accessing a virtual expert address in a pre-created virtual address space according to the first address information; based on the size of the to-be-processed data, applying for a corresponding actual physical space for the virtual expert address; and writing the to-be-processed data into the actual physical space. According to the set communication method provided by the invention, set communication can be realized without memory information synchronization.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technologies, specifically to technical fields such as chip processors, computing power clusters, collective communication operations, large models, etc., and particularly relates to a collective communication method, system, device, electronic device, computer-readable storage medium, and computer program product based on a mixture-of-experts model. Background Art

[0002] In large-scale distributed training, the mixture-of-experts (MoE) model is a model architecture that processes different tasks through multiple "expert" networks. In the MoE scenario, many-to-many variable-count communication (alltoallv) is required between multiple processors participating in training. Summary of the Invention

[0003] Embodiments of the present disclosure propose a collective communication method, system, electronic device, computer-readable storage medium, and computer program product based on a mixture-of-experts model.

[0004] In a first aspect, embodiments of the present disclosure propose a collective communication method based on a mixture-of-experts model, including: receiving a data writing instruction, where the data writing instruction includes data to be processed and first address information; accessing a virtual expert address in a pre-created virtual address space according to the first address information; applying for a corresponding actual physical space for the virtual expert address based on the size of the data to be processed; and writing the data to be processed into the actual physical space.

[0005] In a second aspect, embodiments of the present disclosure propose a collective communication system based on a mixture-of-experts model, including: a plurality of processors and a switch, where the switch is configured to receive an instruction, and the instruction includes data to be processed, address information, and data routing information; and multicast the instruction group to corresponding processors based on the data routing information; the processor is configured to, in response to receiving the instruction as a data writing instruction, access a virtual expert address in a pre-created virtual address space according to the address information; apply for a corresponding actual physical space for the virtual expert address based on the size of the data to be processed; and write the data to be processed into the actual physical space.

[0006] In a third aspect, embodiments of the present disclosure propose a collective communication device based on a mixture-of-experts model, including: a receiving unit configured to receive a data writing instruction, where the data writing instruction includes data to be processed and first address information; an accessing unit configured to access a virtual expert address in a pre-created virtual address space according to the first address information; an applying unit configured to apply for a corresponding actual physical space for the virtual expert address based on the size of the data to be processed; and a writing unit configured to write the data to be processed into the actual physical space.

[0007] Fourthly, embodiments of the present disclosure provide an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to implement the collective communication method described in any implementation manner of the first aspect.

[0008] Fifthly, embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the collective communication method described in any implementation manner of the first aspect when executed.

[0009] Sixthly, embodiments of the present disclosure provide a computer program product including a computer program, which can implement the collective communication method described in any implementation manner of the first aspect when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objectives, and advantages of the present disclosure will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0011] Figure 1 FIG. shows an exemplary system architecture to which the collective communication method, system, electronic device, and computer-readable storage medium of the present disclosure can be applied;

[0012] Figure 2 FIG. is a flowchart of a collective communication method based on a mixture of experts model provided by an embodiment of the present disclosure;

[0013] Figure 3 FIG. is a schematic diagram of a virtual numbered address space and a virtual expert address space in collective communication based on a mixture of experts model provided by an embodiment of the present disclosure;

[0014] Figure 4 FIG. is a flowchart of another collective communication method based on a mixture of experts model provided by an embodiment of the present disclosure;

[0015] Figure 5 FIG. is a schematic diagram of data multicast in the Dispatch stage provided by an embodiment of the present disclosure;

[0016] Figure 6 is related to Figure 5 FIG. is a schematic diagram of the first routing information carried by a data write instruction;

[0017] Figure 7 FIG. is a schematic diagram of data reduction in the Combine stage provided by an embodiment of the present disclosure;

[0018] Figure 8 Schematic diagram of second routing information carried by a data read instruction related to Figure 7 ;

[0019] Figure 9 Schematic diagram of a collective communication system based on a mixture of experts model provided by the present disclosure;

[0020] Figure 10 Structural diagram of a physical memory allocation device provided by an embodiment of the present disclosure;

[0021] Figure 11 Structural diagram of another physical memory allocation device provided by an embodiment of the present disclosure;

[0022] Figure 12 Structural block diagram of a collective communication device based on a mixture of experts model provided by the present disclosure;

[0023] Figure 13 Schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0024] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted below. It should be noted that, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0025] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0026] An embodiment of the present disclosure provides a processing procedure for alltoallv communication in a MoE scenario. Taking communication between 4 GPUs as an example, Rank is a logical number for GPUs, used to identify different GPUs. For example, Rank0 is used to identify the GPU numbered 0, Rank1 is used to identify the GPU numbered 1, and so on. Taking the case where there is only one expert model in each Rank and the communication initiator communicates with two communication receivers each time as an example for illustration. The alltoallv process of MoE is divided into two stages, the dispatch stage (Dispatch stage) and the combine stage (Combine stage).

[0027] Specifically, in the Dispatch stage, first, based on attention calculation, addition operation, and normalization operation, the input data (Token) is calculated. Among them, the core of the attention calculation is to calculate the attention weights of the input data. Common attention mechanisms include dot-product attention and multi-head attention; the addition operation is used to add the output of the attention layer to the input data; the normalization operation is used to normalize the result of the addition operation.

[0028] Then, the Token is used as the input and sent to the corresponding expert model according to the routing graph output by the gating network, and different expert models are used to complete the feed-forward neural network calculation. Among them, the routing graph is used to mark the identifier of the target expert model for data transmission. The routing graph can be specifically represented in the form of a list. The rows of the list are the identifiers of the expert models, and the columns are the identifiers of the Tokens. From the perspective of Rank0, Rank0 needs to send the Token to the expert models on other Ranks. For all Rank nodes, the Token needs to be sent to other Ranks according to the routing graph.

[0029] In the Combine stage, the process of Combine is actually the reverse process of Dispatch. After all expert models complete the feed-forward neural network calculation, the Token needs to be returned to the original Rank, and based on the preset weight values of each Token, the weight calculation of all Tokens is completed. From the perspective of Rank0, Rank0 needs to collect all the calculated Tokens from the expert models on other Ranks and complete the weight calculation of the Tokens based on the weight values.

[0030] In the alltoallv data transmission process in the above MoE process, since the process of sending data needs to perceive the specific location where the data needs to be placed at the remote end, it is inevitable to complete some memory information synchronization, and this process is during inference or training, which is very time-consuming. At the same time, the sending end needs to inform the receiving end how much memory is required, and then the receiving end needs to allocate the memory space and synchronize the memory information to the sending end. After that, the sending end sends the data to the remote end based on the remote memory address information.

[0031] Based on this, the present disclosure further provides a collective communication method, system, electronic device, computer-readable storage medium, and computer program product based on a hybrid expert model that do not require synchronization of address information.

[0032] Figure 1 Exemplary system architecture 100 shows an example of an embodiment in which the collective communication method, system, electronic device, and computer-readable storage medium of the present disclosure can be applied.

[0033] As Figure 1As shown, the system architecture 100 may include a server 110, a network 120, and a computing power cluster 130. Among them, the computing power cluster 130 includes multiple graphics processors, such as 131, 132, 133, 134, and 135 in the form of graphics cards (not only 5, Figure 1 only a few are taken as examples here), and the graphics cards 131, 132, 133, 134, and 135 contain multiple graphics processing chips (i.e., GPUs), and multiple graphics processing chips can communicate with each other in the MOE scenario. The network 120 is used to provide a medium for the communication link between the server 110 and the computing power cluster 130. The network 120 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0034] Users can use the terminal device to interact with the server 110 through the network 120 to receive or send messages, etc. Various applications for implementing information communication between the two can be installed on the server 110, the computing power cluster 130, and the terminal device to enable communication between multiple image processing chips.

[0035] The terminal device and the server 110 can be either hardware or software. When the terminal device is hardware, it can be various electronic devices with a display screen, including but not limited to smartphones, tablets, laptop computers, and desktop computers, etc.; when the terminal device is software, it can be installed in the above-listed electronic devices, and it can be implemented as multiple software or software modules, or it can be implemented as a single software or software module, which is not specifically limited here. When the server 110 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or it can be implemented as a single server; when the server 110 is software, it can be implemented as multiple software or software modules, or it can be implemented as a single software or software module, which is not specifically limited here. The graphics processors that make up the computing power cluster 130 usually appear as hardware, but in special scenarios (such as simulation scenarios), they can also appear as software or software running products, which is not specifically limited here.

[0036] The server 110 can provide various services through various built-in applications. For example, it can send communication instructions to enable multiple GPUs in the computing power cluster 130 to communicate with each other.

[0037] Furthermore, after multiple GPUs in the computing power cluster 130 complete communication, they can also feedback to the server 110.

[0038] It should be understood that Figure 1 the numbers of the server, network, computing power cluster, and graphics cards are only illustrative. According to the implementation needs, there can be any number of servers, networks, computing power clusters, and graphics cards.

[0039] Please refer toFigure 2 , Figure 2 is a flowchart of a collective communication method based on a mixture of experts model provided by an embodiment of the present disclosure. This method is applied to a communication receiving end, and process 200 includes the following steps:

[0040] Step 201: Receive a data writing instruction.

[0041] In collective communication, each execution entity that communicates with each other can either act as a communication sender to initiate a communication request or act as a communication receiver to process the communication according to the received communication request. This step is intended to be processed by the execution entity of the collective communication method based on the mixture of experts model as the communication receiving end. Among them, the execution entity can be Figure 2 one or more GPUs in the computing power cluster 130 shown, or other electronic devices capable of receiving and processing data. The present disclosure places no restrictions on this.

[0042] Specifically, when a communication requester needs to communicate with other execution entities, the communication requester calls a communication instruction and sends the communication instruction to the communication requester. Among them, the communication instruction can be a data writing instruction or a data reading instruction, or other communication instructions. The present disclosure does not limit this.

[0043] Among them, the instruction form of the data writing instruction can be inc_v.store, and the instruction form of the data reading instruction can be inc_v.load_reduce.

[0044] In inc_v.store(input_buffer, inc_group_buffer_va, length, rank_info, expert_info), input_buffer represents the source address of the data to be processed, inc_group_buffer_va represents the target address of the data to be processed, length represents the length of the data to be processed, rank_info represents the rank ID participating in the inc_v store operation within the inc group; expert_info represents the ExpertID that the token input data needs to reach. That is, inc_v.store(input_buffer, inc_group_buffer_va, length, rank_info, expert_info) means reading data of size length from the local input buffer and writing the data to the corresponding VA of the INC group based on rank_info and expert_info.

[0045] In inc_v.load_reduce(inc_group_buffer_va, output_buffer, length, rank_info, expert_info, weight_info), inc_group_buffer_va represents the source address of the data to be processed, output_buffer represents the destination address of the data to be processed, length represents the length of the data to be processed, and weight_info represents the weight information of the token corresponding to the expert. That is, inc_v.load_reduce(inc_group_buffer_va, output_buffer, length) means reading data of size length from the VA corresponding to the INC group and writing it to the local output buffer based on rank_info, expert_info, and weight_info.

[0046] It should be noted that the above rank_info and expert_info are just one form, and other methods can be used for substitution, such as bitmap, etc.

[0047] In some alternative implementation manners of this embodiment, the data write instruction includes the data to be processed and the first address information. Among them, the data to be processed refers to the data to be written in the data write instruction, that is, the input data (Token) of the communication request end. The data to be processed can be parameters, feature vectors, or other data that need to be transmitted during the collective communication process in the mixture of experts model; the first address information refers to the address information corresponding to the destination address where the data to be processed is written.

[0048] In some alternative implementation manners of this embodiment, the data write instruction further includes the first routing information.

[0049] Specifically, the first routing information is used to identify the communication receiving end that sends the data to be processed. Among them, the first routing information is output by the gating network in the mixture of experts model according to the data to be processed. The communication request end calls the data write instruction and sends the data write instruction to the switch. The switch sends the data write instruction to the communication receiving end according to the first routing information in the data write instruction. Among them, the communication receiving end can be one or multiple, which is specifically determined according to the first routing information, and the present disclosure does not limit this.

[0050] It should be noted that after the communication instruction is transmitted to the C2C engine through the Network on Chip (NOC), the C2C engine converts the instruction into the form of a data message or a packet and transmits it in the collective communication network. The communication instructions (data write instructions or data read instructions) mentioned in all the following embodiments of the present disclosure refer to the data messages corresponding to the communication instructions.

[0051] Step 202: Access the virtual expert address in the pre-created virtual address space according to the first address information.

[0052] Based on step 201, this step aims to perform a data write operation by the communication receiver according to the received data write instruction.

[0053] In the embodiments of the present disclosure, each execution entity in the communication set has pre-created its own virtual address space, which is used to store the data of each expert model in the hybrid expert model. Among them, there is one or more expert models in each execution entity in the communication set. The virtual expert address in the virtual address space corresponds to the expert model in the execution entity. The virtual expert address is a certain block of addresses in the virtual address space of the execution entity, and the virtual expert address is the address segment allocated to a specific expert model in the virtual address space.

[0054] When the execution entity receives a data write instruction as the communication receiver, it first accesses the pre-created virtual address space, and locates the virtual expert address in the virtual space according to the first address information carried in the data write instruction, so as to write the data to be processed in the data write instruction into the expert model corresponding to the communication receiver.

[0055] Step 203: Apply for the corresponding actual physical space for the virtual expert address based on the size of the data to be processed.

[0056] Based on step 202, this step aims to allocate actual physical space for the data to be processed by the communication receiver.

[0057] The virtual expert address corresponds to the address pointed to by the first address information. When the data write instruction arrives and accesses this virtual address, the actual physical space corresponding to the virtual expert address is applied for. The actual physical space refers to the physical memory space in the computing device, which is used to actually store data. The communication receiver will dynamically allocate physical memory according to the size of the data to be processed to ensure that there is enough space to store the data to be processed. Among them, the size of the actual physical space is the same as the size of the data to be written. Of course, the size of the actual physical space can also be slightly larger or smaller than the size of the data to be processed, which can be determined according to the actual situation, and the present disclosure does not limit this.

[0058] Step 204: Write the data to be processed into the actual physical space.

[0059] Based on step 203, this step aims to write the data to be processed to the target address according to the data writing instruction to complete this communication.

[0060] The communication receiving end writes the data to be processed into the actual physical space applied for in step 203 to complete the data storage process. By writing the data into the actual physical space, the mixture-of-experts model can access and use this data in subsequent calculation processes.

[0061] In the embodiments of the present disclosure, instead of applying for actual physical memory, a virtual address space is pre-created in each execution entity. When the execution entity of the communication receiving end receives a data writing instruction (i.e., a data packet), and accesses the virtual expert address of the virtual address space according to the first address information in the data writing instruction, the actual physical space is allocated on demand at the address pointed to by the first address information according to the size of the data to be processed in the data writing instruction. The collective communication method provided in the present disclosure does not require synchronizing address information during the communication process, saving time overhead and improving processing performance; moreover, the communication method provided in the present disclosure applies for the corresponding actual physical space according to the size of the data to be processed only when the data packet arrives and accesses the virtual expert address, which can prevent resource explosion caused by excessive memory occupation and further improve processing performance.

[0062] In some alternative implementation manners of this embodiment, the virtual address space is created based on the total size of all the data to be processed in the collective communication.

[0063] Specifically, the virtual address space is pre-created inside the execution entity. Before the communication receiving end performs collective communication, it will pre-estimate the total size of all the data to be processed that needs to be transmitted during the collective communication process. Based on this total size, the communication receiving end creates a virtual address space of the corresponding size to ensure that there is enough virtual address space to map all the data to be processed.

[0064] Since the execution entity does not know how many data it will receive during the training or inference process, the virtual address space is created according to the total size of all the data to be processed of all the execution entities in the collective communication to ensure that the virtual expert address corresponding to the data to be processed can be found in the virtual address space according to the first address information during the communication process, so that the actual physical space can be applied for according to this virtual expert address, and thus the data communication can be accurately completed.

[0065] For example, if the total size of all the data to be processed during the collective communication is 1GB, the communication receiving end will create a virtual address space with a size of at least 1GB. This can ensure that all the data to be processed can find corresponding positions in the virtual address space.

[0066] Among them, the total size of all the data to be processed in the collective communication can be obtained by calculating the accumulation of the sizes of all the data to be processed. Optionally, the total size of all the data to be processed in the collective communication can also be determined according to a pre-stored log file, or determined according to a data set set in advance for training or inference, etc. The present disclosure does not limit this.

[0067] In some optional implementation manners of this embodiment, the size of the virtual address space is determined by the total number of data to be processed and the size of the data to be processed in the collective communication.

[0068] Specifically, each execution entity can process one or more data to be processed, and the sizes of the data to be processed are equal. Then, the total size of all the data to be processed in the collective communication can be obtained by calculating the product of the total number of data to be processed and the size of the data to be processed. That is, all the Token numbers * hidden Size, where hidden Size is the Token Size, and the Token Size is a fixed value. Determining the size of the virtual address space in this way can ensure that the virtual address space is neither too large to waste resources nor too small to cause insufficient space.

[0069] In some optional implementation manners of this embodiment, the virtual address space includes a virtual numbered address space and a virtual expert address space. Among them, the virtual numbered address space is used to store the number information of each participant in the collective communication (that is, each execution entity, including the execution entity corresponding to the communication request end and the execution entity corresponding to the communication receiving end). The virtual expert address space is used to store the data of each expert model, and each virtual expert address space corresponds to a data storage area for an expert model. When the execution entity of the communication receiving end accesses the virtual numbered address space, it will route to access the corresponding virtual expert address space.

[0070] Among them, the virtual numbered address space can be created by any execution entity in the collective communication, and then the created virtual numbered address space is broadcast to other execution entities. The virtual expert address space is created by each execution entity respectively. The sizes of the virtual numbered address space and the virtual expert address space are the same, and both are created according to the total size of all the data to be processed during the collective communication process.

[0071] It should be noted that the creation of this virtual numbered address space is for interacting with the switch during the collective communication process. After the switch receives the data writing instruction sent by the communication request end, it first multicasts the data writing instruction to the virtual numbered address space corresponding to the communication receiving end, and there is no actual meaning of storing data. Therefore, this virtual numbered address space may not necessarily be a piece of virtual address space, and it may also be just a handle of a virtual address space. The virtual expert address space will apply for real memory, and the virtual expert address space will be bound and mapped to the virtual numbered address space.

[0072] Figure 3 Schematic diagram of a virtual numbered address space and a virtual expert address space in collective communication based on a hybrid expert model provided by an embodiment of the present disclosure.

[0073] As Figure 3 shown in, assume that the execution entity is a processor. Taking the communication among four processors, namely processor Rank0, processor Rank1, processor Rank2, and processor Rank3, as an example for illustration. Among them, the input of each processor includes 4 input Tokens, and each processor includes two expert models. Therefore, the size of the virtual numbered address space created by the processor is equal to the total size of the input Tokens of the four processors, that is, the size occupied by 16 input Tokens. The mapping relationship between the virtual numbered address space and the virtual expert address space is a one-to-two mapping.

[0074] Refer to again Figure 3 , issue a data writing instruction, where the address pointed to by the first address information in the data writing instruction is the virtual space address in the virtual numbered address space. The execution entity of the communication request end sends the data writing instruction (message) to the switch, and the switch copies the data writing instruction into multiple copies for multicasting the data writing instruction, sends it to the virtual numbered address space of the communication receiving end, and accesses the virtual expert address of the virtual expert address space based on the mapping relationship between the virtual numbered address space and the virtual expert address space.

[0075] Figure 4 Flowchart of another collective communication method based on a hybrid expert model provided by an embodiment of the present disclosure. As Figure 4 shown, this method 400 includes the following steps:

[0076] Step 401: Receive a data writing instruction.

[0077] The specific operation of this step is the same as that of step 201 above, and will not be repeated here.

[0078] Step 402: Access the virtual numbered address of the pre-created virtual numbered address space according to the first address information.

[0079] Specifically, the communication receiving end locates the corresponding virtual numbered address in the virtual numbered address space according to the first address information. The virtual numbered address contains information related to a specific number, such as the number value, the expert model corresponding to the number, etc.

[0080] Step 403: Through the mapping between the virtual numbered address space and the virtual expert address space, access the virtual expert address in the virtual expert address space.

[0081] Specifically, there is a mapping relationship between the virtual numbered address space and the virtual expert address space. The communication receiving end finds the corresponding virtual expert address in the virtual expert address space through this mapping relationship from the virtual numbered address.

[0082] Among them, each execution entity may include one expert model or multiple expert models.

[0083] When the execution entity of the communication receiving end includes only one expert model, the virtual numbered address space and the virtual expert address space are linearly mapped. At this time, accessing the virtual numbered address space is equal to accessing the virtual expert address space. After accessing the virtual numbered address in the virtual numbered address space according to the first address information, the corresponding virtual expert address space can be directly routed and accessed.

[0084] When the execution entity of the communication receiving end includes at least two expert models, the virtual numbered address space and the virtual expert address space are one-to-many mapped. At this time, accessing the virtual numbered address space is equal to accessing multiple virtual expert address spaces, and multiple memory accesses need to be performed inside the execution entity to access multiple virtual expert address spaces. The data write instruction received by the communication receiving end may only need to be written into some of the expert models of the communication receiving end. Therefore, after accessing the virtual numbered address in the virtual numbered address space according to the first address information, the corresponding virtual expert address space can be routed and accessed according to the expert model to which the data to be processed is to be written.

[0085] Through the design of this two-level address space in the embodiments of the present disclosure, flexible mapping between numbers and expert models can be realized, which is convenient for quickly locating the corresponding expert model data according to the number information during the collective communication process.

[0086] Step 404: Based on the size of the data to be processed, apply for the corresponding actual physical space for the virtual expert address.

[0087] The specific operation of this step is the same as that of step 203 above, and will not be repeated here.

[0088] Step 405: Write the data to be processed into the actual physical space.

[0089] The specific operations of this step are the same as those of step 204 above, and will not be repeated here.

[0090] In some alternative implementation manners of this embodiment, the data write instruction further includes first routing information. Step 403: Access the virtual expert address in the virtual expert address space through the mapping between the virtual number address space and the virtual expert address space, which specifically includes: in response to the virtual number address space corresponding to at least two virtual expert address spaces, according to the first routing information, access the virtual expert address in the virtual expert address space through the mapping between the virtual number address space and the virtual expert address space.

[0091] Specifically, one virtual number address may correspond to multiple virtual expert address spaces, that is, one execution entity may be associated with multiple expert models. In this case, the communication receiving end needs to determine the virtual expert address space to be accessed according to the first data routing information.

[0092] The first data routing information may be in the form of an expert bitmap or in the form of other data structure information. For example, the first data routing information may be an indicator or identifier for indicating which one to select among multiple possible virtual expert address spaces. The communication receiving end selects one from the multiple virtual expert address spaces corresponding to the virtual number address according to the first data routing information, and then accesses the virtual expert address in the virtual expert address space.

[0093] In this way, the present disclosure can achieve flexible mapping of one number corresponding to multiple expert models, enhancing the scalability and flexibility of the system.

[0094] In some alternative implementation manners of this embodiment, step 404: Apply for a corresponding actual physical space for the virtual expert address based on the size of the data to be processed, which specifically includes the following steps:

[0095] In response to writing the data to be processed into the virtual expert address, a page fault interrupt is triggered. Based on the page fault interrupt, an actual physical space is allocated for the virtual expert address, and the mapping relationship between the virtual expert address and the actual physical space is written into the page table memory. The actual physical space corresponding to the virtual expert address is obtained based on the page table memory.

[0096] Specifically, when the communication receiving end accesses the virtual expert address in the pre-created virtual address space according to the first address information, the communication receiving end attempts to write the data to be processed into the virtual expert address. At this time, since the virtual expert address has not been mapped to the actual physical memory, the system will trigger a page fault interrupt. The page fault interrupt is an exception handling mechanism in the operating system for handling the situation of missing mapping from virtual address to physical address.

[0097] The system responds to a page error interrupt and allocates actual physical memory space for the virtual expert address. After the allocation is completed, the system writes the mapping relationship between the virtual expert address and the actual physical space into the page table memory. The page table memory is a data structure used by the operating system to store the mapping relationship between virtual addresses and physical addresses.

[0098] The communication receiving end obtains the address of the actual physical space corresponding to the virtual expert address by querying the page table memory. Based on this actual physical address, the communication receiving end can write the data to be processed into the actual physical memory.

[0099] In the embodiments of the present disclosure, when the data write instruction reaches the communication receiving end and accesses the virtual expert address of the communication receiving end, a page terminal error is triggered to apply for the corresponding actual physical memory for the virtual expert address. There is no need to synchronize memory information during the communication process, which reduces the time consumption and improves the performance. Further, by this way of allocating physical memory on demand, the memory usage efficiency can be improved, and the pre-allocation of a large amount of physical memory that may not be used can be avoided.

[0100] In some optional implementation manners of this embodiment, allocating actual physical space for the virtual expert address in step 404 specifically includes: allocating actual physical space for the virtual expert address through the memory processing module in the processor.

[0101] Specifically, the processor integrates a dedicated memory processing module for processing the mapping from virtual addresses to physical addresses and the allocation of physical memory. When it is necessary to allocate actual physical space for the virtual expert address, the system will call this memory processing module to complete the allocation task.

[0102] The memory processing module will allocate a physical memory block of an appropriate size from the available physical memory pool according to the size of the data to be processed, and then establish a mapping relationship from the virtual expert address to this physical memory block.

[0103] By using the dedicated memory processing module in the processor, the efficiency of memory allocation and mapping can be improved, the intervention of the operating system can be reduced, and thus the overall performance of the system can be improved.

[0104] In some optional implementation manners of this embodiment, the memory processing module is the page manager core or the hardware page management engine.

[0105] Specifically, the page manager core is a software-implemented memory management module that runs as part of the operating system kernel. The page manager core is responsible for managing the mapping relationship between virtual memory and physical memory, handling page error interrupts, and allocating and releasing physical memory pages.

[0106] The Hardware PageManager Engine is a hardware-implemented memory management module integrated in the processor or memory controller. The Hardware PageManager Engine can directly handle the translation of virtual addresses to physical addresses without software intervention, thereby improving the efficiency of memory access.

[0107] The memory processing module in the embodiments of the present disclosure can use a software-implemented page manager core or a hardware-implemented Hardware PageManager Engine, both of which can effectively allocate actual physical space for virtual expert addresses and establish a mapping relationship from virtual addresses to physical addresses, which can be flexibly set according to the actual application scenario.

[0108] Figure 5 FIG. is a schematic diagram of data multicast in the Dispatch stage provided for the embodiments of the present disclosure. Figure 6 For Figure 5 schematic diagram of the first routing information carried by the data write instruction related to

[0109] As Figure 5 - 6 shown, taking the communication including 4 processor Ranks in collective communication as an example for illustration, the input of each processor Rank includes 5 input Tokens, and each processor Rank corresponds to 2 expert models.

[0110] Each Rank is based on each Input Token and carries the first routing information corresponding to the Token. The first routing information includes a number bitmap and an expert bitmap.

[0111] Taking Token2 as an example, Token2 of Rank0 needs to be sent to Rank1 and Rank3 according to the number bitmap, where the expert bitmap is used by the Rank GPU at the communication receiving end. Among them, when a processor includes only one expert model, the number bitmap and the expert bitmap can be normalized. It should be noted that: the number bitmap and the expert bitmap are only a schematic representation of the first routing information, and other representation methods can also be used, such as the data structure information corresponding to the input data and the expert.

[0112] As Figure 5 shown, Rank0 calls the data write instruction inc_v.store instruction. In Rank0, the data write instruction is converted into the form of a data packet, which includes a data write request and carries the first routing information (including a number bitmap and an expert bitmap), the first address information, and the data Token2 to be processed corresponding to the data Token2 to be processed. Rank0 sends the data packet to the switch.

[0113] The switch receives the data packet sent by Rank0 and can multicast the data packet to the corresponding communication receivers Rank1 and Rank3 based on the number bitmap, completing the multicast transmission of the data packet.

[0114] For the receivers Rank1 and Rank3 of Rank, based on the expert bitmap, experts 2 and 3 inside Rank1 and experts 6 and 7 inside Rank3 write the data into multiple actual physical spaces corresponding to the virtual expert addresses pointed to by the first address information. After the data writing operation is completed, Rank1 and Rank3 then feedback the write response to the switch, and the switch feeds back the write response to the communication requestor processor Rank0.

[0115] In some alternative implementation manners of this embodiment, as Figure 4 shown, the collective communication method based on the hybrid expert model not only includes the above steps 401 to 405, but also includes the following steps:

[0116] Step 406: Receive a data read instruction, where the data read instruction includes second address information and weight distribution information.

[0117] Specifically, the communication receiver receives the data read instruction, which is used to read specific data from the stored data and perform calculations. The data read instruction includes second address information and weight distribution information. The second address information is used to indicate the storage location of the data to be read, and the weight distribution information is used to indicate how to perform weighted reading of the data in multiple expert models of the communication receiver.

[0118] Step 407: According to the second address information, read the data to be processed from the actual physical address of the virtual expert address in the virtual address space.

[0119] Specifically, the communication receiver locates the virtual expert address in the virtual address space according to the second address information, and then finds the corresponding actual physical address through the virtual expert address, and reads the data to be processed from the actual physical address.

[0120] Step 408: Calculate the data to be processed based on the weight distribution information to obtain response data.

[0121] Specifically, the communication receiver performs weighted calculation on the data to be processed read from the expert model corresponding to the communication receiver according to the weight distribution information to obtain response data. The weight distribution information can be a set of weight values, which are used to indicate the importance or contribution degree of different data in the calculation.

[0122] It should be noted that the process of calculating the response data based on the weight distribution information depends on the hardware to complete the calculation work; if the hardware does not support it, it can also be implemented by software, in which case it is necessary to maintain the physical memory corresponding to the virtual expert address space, calculate the above-mentioned response data based on the virtual expert address space in the expert model, and write it into the actual physical space pointed to by the virtual expert address.

[0123] In this way, the present disclosure can achieve weighted fusion of the outputs of different expert models in the mixture of experts model, so as to obtain the final model output.

[0124] In some alternative implementation manners of this embodiment, the data read instruction further includes second routing information, and the virtual address space includes a virtual number address space and a virtual expert address space.

[0125] Step 407: Read the data to be processed from the actual physical address of the virtual expert address in the virtual address space according to the second address information, which specifically includes the following steps:

[0126] According to the second address information, determine the virtual number address of the virtual number address space; according to the second routing information, through the mapping between the virtual number address space and the virtual expert address space, determine the virtual expert address from the virtual expert address space; read the data to be processed from the actual physical address of the virtual expert address.

[0127] Specifically, the communication receiving end locates the corresponding virtual number address in the virtual number address space according to the second address information. When a virtual number address corresponds to multiple virtual expert address spaces, the communication receiving end needs to determine which virtual expert address space to access according to the second routing information. The communication receiving end selects one from the multiple virtual expert address spaces corresponding to the virtual number address space according to the second routing information, and then determines the virtual expert address in the virtual expert address space.

[0128] The communication receiving end finds the corresponding actual physical address through the virtual expert address, and then reads the data to be processed from the actual physical address.

[0129] In this way, the present disclosure can flexibly read the data of a specific expert model in the case of multiple virtual expert address spaces, providing support for the calculation of the mixture of experts model.

[0130] It can be understood that the operation corresponding to the data read instruction is the reverse process of the above-mentioned data write instruction, and the specific implementation processes of the two are similar, so details are not described here again.

[0131] Figure 7 It is a schematic diagram of data reduction in the Combine stage provided by an embodiment of the present disclosure.Figure 8 Schematic diagram of second routing information carried by a data read instruction related to Figure 7

[0132] As Figure 7 - 8 described, in this embodiment, communication still includes 4 processor Ranks in collective communication, the input of each processor Rank includes 5 input Tokens, and each processor Rank corresponds to 2 expert models as an example for description.

[0133] Still taking the input Token2 as an example, Rank0 calls the data read instruction inc_v.ldr instruction, and the AI Core sends the data read instruction to Rank0. In Rank0, the data read instruction is converted into the form of a data packet, and the data packet includes a data read request and carries the second routing information (including a number bitmap and an expert bitmap), second address information, and weight distribution information corresponding to the to-be-processed data Token2. Rank0 sends the data packet to the switch.

[0134] The switch receives the data packet sent by Rank0, and based on the number bitmap, can multicast the data packet to the corresponding communication receiving ends Rank1 and Rank3 to complete the multicast transmission of the data packet.

[0135] For the receiving ends Rank1 and Rank3 of the Rank, at the port nodes of Expert 2 and Expert 3 inside Rank1 and Expert 6 and Expert 7 inside Rank3 based on the expert bitmap, it is necessary to complete the reading of multiple expert data and perform calculations based on the weight distribution information to obtain the response data.

[0136] That is, Token2.rank3.data = Token2.Expert6.weigth * Token2.Expert6_data + Token2.Expert7.weigth * Token2.Expert7_data.

[0137] Rank1 and Rank3 return the calculated response data to the switch. The switch receives the corresponding data of Rank1 and Rank3 and completes the corresponding reduction calculation task; finally, it returns the reduction calculation result to Rank0 and writes the data to the memory pre-created by Rank0.

[0138] ​In the embodiments of the present disclosure, a virtual address space is pre-created in each execution entity instead of applying for actual physical memory. When the execution entity at the communication receiving end receives a data writing instruction (i.e., a data packet), and accesses the virtual expert address in the virtual address space according to the first address information in the data writing instruction, only then is actual physical space allocated as needed at the address pointed to by the first address information according to the size of the data to be processed in the data writing instruction. In the collective communication method provided by the present disclosure, there is no need to synchronize address information during the communication process, saving time overhead and improving processing performance. Moreover, in the communication method provided by the present disclosure, the corresponding actual physical space is applied for only when the data packet arrives and accesses the virtual expert address according to the size of the data to be processed, which can prevent resource explosion caused by excessive memory occupation and further improve processing performance.

[0139] Further referring to Figure 9 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a collective communication system based on a mixture of experts model. This communication system embodiment corresponds to Figure 2 the method embodiment shown, and this communication system can be specifically applied to various electronic devices.

[0140] As Figure 9 shown, the communication system 900 of this embodiment may include: a plurality of processors 901 and a switch 902. Among them, these processors 901 can both initiate a communication request to other processors 901 as a communication request end, and receive a communication request sent by other processors 901 as a communication receiving end.

[0141] When the processor 901 receives a sent instruction as a communication request end, the processor 901 sends the instruction to the switch 902.

[0142] The switch 902 is configured to receive an instruction, where the instruction includes data to be processed, address information, and data routing information; and based on the data routing information, multicast the instruction to the corresponding processor.

[0143] When the processor 902 is a communication receiving end, it is configured to, in response to the received instruction being a data writing instruction, access the virtual expert address in the pre-created virtual address space according to the address information; and based on the size of the data to be processed, apply for the corresponding actual physical space for the virtual expert address; and write the data to be processed into the actual physical space.

[0144] In this embodiment, in the communication system 900: the specific processing of the processor 901 as a communication receiving end and the technical effects brought by it can respectively refer to Figure 2 the relevant descriptions of steps 201-204 in the corresponding embodiment, which will not be elaborated here.

[0145] The communication system provided by the embodiments of the present disclosure transfers the operation of data multicasting to a switch for processing, enabling the switch to perform multicasting on an instruction according to the data routing information carried by the instruction, reducing the overhead of the processor for multicasting data, and further improving the computing efficiency of the communication system. The solution of the present disclosure has strong scalability, can meet the acceleration requirements of various collective communication operations, and can be applied to various implementation modes such as internal interconnection of hosts and networking interconnection between hosts.

[0146] In some alternative implementation manners of this embodiment, the virtual address space includes a virtual numbered address space and a virtual expert address space. The processor 901 is specifically configured to, in response to the received instruction being a data writing instruction, access the virtual numbered address of the pre-created virtual numbered address space based on the address information; and access the virtual expert address of the virtual expert address space through the mapping between the virtual numbered address space and the virtual expert address space.

[0147] In some alternative implementation manners of this embodiment, the routing information includes an expert bitmap and a numbered bitmap. The switch 902 is specifically configured to multicast the instruction to the corresponding processor 901 based on the numbered bitmap. The processor 901 is specifically configured to, in response to the received instruction being a data writing instruction, and in response to the virtual numbered address space corresponding to at least two virtual expert address spaces, access the virtual numbered address of the pre-created virtual numbered address space based on the address information; and access the virtual expert address of the virtual expert address space according to the expert bitmap through the mapping between the virtual numbered address space and the virtual expert address space.

[0148] In this embodiment, in the communication system 900: The specific processing of the processor 801 as the communication receiving end and the technical effects brought thereby can be respectively referred to the relevant descriptions of steps 402-403 in the above method embodiment, and will not be elaborated here.

[0149] Figure 10 It is a structural diagram of a physical memory allocation device provided by the embodiments of the present disclosure.

[0150] Such as Figure 10As shown in the figure, when the processor 901 of the communication receiving end accesses the virtual expert address in the pre-created virtual address space according to the first address information and attempts to write the data to be processed into the virtual expert address. The communication interface 1012 in the processor 901 is routed to the memory management unit 1022 in the processor 901 through the address access bus of the virtual address space; the memory management unit 1022 in the processor 901 triggers a PageFault interrupt and sends it to the processor driver 1011 of the host 1001; the processor driver 1011 of the host 1001 allocates physical memory for the virtual expert address and writes the page table into the page table memory 1032 of the processor; finally, the communication interface 1012 can access the actual physical space corresponding to the virtual expert address.

[0151] Figure 11 It is a structural diagram of another physical memory allocation device provided by an embodiment of the present disclosure.

[0152] In some optional implementation manners of this embodiment, as Figure 11 shown, the processor 901 includes a communication interface 1012, a memory management unit 1022, and a memory processing module 1042. The communication interface 1012 is configured to access the memory management unit 1022 based on the virtual expert address access bus. The memory management unit 1022 is configured to trigger a page fault interrupt in response to writing the data to be processed into the virtual expert address and send the interrupt to the memory processing module 1042. The memory processing module 1042 is configured to allocate an actual physical space for the virtual expert address, write the mapping relationship between the virtual expert address and the actual physical space into the page table memory, and obtain the actual physical space corresponding to the virtual expert address based on the page table memory.

[0153] Specifically, as Figure 11 shown, when the processor 1002 of the communication receiving end accesses the virtual expert address in the pre-created virtual address space according to the first address information and attempts to write the data to be processed into the virtual expert address. The communication interface 1012 in the processor 1002 carries the address access bus of the virtual address space and is routed to the memory management unit 1022 in the processor 1002; the memory management unit 1022 in the processor 1002 triggers a PageFault interrupt and sends it to the memory processing module 1042 inside the processor 1002 for processing; the memory processing module 1042 inside the processor 1002 allocates physical memory for the virtual expert address and writes the page table into the page table memory 1032 of the processor; finally, the communication interface 1012 can access the actual physical space corresponding to the virtual expert address. By using the dedicated memory processing module in the processor, the efficiency of memory allocation and mapping can be improved, the intervention of the operating system can be reduced, and thus the overall performance of the system can be improved.

[0154] In some alternative implementation manners of this embodiment, the memory processing module is a page manager core or a hardware page management engine.

[0155] Specifically, the page manager core is a software-implemented memory management module that runs as part of the operating system kernel. The page manager core is responsible for managing the mapping relationship between virtual memory and physical memory, handling page fault interrupts, and allocating and releasing physical memory pages.

[0156] The hardware page management engine is a hardware-implemented memory management module integrated in the processor or memory controller. The hardware page management engine can directly handle the conversion from virtual addresses to physical addresses without software intervention, thereby improving the efficiency of memory access.

[0157] The memory processing module 1042 in the embodiments of the present disclosure can use a software-implemented page manager core or a hardware-implemented hardware page management engine, both of which can effectively allocate actual physical space for virtual expert addresses and establish a mapping relationship from virtual addresses to physical addresses, which can be specifically set flexibly according to the actual application scenario.

[0158] In some alternative implementation manners of this embodiment, the processor 901 is further configured to, in response to a data read instruction, where the data read instruction further includes weight distribution information, read the data to be processed from the actual physical address of the virtual expert address in the virtual address space according to the address information, and perform calculations on the data to be processed based on the weight distribution information to obtain response data. The switch 902 is further configured to perform reduction calculations on the response data and feed back the reduction calculation result to the instruction sending end.

[0159] In this embodiment, for the specific processing procedure of the data read received by the processor 901 in the communication system 900 and the technical effects brought thereby, reference may be made to the relevant descriptions of steps 406 to 408 in the above method embodiment, which will not be elaborated herein.

[0160] It should be noted that other details in the collective communication system provided in the embodiments of the present disclosure are the same as those in any of the above collective communication methods, and will not be repeated herein.

[0161] This embodiment exists as a communication system embodiment corresponding to the above method embodiment. The communication system provided in this embodiment pre-creates a virtual address space in each processor instead of applying for actual physical memory. When the processor at the communication receiving end receives a data writing instruction (i.e., a data packet), and accesses the virtual expert address in the virtual address space according to the address information in the data writing instruction, it then allocates actual physical space as needed in the address pointed to by the address information according to the size of the data to be processed in the data writing instruction. In the collective communication system provided in this disclosure, there is no need to synchronize address information during the communication process, saving time overhead and improving processing performance. Moreover, the communication system provided in this disclosure applies for the corresponding actual physical space according to the size of the data to be processed only when the data packet arrives and accesses the virtual expert address, which can prevent resource explosion caused by excessive memory occupation and further improve processing performance. Further, in the communication system provided in this embodiment of the disclosure, the operation of data multicast is transferred from the processor to the switch, so that the switch performs multicast on the instruction according to the data routing information carried in the instruction, reducing the overhead of the processor for multicasting data and further improving the computing efficiency of the communication system. The solution of this disclosure has strong scalability, can meet the acceleration requirements of various collective communication operations, and can be applied to various implementation methods such as internal interconnection of hosts and networking interconnection between hosts.

[0162] Figure 12 This is a collective communication device 1200 based on a mixture of experts model provided in an embodiment of the present disclosure. Further reference is made to Figure 12 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a collective communication device based on a mixture of experts model. This device embodiment corresponds to Figure 2 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0163] As Figure 12 shown, the collective communication device 1200 based on a mixture of experts model in this embodiment may include a receiving unit 1201, an accessing unit 1202, an applying unit 1203, and a writing unit 1204. Among them, the receiving unit 1201 is configured to receive a data writing instruction, where the data writing instruction includes data to be processed and first address information; the accessing unit 1202 is configured to access a virtual expert address in a pre-created virtual address space according to the first address information; the applying unit 1203 is configured to apply for a corresponding actual physical space for the virtual expert address based on the size of the data to be processed; the writing unit 1204 is configured to write the data to be processed into the actual physical space.

[0164] In this embodiment, in the collective communication device 1200 based on the mixture-of-experts model: For the specific processing of the receiving unit 1201, the access unit 1202, the application unit 1203, and the writing unit 1204 and the technical effects brought thereby, reference can be respectively made to Figure 2 the relevant descriptions of steps 201-204 in the corresponding embodiment, which will not be elaborated herein.

[0165] In some alternative implementation manners of this embodiment, the virtual address space is created based on the total size of all the data to be processed in the collective communication.

[0166] In some alternative implementation manners of this embodiment, the size of the virtual address space is determined by the total number of the data to be processed and the size of the data to be processed in the collective communication.

[0167] In some alternative implementation manners of this embodiment, the virtual address space includes a virtual number address space and a virtual expert address space; the access unit 1202 is further configured to: access the virtual number address of the pre-created virtual number address space according to the first address information; and access the virtual expert address of the virtual expert address space through the mapping between the virtual number address space and the virtual expert address space.

[0168] In some alternative implementation manners of this embodiment, the data writing instruction further includes first routing information, and the access unit 1202 is further configured to: in response to the virtual number address space corresponding to at least two virtual expert address spaces, according to the first routing information, access the virtual expert address of the virtual expert address space through the mapping between the virtual number address space and the virtual expert address space.

[0169] In some alternative implementation manners of this embodiment, the application unit 1203 is further configured to: in response to writing the data to be processed into the virtual expert address, trigger a page fault interrupt; based on the page fault interrupt, allocate an actual physical space for the virtual expert address, and write the mapping relationship between the virtual expert address and the actual physical space into the page table memory; and obtain the actual physical space corresponding to the virtual expert address based on the page table memory.

[0170] In some alternative implementation manners of this embodiment, the application unit 1203 is further configured to: allocate an actual physical space for the virtual expert address through the memory processing module in the processor.

[0171] In some alternative implementation manners of this embodiment, the memory processing module is a page manager core or a hardware page management engine.

[0172] In some alternative implementation manners of this embodiment, the receiving unit 1201 is further configured to: receive a data read instruction, where the data read instruction includes second address information and weight allocation information. The apparatus 1200 further includes a data reading unit and a calculation unit. The data reading unit is configured to read data to be processed from the actual physical address of the virtual expert address in the virtual address space according to the second address information. The calculation unit is configured to calculate the data to be processed based on the weight allocation information to obtain response data.

[0173] In some alternative implementation manners of this embodiment, the data read instruction further includes second routing information. The virtual address space includes a virtual number address space and a virtual expert address space. The data reading unit is further configured to: determine the virtual number address of the virtual number address space according to the second address information; determine the virtual expert address from the virtual expert address space through the mapping between the virtual number address space and the virtual expert address space according to the second routing information; and read the data to be processed from the actual physical address of the virtual expert address.

[0174] This embodiment exists as an apparatus embodiment corresponding to the above method embodiment. The collective communication apparatus based on the mixture of experts model provided in this embodiment creates a virtual address space in advance instead of applying for actual physical memory. When a data write instruction (i.e., a data packet) is received and the virtual expert address of the virtual address space is accessed according to the first address information in the data write instruction, actual physical space is allocated on demand at the address pointed to by the first address information according to the size of the data to be processed in the data write instruction. The collective communication apparatus provided in this disclosure does not need to synchronize address information during the communication process, saving time overhead and improving processing performance. Moreover, the communication apparatus provided in this disclosure applies for the corresponding actual physical space according to the size of the data to be processed only when the data packet arrives and accesses the virtual expert address, which can prevent resource explosion caused by excessive memory occupation and further improve processing performance.

[0175] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor. When the instructions are executed by the at least one processor, the at least one processor is enabled to implement the collective communication method based on the mixture of experts model described in any of the above embodiments.

[0176] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions for enabling a computer to implement the collective communication method based on the mixture of experts model described in any of the above embodiments when executed.

[0177] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which, when executed by a processor, can implement the set communication method based on the mixture-of-experts model described in any of the above embodiments.

[0178] Figure 13 FIG. shows a schematic block diagram of an exemplary electronic device 1300 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0179] As Figure 13 shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other through a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0180] Multiple components in the electronic device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a magnetic disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the electronic device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0181] The computing unit 1301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 executes the various methods and processes described above, such as the collective communication method based on the mixture-of-experts model. For example, in some alternative implementations of this embodiment, the collective communication method based on the mixture-of-experts model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1308. In some alternative implementations of this embodiment, part or all of the computer program can be loaded and / or installed onto the electronic device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the collective communication method based on the mixture-of-experts model described above can be executed. Alternatively, in other embodiments, the computing unit 1301 can be configured to execute the collective communication method based on the mixture-of-experts model in any other suitable manner (e.g., by means of firmware).

[0182] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program codes cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, executed partially on the machine as an independent software package and partially on a remote machine, or executed entirely on a remote machine or server.

[0184] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0187] A computer system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of high management difficulty and weak business scalability existing in traditional physical hosts and virtual private server (VPS, Virtual Private Server) services.

[0188] According to the technical solution of the embodiment of the present disclosure, by pre-creating a virtual address space in each processor instead of applying for actual physical memory, when the processor at the communication receiving end receives a data write instruction (i.e., a data packet), and accesses the virtual expert address of the virtual address space according to the address information in the data write instruction, only then is the actual physical space allocated on demand according to the size of the data to be processed in the data write instruction at the address pointed to by the address information. The collective communication scheme provided in the present disclosure does not require synchronization of address information during the communication process, saving time overhead and improving processing performance; moreover, the communication system provided in the present disclosure applies for the corresponding actual physical space according to the size of the data to be processed only when the data packet arrives and accesses the virtual expert address, which can prevent resource explosion caused by excessive memory occupation and further improve processing performance. Furthermore, in the solution provided by the embodiment of the present disclosure, the operation of data multicast is transferred from the processor to the switch, so that the switch performs multicast on the instruction according to the data routing information carried in the instruction, reducing the overhead of the processor for multicasting data and further improving the computing efficiency of the communication system. The solution of the present disclosure has strong scalability, can meet the acceleration requirements of various collective communication operations, and can be applied to various implementation methods such as interconnection inside the host and networking interconnection between hosts.

[0189] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitations are imposed herein.

[0190] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A collective communication method based on a hybrid expert model, the method being applied to a communication receiving end, comprising: Receiving a data write instruction, wherein the data write instruction includes data to be processed and first address information; Accessing a virtual expert address in a pre-created virtual address space according to the first address information; Based on the size of the data to be processed, applying for a corresponding actual physical space for the virtual expert address; The data to be processed is written into the actual physical space.

2. The method according to claim 1, wherein: The virtual address space includes a virtual numbered address space and a virtual expert address space; the accessing of the virtual expert address in the pre-created virtual address space according to the first address information includes: Accessing a virtual numbered address of a pre-created virtual numbered address space according to the first address information; The virtual expert address of the virtual expert address space is accessed through the mapping between the virtual number address space and the virtual expert address space.

3. The method according to claim 2, wherein: The data write instruction further includes first routing information, and the accessing of the virtual expert address of the virtual expert address space through the mapping between the virtual number address space and the virtual expert address space includes: In response to the virtual numbered address space corresponding to at least two virtual expert address spaces, the virtual expert address of the virtual expert address space is accessed through mapping between the virtual numbered address space and the virtual expert address space according to the first routing information.

4. The method according to claim 1, wherein: Based on the size of the data to be processed, applying for a corresponding actual physical space for the virtual expert address includes: In response to writing the to-be-processed data into the virtual expert address, triggering a page fault interrupt; Based on the page fault interrupt, allocating actual physical space to the virtual expert address, and writing the mapping relationship between the virtual expert address and the actual physical space into a page table memory; An actual physical space corresponding to the virtual expert address is obtained based on the page table memory.

5. The method according to claim 4, wherein: The allocating actual physical space to the virtual expert address comprises: Allocate actual physical space for the virtual expert address through a memory processing module in the processor; Wherein, the memory processing module is a page manager core or a hardware page management engine.

6. The method according to claim 1, further comprising: receiving a data read instruction, wherein the data read instruction includes second address information and weight allocation information; Reading the data to be processed from the actual physical address of the virtual expert address in the virtual address space according to the second address information; The data to be processed is calculated based on the weight distribution information to obtain response data.

7. The method according to claim 6, wherein: The data read instruction further includes second routing information, the virtual address space includes a virtual numbered address space and a virtual expert address space, and the reading of the to-be-processed data from the actual physical address of the virtual expert address in the virtual address space according to the second address information includes: determining the virtual numbered address of the virtual numbered address space according to the second address information; According to the second routing information, determining the virtual expert address from the virtual expert address space by mapping the virtual number address space with the virtual expert address space; The data to be processed is read from the actual physical address of the virtual expert address.

8. The method according to any one of claims 1 to 7, wherein: The virtual address space is created based on the total size of all data to be processed in the collective communication.

9. A collective communication system based on a hybrid expert model, the collective communication system comprising a plurality of processors and switches, wherein: The switch is configured to receive an instruction, the instruction including data to be processed, address information and data routing information; and multicast the instruction to a corresponding processor based on the data routing information; The processor is configured to access a virtual expert address in a pre-created virtual address space according to the address information in response to the received instruction being a data write instruction; and based on the size of the data to be processed, applying for a corresponding actual physical space for the virtual expert address; And writing the data to be processed into the actual physical space.

10. The communication system according to claim 9, wherein: The virtual address space includes a virtual number address space and a virtual expert address space; The processor is specifically configured to access a virtual numbering address of a pre-created virtual numbering address space based on the address information in response to the received instruction being a data write instruction; and access a virtual expert address of the virtual expert address space through a mapping between the virtual numbering address space and the virtual expert address space.

11. The communication system according to claim 10, wherein: The routing information includes an expert bitmap and a number bitmap, The switch is specifically configured to multicast the instruction to the corresponding processor based on the numbering bitmap; The processor is specifically configured to respond to the received instruction being a data write instruction, and in response to the virtual numbering address space corresponding to at least two virtual expert address spaces, access the virtual numbering address of the pre-created virtual numbering address space based on the address information; and according to the expert bitmap, access the virtual expert address of the virtual expert address space through the mapping of the virtual numbering address space and the virtual expert address space.

12. The communication system according to claim 9, wherein: The processor includes a communication interface, a memory management unit and a memory processing module. The communication interface is configured to access the memory management unit based on the virtual expert address access bus; The memory management unit is configured to trigger a page fault interrupt in response to writing the to-be-processed data into the virtual expert address, and send the interrupt to the memory processing module; The memory processing module is configured to allocate actual physical space to the virtual expert address, write the mapping relationship between the virtual expert address and the actual physical space into the page table memory, and obtain the actual physical space corresponding to the virtual expert address based on the page table memory.

13. The communication system according to claim 9, wherein: The processor is further configured to, in response to the instruction being a data read instruction, the data read instruction further comprising weight allocation information, read the data to be processed from the actual physical address of the virtual expert address in the virtual address space according to the address information, and calculate the data to be processed based on the weight allocation information to obtain response data; The switch is further configured to perform protocol calculation on the response data and feed back the protocol calculation result to the instruction sending end.

14. A collective communication device based on a hybrid expert model, comprising: A receiving unit, configured to receive a data writing instruction, wherein the data writing instruction includes data to be processed and first address information; an access unit, configured to access a virtual expert address in a pre-created virtual address space according to the first address information; An application unit, configured to apply for a corresponding actual physical space for the virtual expert address based on the size of the data to be processed; A writing unit is configured to write the data to be processed into the actual physical space.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the collective communication method according to any one of claims 1 to 8.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the collective communication method according to any one of claims 1 to 8.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the collective communication method according to any one of claims 1 to 8 are implemented.

Images