Data distribution method and apparatus, device, and medium

Abstract

The present disclosure provides a data distribution method and device, equipment and medium, relates to the chip technical field, and particularly relates to the distributed data processing and artificial intelligence model inference technical field. The implementation scheme is: receiving an input sequence including a plurality of data blocks and information of a to-be-distributed computing unit corresponding to each data block; splitting the input sequence into a plurality of sub-sequences based on a plurality of processing cores; determining a first number corresponding to each processing core and each computing unit, to indicate that a sub-sequence corresponding to the processing core includes a number of data blocks corresponding to the computing unit; using each processing core, for each computing unit, summing up the first number corresponding to each previous processing core of the processing core and the computing unit to determine an initial writing position in a target storage area corresponding to the computing unit, and writing the data block corresponding to the computing unit based on the initial writing position; and distributing the content in the plurality of target storage areas to the plurality of computing units.

Description

Technical Field

[0001] This disclosure relates to the field of chip technology, and more particularly to the field of distributed data processing and artificial intelligence model inference technology, specifically to a data distribution method, a hybrid expert model inference method and apparatus, a chip, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] In practical applications, when multiple processing units are needed to process a sequence of input data blocks, it is necessary to distribute multiple data blocks in the input sequence to different processing units. For example, in the inference and training scenarios of Mixture of Experts (MoE) models, the MoE model uses a routing mechanism to allocate different input data blocks (e.g., words / tokens) to several expert networks to achieve computational sparsity and efficient inference. In this case, it is necessary to distribute the input words to the inference hardware corresponding to different expert networks according to the expert allocation information.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a data distribution method, a reasoning method and apparatus for hybrid expert models, a chip, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of this disclosure, a data distribution method is provided, the method being used to distribute a plurality of sequentially arranged data blocks to a plurality of computing units, the method comprising: receiving an input sequence including the plurality of data blocks and information on one or more computing units to which each data block is to be distributed; splitting the input sequence into a plurality of sub-sequences corresponding to a plurality of processing cores respectively; determining a first quantity corresponding to each of the plurality of processing cores and each of the plurality of computing units, wherein the first quantity indicates the number of data blocks corresponding to the computing unit included in the sub-sequence corresponding to the processing core; and performing the following operations in parallel using each of the plurality of processing cores: for each of the plurality of computing units, in response to determining that at least one of the processing cores is present... The system comprises: a plurality of preprocessing cores; obtaining a first quantity corresponding to each of the at least one preprocessing cores and the computing unit, wherein the subsequence corresponding to the at least one preprocessing core is located before the subsequence corresponding to the processing core in the input sequence; determining an initial write position in the target storage area corresponding to the computing unit based on the sum of the first quantities corresponding to the at least one preprocessing core; determining the starting position of the target storage area corresponding to the unit as the initial write position in response to determining that there is no preprocessing core corresponding to the processing core; writing the data block corresponding to the computing unit, including the subsequence corresponding to the processing core, into the target storage area corresponding to the computing unit based on the initial write position; and distributing the contents of the target storage areas corresponding to the plurality of computing units to the plurality of computing units.

[0007] According to one aspect of this disclosure, a reasoning method for a hybrid expert model is provided, wherein the hybrid expert model includes multiple expert sub-networks, and the method includes: distributing multiple data blocks of an input sequence to be processed to multiple computing units corresponding to the multiple expert sub-networks using the above-described data distribution method; and implementing the reasoning operation of the hybrid expert model using the multiple computing units.

[0008] According to one aspect of this disclosure, a data distribution apparatus is provided, comprising: the apparatus for distributing a plurality of sequentially arranged data blocks to a plurality of computing units, the apparatus comprising: a receiving unit configured to receive an input sequence including the plurality of data blocks and information on one or more computing units to which each data block is to be distributed; a splitting unit configured to split the input sequence into a plurality of sub-sequences corresponding to a plurality of processing cores; a first determining unit configured to determine a first quantity corresponding to each of the plurality of processing cores and each of the plurality of computing units, wherein the first quantity indicates the number of data blocks corresponding to the computing unit included in the sub-sequence corresponding to the processing core; and an execution unit configured to execute operations in parallel using each of the plurality of processing cores, the execution computing unit comprising: a first acquiring subunit configured to, using each of the processing cores, respond to each of the plurality of computing units. The system comprises: upon determining that at least one preceding processing core exists for the processing core; obtaining a first quantity corresponding to each of the at least one preceding processing core and the computing unit, wherein the subsequence corresponding to the at least one preceding processing core is located before the subsequence corresponding to the processing core in the input sequence; a first determining subunit configured to determine an initial write position in the target storage area corresponding to the computing unit based on the sum of the first quantities corresponding to the at least one preceding processing core; a second determining subunit configured to determine the starting position of the target storage area corresponding to the unit as the initial write position in response to determining that there is no preceding processing core for the processing core; a write subunit configured to write the data block corresponding to the computing unit, including the subsequence corresponding to the processing core, into the target storage area corresponding to the computing unit based on the initial write position; and a distribution unit configured to distribute the contents in the target storage areas corresponding to each of the plurality of computing units to the plurality of computing units.

[0009] According to one aspect of this disclosure, an inference apparatus for a hybrid expert model is provided, the hybrid expert model including multiple expert sub-networks, the apparatus comprising: a data distribution device as described above, configured to distribute multiple data blocks including an input sequence to be processed to multiple computing units corresponding to the multiple expert sub-networks respectively; and an inference unit, configured to implement the inference operation of the hybrid expert model using the multiple computing units.

[0010] According to one aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data distribution method or the hybrid expert model reasoning method described above.

[0011] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the above-described data distribution method or the inference method of a hybrid expert model.

[0012] According to one aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, is capable of implementing the above-described data distribution method or the inference method of a hybrid expert model.

[0013] According to one or more embodiments of this disclosure, data distribution efficiency can be improved.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram of an exemplary system in which various methods described herein may be implemented, according to exemplary embodiments of the present disclosure; Figure 2 A flowchart of a data distribution method according to an exemplary embodiment of the present disclosure is shown; Figures 3A-3B A schematic diagram of a data distribution process according to an exemplary embodiment of the present disclosure is shown; Figure 4 A flowchart illustrating a reasoning method for a hybrid expert model according to an exemplary embodiment of the present disclosure is shown; Figure 5 A structural block diagram of a data distribution apparatus according to an exemplary embodiment of the present disclosure is shown; Figure 6 A structural block diagram of a hybrid expert model inference apparatus according to an exemplary embodiment of the present disclosure is shown; Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0017] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0019] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0020] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0021] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0022] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of data distribution methods.

[0023] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0024] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0025] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to send input sequences. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0026] Client devices 101, 102, 103, 104, 105, and / or 106 may include various categories of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various categories and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0027] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0028] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0029] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0030] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0031] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0032] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different categories. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0033] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be different categories of databases, such as key-value stores, object stores, or regular stores supported by a file system.

[0034] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0035] In practical applications, when multiple processing units are needed to process a sequence of input data blocks, it is necessary to distribute multiple data blocks in the input sequence to different processing units. For example, in the inference and training scenarios of Mixture of Experts (MoE) models, the MoE model uses a routing mechanism to allocate different input data blocks (e.g., words / tokens) to several expert networks to achieve computational sparsity and efficient inference. In this case, it is necessary to distribute the input words to the inference hardware corresponding to different expert networks according to the expert allocation information. Simultaneously, to improve the efficiency of various calculations (such as matrix multiplication) performed during the inference process of the neural network model, the distributed words need to be arranged contiguously in memory while maintaining sequence correctness, so that each expert network can achieve more efficient inference computation by efficiently reading the input data.

[0036] Regarding the aforementioned data distribution process, one implementation in related technologies involves using a dedicated processing core to determine the global write position corresponding to each word in the input word sequence, and then writing the input word sequence into the target storage area so that each computing unit can read the required input data from the target storage area. However, due to the large amount of data in the input word sequence, the time overhead of implementing this operation using a single processing core is too high, thus affecting computational efficiency. Furthermore, when using multiple processing cores for word rearrangement and distribution, the input sequence is typically split among different processing cores. Since each distributed word needs to maintain the correctness of its original sequence in the target storage area, each processing core, while processing its own assigned words, needs to wait for other processing cores to complete the writing operation of the preceding words. This dependency between multiple cores affects multi-core parallelism, meaning that efficient parallel processing cannot be achieved, thus limiting computational efficiency. For example, suppose the input sequence contains four data blocks t0, t1, t2, and t3, which need to be distributed into two sets of results: t0, t1, and t3, and t0, t1, and t2. t0 and t1, and t2 and t3, are assigned to two different processing cores for processing. In this case, since the write position of t2 / t3 depends on the space occupied by t0 and t1, the processing core responsible for t2 and t3 needs to wait for the first processing core to write t0 and t1 to the target storage area before writing its own tokens to the target storage area. This serial waiting mechanism severely restricts the parallel capabilities of multiple processing cores and limits data processing efficiency.

[0037] Based on this, this disclosure provides a data distribution method that predetermines the number of data blocks that each processing core needs to allocate to each computing unit. On this basis, each processing core determines its initial write position based on the sum of the first number of preceding processing cores, that is, it makes room in the target storage area for the space occupied by the preceding processing cores. Thus, the write operation that requires serial waiting in the prior art can be transformed into an independent write operation that can be implemented by pre-calculating the storage location offset. Each processing core can perform the operation of rearranging data blocks to write to the corresponding target storage area of ​​different computing units in parallel, thereby improving the data block transportation efficiency by improving the multi-core parallelism.

[0038] Figure 2 A flowchart of a data distribution method 200 according to an exemplary embodiment of the present disclosure is shown. Method 200 is used to distribute a plurality of sequentially arranged data blocks to a plurality of computing units. Figure 2 As shown, method 200 includes: Step S210: Receive an input sequence including multiple data blocks and information on one or more computing units to be distributed to each data block; Step S220: Divide the input sequence into multiple sub-sequences corresponding to multiple processing cores respectively; Step S230: Determine a first quantity corresponding to each of the plurality of processing cores and each of the plurality of computing units, wherein the first quantity indicates the number of data blocks corresponding to the computing unit included in the subsequence corresponding to the processing core; Step S240 utilizes each of the multiple processing cores to execute the operations of steps S241-S244 in parallel: Step S241: For each of the plurality of computing units, in response to determining that at least one preceding processing core exists for the processing core, a first number corresponding to each of the at least one preceding processing core and the computing unit is obtained, wherein the subsequence corresponding to the at least one preceding processing core is located before the subsequence corresponding to the processing core in the input sequence. Step S242: Based on the sum of the first number of the computing unit corresponding to at least one preceding processing core, determine the initial write position in the target storage area corresponding to the computing unit; Step S243: In response to determining that there is no preceding processing core for the processing core, determine the starting position of the corresponding target storage area of ​​the unit as the initial write position; Step S244: Based on the initial write position, write the data block corresponding to the computing unit, which is included in the sub-sequence corresponding to the processing core, into the target storage area corresponding to the computing unit; and Step S250: Distribute the contents of the corresponding target storage areas of the multiple computing units to the multiple computing units.

[0039] By applying the method 200 described above, the number of data blocks (i.e., the first quantity) that each processing core needs to allocate to each computing unit can be determined first for multiple processing cores. Based on this, each processing core can determine its initial write position based on the sum of the first quantities of the preceding processing cores, that is, to make way for the position occupied by the preceding processing cores in the target storage area. Thus, by pre-calculating the storage location offset, write operations that require serial waiting for storage location information can be transformed into independent write operations that can be implemented by each processing core. Each processing core does not need to wait for the write results of other processing cores, thereby improving the data block distribution efficiency by increasing the parallelism of multi-cores.

[0040] For example, when the input sequence contains eight data blocks t0, t1, t2, t3, t4, t5, t6, and t7, these data blocks need to be distributed to computing unit 0 and computing unit 1. Computing unit 0 needs to receive data blocks t0, t1, t3, t5, and t7, while computing unit 1 needs to receive data blocks t0, t1, t2, t4, and t6. According to the conventional serial processing method described above, if t0 to t3 are allocated to the first processing core and t4 to t7 are allocated to the second processing core, the second processing core must wait for the first processing core to complete all write operations before it can begin processing its own data blocks. However, according to method 200, the second processing core can pre-obtain the number of data blocks that the first processing core needs to distribute to each computing unit, and thus calculate its own initial write position. This allows it to reserve the space required by the first processing core in the corresponding target storage area of ​​each computing unit, without waiting for the first processing core's operation. For example, if the second processing core determines that the first processing core needs to occupy three positions (t0, t1, t3) in computing unit 0 and two positions (t0, t1, t2) in computing unit 1, it can directly start writing the data block it is responsible for from the fourth position of the corresponding target storage area in computing unit 0 and the third position of the corresponding target storage area in computing unit 1. Thus, the first and second processing cores can execute the operations of determining the data block write position and writing the data block to the target storage area in parallel, improving the data block distribution efficiency by increasing multi-core parallelism.

[0041] According to the scheme provided in this disclosure, a data block refers to a basic data unit that needs to be distributed and processed. For example, when method 200 is applied to the training or inference process of a hybrid expert model, a data block is a feature vector corresponding to each word in the model input information. An input sequence including multiple data blocks refers to a sequence formed by arranging these data blocks in the natural order of the original input. For example, an input text containing eight words, after being encoded, forms eight feature vectors that are arranged in the order in which the words appear in the text, thus constituting the input sequence t0 to t7 in the example above.

[0042] In method 200, the multiple computational units refer to the target computational modules or functional units to which data blocks need to be distributed. For example, when method 200 is applied to the training or inference process of a hybrid expert model, each computational unit may correspond to an expert subnetwork, with different expert subnetworks responsible for processing different content in the input sequence. In this example, the information of one or more computational units to be distributed to for each data block is pre-determined by the routing module of the hybrid expert model to indicate which one or more computational units each data block in the input sequence should be routed to for processing.

[0043] According to the solution provided in this disclosure, multiple processing cores refer to multiple independent processing units capable of executing data processing tasks in parallel, such as multiple GPUs or multiple computing cores in an NPU. In a multi-core parallel architecture, the input sequence can be split and distributed to different processing cores, with each processing core independently completing the distribution and writing operation of its assigned data block to the corresponding target storage area of ​​its respective computing unit, thereby making full use of hardware resources and improving data distribution efficiency.

[0044] In some examples, step S220, which involves splitting the input sequence into multiple subsequences corresponding to multiple processing cores, means dividing the input sequence into several consecutive segments according to the number of processing cores. Each processing core is assigned one of these segments as the subsequence it needs to process. For example, when the input sequence includes eight data blocks t0 to t7 as described above, and when using four processing cores for parallel processing, the input sequence t0 to t7 is sequentially divided into four segments: the first processing core handles t0 and t1, the second processing core handles t2 and t3, the third processing core handles t4 and t5, and the fourth processing core handles t6 and t7. By sequentially dividing the input sequence based on the amount of data, and then evenly distributing it to multiple processing cores, the load balancing among the processing cores can be improved, while maintaining the relative order of the data blocks in the original input sequence, which is beneficial for achieving more accurate data processing.

[0045] In some examples, step S230 determines a first quantity corresponding to each of the multiple processing cores and each of the multiple computing units. This means that for each processing core, the number of data blocks to be distributed to each computing unit in its assigned subsequence needs to be counted. For example, if the first processing core is responsible for subsequences t0 and t1, where t0 needs to be distributed to computing units 0 and 1, and t1 also needs to be distributed to computing units 0 and 1, then the first quantity for the first processing core in computing unit 0 is 2, and the first quantity in computing unit 1 is also 2. If the second processing core is responsible for subsequences t2 and t3, where t2 only needs to be distributed to computing unit 1, and t3 only needs to be distributed to computing unit 0, then the first quantity for the second processing core in computing unit 0 is 1, and the first quantity in computing unit 1 is also 1. By pre-counting the first quantity information corresponding to each processing core and each computing unit, each processing core can obtain global data block write location layout information before actually writing data, thereby improving data distribution efficiency.

[0046] In some examples, the first quantity counting operation in step S230 can be completed by each processing core independently. Each processing core only needs to traverse the data blocks in its own subsequence and count the number of data blocks to be distributed to each computing unit based on the information (i.e., routing information) of one or more computing units to which each data block is to be distributed. That is to say, step S230 can also be implemented based on multi-core parallelism, forming a complete parallel processing pipeline together with the parallel operation in the subsequent step S240, further improving the efficiency of data distribution based on multi-core parallelism.

[0047] According to some embodiments, method 200 further includes: writing a first quantity corresponding to each of the plurality of processing cores and each of the plurality of computing units into a second shared memory of the plurality of processing cores, wherein step S241 of obtaining the first quantity corresponding to each of the at least one preceding processing core and the computing unit includes: obtaining the first quantity corresponding to each of the at least one preceding processing core and the computing unit from the second shared memory. By applying the above technical means, the first quantity information statistically collected by each processing core is written into a second shared memory that can be accessed by the plurality of processing cores, so that when each processing core needs to obtain the first quantity of the preceding processing cores, it only needs to read the contents of the second shared memory, without the need for point-to-point communication between processing cores or reliance on centralized scheduling processes, effectively reducing the information transmission overhead between processing cores, thereby further improving the overall efficiency of data distribution.

[0048] In some examples, besides the aforementioned implementation method of using shared memory to transfer the first quantity information, other technical means can also be applied to transfer the first quantity information between different processing cores. For example, the first quantity information can be requested from other processing cores based on the message passing channel between processing cores, or each processing core can actively send its own first quantity information to other processing cores that need to obtain the information. As long as the transfer of the first quantity information between different processing cores can be achieved, this disclosure does not limit the specific method of obtaining the first quantity information.

[0049] According to some embodiments, multiple processing cores correspond to multiple core indices, which are determined based on the permutation ordinal numbers of multiple subsequences corresponding to the multiple processing cores in the input sequence. In step S241, in response to determining that at least one preceding processing core exists for the processing core, obtaining a first quantity corresponding to each of the at least one preceding processing core and the computing unit includes: in response to determining that the core index of the processing core is not the minimum value among the multiple core indices, sequentially performing a first operation starting with the processing core corresponding to the minimum value to obtain a first quantity corresponding to the at least one preceding processing core and the computing unit, wherein the number of first operations is determined based on the difference between the core index of the processing core and the minimum value.

[0050] By applying the above implementation method, core indices can be assigned to multiple processing cores based on the segmentation order of the input sequence. Furthermore, the first number of each preceding processing core is obtained by sequentially executing the first operation starting from the processing core with the smallest index. Each processing core can determine the number of times it needs to execute the first operation based on the difference between its own index value and the smallest index value (i.e., the index value of the processing core ranked first). This achieves self-organization based on core indices, allowing each processing core to independently and orderly complete the information acquisition steps without the need for a complex global coordination mechanism, reducing atomic operations and further improving data distribution efficiency.

[0051] In one example, multiple core indices are sequentially encoded based on natural numbers, starting with 0. In this example, when the indices corresponding to the multiple processing cores are 0, 1, 2, etc., each core only needs to determine the number of times the first operation needs to be performed based on its own index value. For example, processing core 1 needs to perform the first operation once to obtain the first quantity information corresponding to processing core 0, and processing core 2 needs to perform the first operation twice to obtain the first quantity information corresponding to processing core 0 and processing core 1 respectively.

[0052] In some examples, step S241 can also be implemented in other ways. For example, after each processing core completes its first count, it can share information containing its own core identifier and the first count with all other processing cores. Other processing cores can then filter out information belonging to their predecessor processing core from the shared information and accumulate it.

[0053] According to some embodiments, the first operation further includes: starting with zero, performing an accumulation operation based on the acquired first quantity corresponding to each preceding processing core and the computing unit, to obtain the sum of the first quantities corresponding to at least one preceding processing core and the computing unit. Thus, the total amount of data blocks for all preceding processing cores on the computing unit can be obtained more easily and efficiently by iteratively executing the operation of acquiring the first quantity corresponding to each preceding processing core and the operation of accumulating the acquired quantity information. In this case, the processing core starts from zero and loops through the operation of "acquiring the first quantity of a preceding processing core - accumulating the quantity to the current cumulative value". When all preceding processing cores have been traversed, the cumulative value is the total amount of data blocks that all preceding processing cores of that processing core need to distribute to the computing unit. This looping accumulation method does not require the use of an additional data storage area; each processing core only needs to maintain a variable for temporarily recording the current accumulated value to complete the calculation of the total amount of data blocks for all preceding processing cores, thereby more easily and efficiently determining the initial write position of the data blocks distributed by each processing core to each computing unit.

[0054] In some examples, the operation in step S242 of determining the initial write position in the target storage area corresponding to the computing unit based on the sum of the first quantities corresponding to the computing unit and at least one preceding processing core can be implemented as follows: the sum of the first quantities of the preceding processing cores of the current processing core is used as the write offset of the current processing core in the target storage area, so that there is storage space between the initial write position of the current processing core and the starting position of the target storage area that can accommodate data blocks whose total number is the sum of the aforementioned first quantities. For example, when a processing core has three preceding processing cores on the computing unit, and the first quantities of these three preceding processing cores are 2, 3, and 1 respectively, their sum is 6, then the processing core starts writing its own data block from the 7th position in the target storage area, so that the first 6 positions in the target storage area are reserved for the corresponding data blocks of the preceding processing cores. In one example, the size of the multiple data blocks to be distributed is known (e.g., corresponding to a fixed size, such as 8 bits or 16 bits), and the initial write position can be determined based on the sum of the corresponding first quantities of the preceding processing cores and the size of the data blocks.

[0055] Step S243 above describes the implementation of the processing core corresponding to the first subsequence in the initial sequence. That is, when a processing core does not have a preceding processing core, the starting position of the target storage area is taken as the initial write position of the processing core. Since the processing core is used to process the first subsequence in the initial sequence, that is, no other processing core writes data blocks to the target storage area of ​​the computing unit before the processing core. Therefore, its initial write position is the starting position of the target storage area, and no offset calculation is required.

[0056] According to some embodiments, step S244, based on the initial write position, writing the data block corresponding to the computing unit, which is included in the subsequence corresponding to the processing core, into the target storage area corresponding to the computing unit, includes: writing the first data block corresponding to the computing unit, which is included in the subsequence corresponding to the processing core, into the initial write position; determining the second write position based on the initial write position and the size of the first data block; and writing the second data block into the second write position in response to determining that the subsequence corresponding to the processing core also includes a second data block corresponding to the computing unit.

[0057] By applying the above-mentioned technical means to sequentially write the data blocks corresponding to each processing core to the target storage area corresponding to the current computing unit, the next write position can be calculated based on the size of each data block after each data block is written, starting from the initial write position. This achieves dynamic updating and maintenance of the write position for each write operation. Each processing core can achieve sequential writing of data blocks by tracking its own write progress, and can achieve simpler and more efficient data distribution without the need for additional global write pointers or complex offset management mechanisms.

[0058] The following example illustrates the dynamic data block writing process described above. In this example, when a processing core's initial write position on computing unit 0 is bit 5, the core needs to write three data blocks, each occupying one storage computing unit. First, the core writes the first data block to the 5th storage computing unit. Then, it adds the size of the already written data block (1) to bit 5 to obtain bit 6, using this as the next write position for the second data block. Next, it adds 1 to bit 6 to obtain bit 7, and then writes the third data block to bit 7. In this process, the core only needs to maintain an incrementing bit pointer to self-manage its own write operations, enabling simpler and more efficient continuous writing of multiple data blocks, thereby improving data distribution efficiency.

[0059] According to some embodiments, method 200 further includes: determining a sequence of computing units comprising a plurality of computing units; determining a second number of data blocks corresponding to each of the plurality of computing units based on information of one or more computing units to be distributed to each of the plurality of data blocks; and determining a target storage area corresponding to each of the plurality of computing units in a first shared memory of the plurality of computing units based on the ordinal number of the plurality of computing units in the sequence of computing units and the second number corresponding to each of the plurality of computing units, wherein step S250 of distributing the contents of the target storage area corresponding to each of the plurality of computing units to the plurality of computing units includes: causing each of the plurality of computing units to read the contents of the target storage area corresponding to that computing unit in the first shared memory.

[0060] By applying the above implementation method, when data block distribution is achieved using the first shared memory of multiple computing units, a target storage area is pre-planned for each computing unit based on the order of the computing units indicated by the computing unit sequence and the total number of data blocks that each computing unit should receive (i.e., the second quantity), thereby achieving sequential planning of the storage area. Since the target storage areas obtained by each computing unit are arranged contiguously in the shared memory, and the size of each storage area precisely matches the amount of data that the corresponding computing unit actually needs to receive, memory fragmentation can be effectively reduced, and the utilization efficiency of storage resources can be improved. Simultaneously, each computing unit can obtain its input data by directly reading its corresponding target storage area in the shared memory, achieving more efficient data transfer and thus improving data distribution efficiency.

[0061] In some examples, each of the multiple computing units includes a computing unit identifier, and the sequence of computing units can be determined based on the computing unit identifier. For example, when the computing unit identifier is in numerical form, the sequence of computing units can be determined by numerical sorting. As long as the target storage areas corresponding to each of the multiple computing units can be allocated and managed based on the sequence relationship between the multiple computing units contained in the computing unit sequence, this disclosure does not limit the specific method of determining the computing unit sequence.

[0062] In some examples, the target storage areas for multiple computing units are not limited to the shared memory described above. For instance, multiple processing cores can write the data blocks that need to be distributed to each computing unit directly into the dedicated storage area corresponding to each computing unit, so that each computing unit can directly read the required input data.

[0063] In some examples, the data distribution operation in step S250 can be implemented in various ways. For example, as described above, each computing unit can actively read the input data it needs. Alternatively, a data transfer module can be used to transfer data from the target storage area to the local cache within each computing unit based on the starting address and data size of the target storage area.

[0064] According to some embodiments, determining a target storage area corresponding to each of the multiple computing units in a first shared memory of the multiple computing units, based on the ordinal number of the multiple computing units in a computing unit sequence and a second quantity corresponding to each of the multiple computing units, includes: for each of the multiple computing units, in response to determining that at least one preceding computing unit of the computing unit exists in the computing unit sequence, determining the starting position of the target storage area corresponding to the computing unit based on the sum of the second quantities corresponding to each of the at least one preceding computing unit; and in response to determining that there is no preceding computing unit of the computing unit in the computing unit sequence, determining the starting position of the first shared memory as the starting position of the target storage area corresponding to the computing unit.

[0065] Therefore, by summing the second number of preceding computation units in the computation unit sequence, the starting position of the target storage area of ​​the current computation unit is determined, so that the target storage areas of each computation unit are closely arranged in the first shared memory according to the order of the computation unit sequence. This method of pre-calculating the starting position of each storage area based on global information enables subsequent processing cores to accurately locate the write area corresponding to each computation unit when writing data, thereby improving the utilization rate of hardware storage resources while achieving correct data distribution.

[0066] Figures 3A-3B A schematic diagram of a data distribution process according to an exemplary embodiment of the present disclosure is shown.

[0067] See Figure 3A In this example, the data distribution operation is implemented using two processing cores, indexed as thread0 and thread1 respectively. The input sequence consists of eight data blocks from t0 to t7 (i.e., ... Figure 3A As shown in t0, t1, t2, t3, t4, t5, t6, t7, the data blocks need to be distributed to three computational units numbered E0, E1, and E2 respectively (e.g., experts E0, E1, and E2 in the MoE model). Figure 3A As shown, the input sequence is split into two subsequences of length 4 and assigned to thread0 and thread1 for processing respectively. While obtaining the input sequence, thread0 and thread1 also obtain the computation unit allocation information corresponding to each data block. Specifically, t0 is assigned to computation units E0 and E2, t1 is assigned to computation units E1 and E2, t2 is assigned to computation units E0 and E2, t3 is assigned to computation units E0 and E1, t4 is assigned to computation units E1 and E2, t5 is assigned to computation units E1 and E2, t6 is assigned to computation units E0 and E2, and t7 is assigned to computation units E1 and E2.

[0068] Based on the above input information, thread0 and thread1 distribute the data blocks included in the input sequence through the following steps.

[0069] like Figure 3A As shown in the data table on the right, thread0 and thread1 each maintain a data set of length 3 (denoted as local_num_token_per_expert) to count the number of data blocks allocated to each computing unit (i.e., to count the first quantity information corresponding to each processing core and each computing unit as mentioned earlier). Thread0 iterates through t0-t3 and their corresponding computing unit allocation information to identify the expert number assigned to each data block, and updates the fields corresponding to different computing units in local_num_token_per_expert based on the identification results. For example, if t0 is allocated to computing units E0 and E2, then local_num_token_per_expert is updated to 1-0-1 to represent the first quantity information currently counted for E0, E1, and E2. If t1 is allocated to computing units E1 and E2, then local_num_token_per_expert is updated to 1-1-2. After the iteration, the result of local_num_token_per_expert for thread0 is 3-2-2, representing the first quantity information corresponding to thread0 and E0, E1, and E2 respectively.

[0070] Similarly, thread1 iterates through t4-t7 and their corresponding computational unit allocation information to identify the expert number assigned to each data block. Based on the identification results, it updates the fields corresponding to different computational units in local_num_token_per_expert, resulting in local_num_token_per_expert for thread1 being 1-3-4, which represents the first quantity information corresponding to thread1 and E0, E1, and E2 respectively.

[0071] Based on this, thread0 and thread1 can, for example, share their respective initial quantity information by sharing memory, thereby achieving... Figure 3B The data write operation is shown.

[0072] See Figure 3BIn this example, the target storage areas corresponding to E0, E1, and E2 are closely arranged. As described earlier, the starting position of each target storage area is determined based on the number of data blocks corresponding to each computing unit. In this example, the starting positions of the target storage areas corresponding to E0, E1, and E2 are 0, 4, and 9, respectively. Based on this, thread0 and thread1 each maintain a write position index data of length 3, using the starting position of the target storage area as the initial value, denoted as index_start, to indicate the write-back position of the data block. That is, the initial value of index_start for both thread0 and thread1 is 0-4-9.

[0073] Furthermore, thread0 and thread1 update their respective `index_start` values ​​using steps S241-S243 as described above. In this example, there is no preceding processing core for thread0; therefore, the initial write position of each computational unit corresponding to thread0 is the starting position of each target storage area, i.e., the initial value is 0-4-9. Simultaneously, thread1 updates its `index_start` value based on the initial value 0-4-9 and the first quantity information of its preceding processing core, thread0. Since thread1's preceding processing core is only thread0, the value of `local_num_token_per_expert` for thread0, 3-2-2, corresponds to the offset that thread1 should follow when writing data blocks to the target storage areas corresponding to each computational unit.

[0074] In this example, since the starting position of each target storage area is marked based on the global position information of the entire storage area, the index_start of thread0 and thread1 can indicate the write position of the data block based on the starting position of each target storage area and the above offset. thread1 adds 3-2-2 to the initial value 0-4-9 to obtain its own global write position information 3-6-12.

[0075] Based on the index_start values ​​obtained by thread0 and thread1 respectively, thread0 and thread1 can write data blocks to the target storage area of ​​each computing unit according to the write position information indicated by the index_start data, so as to obtain... Figure 3BThe data block write results shown can then be used to distribute data blocks to E0, E1, and E2 based on the contents of each target memory area. During this process, when a processing core writes multiple data blocks it is responsible for to the same computation unit, it increments the value corresponding to that computation unit in the `index_start` data, thus managing the write positions through auto-increment. For example, in thread1, t4, t5, and t7 all need to be written to the corresponding target memory of computation unit E1. Based on the aforementioned auto-increment mechanism, their write positions are 6, 7, and 8, respectively. Simultaneously, in thread1, t4, t5, t6, and t7 all need to be written to the corresponding target memory of computation unit t2. Based on the aforementioned auto-increment mechanism, their write positions are 12, 13, 14, and 15, respectively.

[0076] According to one aspect of this disclosure, a reasoning method based on a hybrid expert model is also provided. Figure 4 A flowchart of an inference method 400 for a hybrid expert model according to an exemplary embodiment of the present disclosure is shown. The hybrid expert model includes multiple expert sub-networks. Figure 4 As shown, method 400 includes: Step S401: Using method 200, distribute the multiple data blocks of the input sequence to be processed to multiple computing units corresponding to the multiple expert sub-networks; and Step S402: Implement the inference operation of the hybrid expert model using multiple computing units.

[0077] Therefore, by applying the above data distribution method 200 in the data block (e.g., token) distribution stage of the hybrid expert model inference, the tokens to be processed can be distributed to the corresponding computing units of different expert sub-networks more efficiently, thereby improving the inference speed of the hybrid expert model.

[0078] In some examples, multiple expert subnetworks of the hybrid expert model are deployed in different computing units. After the input data required by different expert subnetworks are distributed to the corresponding computing units of the expert subnetworks in step S401, the inference computation of the expert subnetworks can be performed by different computing units in step S402.

[0079] According to one aspect of this disclosure, a data distribution apparatus is also provided. Figure 5 A structural block diagram of a data distribution apparatus 500 according to an exemplary embodiment of the present disclosure is shown. The apparatus 500 is used to distribute a plurality of sequentially arranged data blocks to a plurality of computing units. Figure 5 As shown, the device 500 includes: The receiving unit 510 is configured to receive an input sequence comprising multiple data blocks and information on one or more computing units to be distributed to each of the multiple data blocks; The splitting unit 520 is configured to split the input sequence into multiple subsequences corresponding to multiple processing cores respectively; The first determining unit 530 is configured to determine a first quantity corresponding to each of the plurality of processing cores and each of the plurality of computing units, wherein the first quantity indicates the number of data blocks corresponding to the computing unit included in the subsequence corresponding to the processing core. Execution unit 540 is configured to execute operations in parallel using each of a plurality of processing cores. Execution unit 540 includes: The first acquisition subunit 541 is configured to, for each of a plurality of computing units, in response to determining that at least one preceding processing core exists for the processing core, acquire a first quantity corresponding to each of the at least one preceding processing core and the computing unit, wherein the subsequence corresponding to the at least one preceding processing core is located before the subsequence corresponding to the processing core in the input sequence. The first determining subunit 542 is configured to determine an initial write position in the target storage area corresponding to the computing unit based on the sum of a first number of at least one preceding processing core corresponding to the computing unit. The second determining subunit 543 is configured to determine the starting position of the corresponding target storage area of ​​the unit as the initial write position in response to determining that there is no preceding processing core of the processing core. Write subunit 544 is configured to write data blocks corresponding to the computing unit, including the subsequence corresponding to the processing core, to the target storage area corresponding to the computing unit based on the initial write position; and Distribution unit 550 is configured to distribute the contents of the respective target storage areas of multiple computing units to the multiple computing units.

[0080] According to some embodiments, multiple processing cores correspond to multiple core indices, which are determined based on the permutation ordinal numbers of multiple subsequences corresponding to the multiple processing cores in the input sequence. A first acquisition subunit 541 is configured to, in response to determining that the core index of the processing core is not the minimum among the multiple core indices, sequentially perform a first operation starting with the processing core corresponding to the minimum value to acquire a first quantity corresponding to at least one preceding processing core and the computing unit, wherein the number of first operations is determined based on the difference between the core index of the processing core and the minimum value.

[0081] According to some embodiments, the first operation further includes: starting with zero, performing an accumulation operation based on the acquired first quantity corresponding to each preceding processing core and the computing unit, to obtain the sum of the first quantity corresponding to the computing unit for at least one preceding processing core.

[0082] According to some embodiments, the apparatus 500 further includes: a second determining unit configured to determine a sequence of computing units including a plurality of computing units based on a computing unit identifier of each computing unit; a third determining unit configured to determine a second number of data blocks corresponding to each of the plurality of computing units based on information of one or more computing units to be distributed to each of the plurality of data blocks; and a fourth determining unit configured to determine a target storage area corresponding to each of the plurality of computing units in a first shared memory of the plurality of computing units based on the ordinal number of the plurality of computing units in the sequence of computing units and the second number corresponding to each of the plurality of computing units, wherein the distribution unit 550 is configured to cause each of the plurality of computing units to read the contents of the target storage area corresponding to that computing unit in the first shared memory.

[0083] According to some embodiments, the fourth determining unit is configured to: for each of the plurality of computing units, in response to determining that at least one preceding computing unit of the computing unit exists in the computing unit sequence, determine the starting position of the corresponding target storage area of ​​the computing unit based on the sum of the corresponding second quantities of the at least one preceding computing unit; and in response to determining that there is no preceding computing unit of the computing unit in the computing unit sequence, determine the starting position of the first shared memory as the starting position of the corresponding target storage area of ​​the computing unit.

[0084] According to some embodiments, the apparatus 500 further includes: a writing unit configured to write a first quantity corresponding to each of the plurality of processing cores and each of the plurality of computing units to a second shared memory of the plurality of processing cores, wherein a first acquisition subunit 541 is configured to acquire from the second shared memory the first quantity corresponding to each of at least one preceding processing core and the computing unit.

[0085] According to some embodiments, the write subunit 544 is configured to: write the first data block corresponding to the computing unit, which is included in the subsequence corresponding to the processing core, to an initial write position; determine the second write position based on the initial write position and the size of the first data block; and write the second data block to the second write position in response to determining that the subsequence corresponding to the processing core also includes a second data block corresponding to the computing unit.

[0086] According to one aspect of this disclosure, a hybrid expert model inference apparatus is also provided. Figure 6 A structural block diagram of an inference apparatus 600 for a hybrid expert model according to an exemplary embodiment of the present disclosure is shown. The hybrid expert model includes multiple expert sub-networks. Figure 6 As shown, the device 600 includes: Data distribution device 500 is configured to distribute multiple data blocks comprising the input sequence to be processed to multiple computing units corresponding to multiple expert subnetworks; and The inference unit 601 is configured to implement inference operations of a hybrid expert model using multiple computing units.

[0087] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0088] According to one aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data distribution method described above.

[0089] According to one aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform the data distribution method described above.

[0090] According to one aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above-described data distribution method.

[0091] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0092] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage computing unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. Computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0093] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to device 700. Input unit 706 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, a hard disk and an optical disk. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0094] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as data distribution methods. For example, in some embodiments, the data distribution method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the data distribution method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data distribution method by any other suitable means (e.g., by means of firmware).

[0095] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0099] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0100] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0101] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0102] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A data distribution method, wherein, The method is used to distribute multiple sequentially arranged data blocks to multiple computing units, and the method includes: Receive an input sequence including the plurality of data blocks and information on one or more computing units to be distributed to each of the plurality of data blocks; The input sequence is split into multiple sub-sequences, each corresponding to a different processing core. A first quantity is determined corresponding to each of the plurality of processing cores and each of the plurality of computing units, wherein the first quantity indicates the number of data blocks corresponding to the computing unit included in the subsequence corresponding to the processing core; The following operations are performed in parallel using each of the plurality of processing cores: For each of the plurality of computing units, In response to determining the existence of at least one preceding processing core of the processing core, Obtain a first quantity corresponding to each of the at least one preprocessing cores and the computing unit, wherein the subsequence corresponding to the at least one preprocessing core is located before the subsequence corresponding to the processing core in the input sequence; and Based on the sum of the first number of the at least one preprocessing core corresponding to the computing unit, the initial write position is determined in the target storage area corresponding to the computing unit. In response to the determination that a preceding processing core does not exist, the starting position of the corresponding target memory area of ​​the unit is determined as the initial write position; and Based on the initial write position, the data block corresponding to the computing unit, including the sub-sequence corresponding to the processing core, is written to the target storage area corresponding to the computing unit; and The contents of the respective target storage areas of the plurality of computing units are distributed to the plurality of computing units.

2. The method as described in claim 1, wherein, The plurality of processing cores correspond to a plurality of core indices, which are determined based on the permutation ordinal numbers of the plurality of subsequences corresponding to the plurality of processing cores in the input sequence. The step of obtaining a first quantity corresponding to each of the at least one preceding processing core and the computing unit in response to determining the existence of the processing core includes: In response to determining that the core index of the processing core is not the minimum value among the plurality of core indices, a first operation is sequentially performed starting with the processing core corresponding to the minimum value to obtain a first number corresponding to the at least one preceding processing core and the computing unit, wherein the number of the first operation is determined based on the difference between the core index of the processing core and the minimum value.

3. The method as described in claim 2, wherein, The first operation also includes: Starting with zero, an accumulation operation is performed based on the first quantity obtained corresponding to each preprocessing core and the computing unit to obtain the sum of the first quantity corresponding to the computing unit for the at least one preprocessing core.

4. The method according to any one of claims 1-3, further comprising: Determine the sequence of computing units that includes the plurality of computing units; Based on information about one or more computing units to be distributed to each of the plurality of data blocks, a second number of data blocks corresponding to each of the plurality of computing units is determined; as well as Based on the ordinal numbers of the plurality of computing units in the sequence of computing units and the second quantity corresponding to each of the plurality of computing units, a target storage area corresponding to each of the plurality of computing units is determined in the first shared memory of the plurality of computing units. The step of distributing the contents of the respective target storage areas of the plurality of computing units to the plurality of computing units includes: Each of the plurality of computing units is instructed to read the contents of the target storage area corresponding to that computing unit in the first shared memory.

5. The method of claim 4, wherein, The step of determining the target storage area corresponding to each of the plurality of computing units in the first shared memory of the plurality of computing units, based on the ordinal number of the plurality of computing units in the computing unit sequence and the second quantity corresponding to each of the plurality of computing units, includes: For each of the plurality of computing units, In response to determining that at least one preceding computational unit exists in the sequence of computational units, the starting position of the corresponding target storage area of ​​the computational unit is determined based on the sum of the corresponding second quantities of each of the at least one preceding computational unit; In response to determining that there is no preceding computing unit in the computing unit sequence, the starting position of the first shared memory is determined as the starting position of the corresponding target storage area of ​​the computing unit.

6. The method according to any one of claims 1-5, further comprising: A first quantity corresponding to each of the plurality of processing cores and each of the plurality of computing units is written into the second shared memory of the plurality of processing cores. Wherein, obtaining the first quantity corresponding to each of the at least one preprocessing cores and the computing unit includes: Obtain a first number corresponding to each of the at least one preceding processing cores and the computing unit from the second shared memory.

7. The method according to any one of claims 1-6, wherein, The step of writing the data block corresponding to the computing unit, which is part of the sub-sequence corresponding to the processing core, into the target storage area corresponding to the computing unit based on the initial write position includes: The first data block corresponding to the computing unit, which is included in the sub-sequence corresponding to the processing core, is written to the initial write position; Based on the initial write position and the size of the first data block, determine the second write position; and In response to determining that the subsequence corresponding to the processing core also includes a secondary data block corresponding to the computing unit, the secondary data block is written to the secondary write position.

8. A reasoning method based on a hybrid expert model, wherein, The hybrid expert model comprises multiple expert sub-networks, and the method includes: Using the method of any one of claims 1-7, the input sequence to be processed comprises multiple data blocks distributed to multiple computing units corresponding to the multiple expert sub-networks, respectively; and The inference operations of the hybrid expert model are implemented using the multiple computing units.

9. A data distribution device, wherein, The apparatus is used to distribute a plurality of sequentially arranged data blocks to a plurality of computing units, the apparatus comprising: The receiving unit is configured to receive an input sequence including the plurality of data blocks and information of one or more computing units to be distributed to each of the plurality of data blocks; The splitting unit is configured to split the input sequence into multiple sub-sequences corresponding to multiple processing cores respectively; The first determining unit is configured to determine a first quantity corresponding to each of the plurality of processing cores and each of the plurality of computing units, wherein the first quantity indicates the number of data blocks corresponding to the computing unit included in the subsequence corresponding to the processing core; An execution unit, configured to perform operations in parallel using each of the plurality of processing cores, the execution computation unit comprising: The first acquisition subunit is configured to, using each of the processing cores, for each of the plurality of computing units, in response to determining that at least one preceding processing core exists for the processing core, acquire a first quantity corresponding to each of the at least one preceding processing core and the computing unit, wherein the subsequence corresponding to the at least one preceding processing core is located in the input sequence preceding the subsequence corresponding to the processing core. The first determining subunit is configured to determine the initial write position in the target storage area corresponding to the computing unit based on the sum of a first number of at least one preceding processing core corresponding to the computing unit. The second determining subunit is configured to determine the starting position of the corresponding target memory area of ​​the unit as the initial write position in response to determining that there is no preceding processing core of the processing core. The write subunit is configured to, based on the initial write position, write the data block corresponding to the computing unit, including the subsequence corresponding to the processing core, to the target storage area corresponding to the computing unit; and The distribution unit is configured to distribute the contents of the respective target storage areas of the plurality of computing units to the plurality of computing units.

10. A reasoning apparatus based on a hybrid expert model, wherein, The hybrid expert model includes multiple expert sub-networks, and the device includes: The apparatus of claim 9 is configured to distribute a plurality of data blocks comprising an input sequence to be processed to a plurality of computing units corresponding to the plurality of expert subnetworks, respectively; and The inference unit is configured to implement the inference operation of the hybrid expert model using the plurality of computing units.

11. A chip, comprising: The apparatus as described in claim 9; or The apparatus as described in claim 10.

12. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

14. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-8.

Images