Data processing method, data processing apparatus, electronic device, and storage medium
By moving the data processing of the MoE model from the GPU to CPU shared memory for distribution and aggregation, the problems of high data transmission latency and resource consumption in multi-GPU environments are solved, achieving more efficient data processing and resource utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI BIREN TECH CO LTD
- Filing Date
- 2025-11-28
- Publication Date
- 2026-06-26
AI Technical Summary
Existing MoE models suffer from high data transmission latency and high computational resource consumption in multi-GPU environments, resulting in poor overall performance.
By distributing and aggregating data from the AI processor to the central processing unit's shared memory, the system leverages the efficiency of memory copying, reduces the consumption of GPU computing resources, and relies on high-speed interconnect systems.
It improves data transmission rate, reduces latency, enhances system resource utilization, saves GPU computing resources, and improves the computational performance of the MoE model.
Smart Images
Figure CN121210152B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this disclosure relate to the field of computer software, and more specifically to a data processing method, a data processing apparatus, an electronic device, and a storage medium. Background Technology
[0002] Mixture of Experts (MoE) models are architectures that enhance model capabilities through sparse activation, with the core idea of "divide and conquer." Instead of using a single massive neural network to process all inputs, a series of "expert" networks are designed, each specializing in a particular type of data. A gate network then determines which expert(s) to assign the data to based on its characteristics. Experts are relatively small neural networks (e.g., feedforward neural networks), each with its own parameters and focused on learning specific patterns or domains within the data. The gate network acts as a router, receiving the input data and outputting a probability distribution representing the weights each expert places on that data.
[0003] When a model is too large to fit into the memory of a single GPU, we need to split the model across multiple GPUs. Expert Parallelism (EP) is a model parallelism strategy specifically designed for MoE models. It places different "experts" on different GPUs / computing devices, supporting the operation of the MoE model through parallel collaboration between the devices. Summary of the Invention
[0004] At least one embodiment of this disclosure provides a data processing method for a central processing unit in a first server of a computer system comprising multiple servers, wherein the first server further comprises multiple artificial intelligence processors. The data processing method includes: receiving the first data from the first artificial intelligence processor to a shared memory of the central processing unit in response to the first artificial intelligence processor receiving the input first data; sending at least a first portion of the first data from the shared memory of the central processing unit to the first artificial intelligence processor in response to computational processing required for the first data; and sending at least a second portion of the first data to at least one second artificial intelligence processor other than the first artificial intelligence processor among the multiple artificial intelligence processors of the first server.
[0005] For example, the data processing method provided in at least one embodiment of this disclosure further includes: receiving first result data after a first artificial intelligence processor has processed at least a first part and second result data after at least a second artificial intelligence processor has processed at least a second part by a shared memory of a central processing unit, and aggregating the first result data and the second result data.
[0006] For example, in the data processing method provided in at least one embodiment of this disclosure, at least a first part is local data, at least a second part is cross-card data, and at least a first part of the first data is sent to a first artificial intelligence processor, and at least a second artificial intelligence processor other than the first artificial intelligence processor is sent to at least a plurality of artificial intelligence processors of a first server, including: performing routing decision calculation to obtain routing information of the first data; and sending local data to the first artificial intelligence processor and cross-card data to at least one second artificial intelligence processor according to the routing information.
[0007] For example, in the data processing method provided in at least one embodiment of this disclosure, a computer system is configured to deploy a hybrid expert model, multiple servers are configured to deploy multiple expert models, and multiple artificial intelligence processors in a first server are respectively configured to deploy at least one expert model to process the received data to be processed.
[0008] For example, in a data processing method provided in at least one embodiment of this disclosure, receiving first data from a first artificial intelligence processor to the shared memory of a central processing unit includes: receiving first data from the first artificial intelligence processor to the shared memory of a central processing unit via a PCIe bus.
[0009] At least one embodiment of this disclosure also provides another data processing method for a first artificial intelligence processor among a plurality of artificial intelligence processors in a first server of a computer system including multiple servers, wherein the first server further includes a central processing unit (CPU). The data processing method includes: sending first data input to the first artificial intelligence processor to a shared memory of the CPU; in response to receiving at least a first portion of the first data sent from the shared memory of the CPU, processing the at least a first portion of the first data to obtain first result data, and sending the first result data back to the shared memory of the CPU for aggregation.
[0010] For example, in a data processing method provided in at least one embodiment of this disclosure, sending first data input to a first artificial intelligence processor to the shared memory of a central processing unit includes: sending the first data to the shared memory of the central processing unit via a PCIe bus.
[0011] For example, at least one embodiment of the data processing method provided in this disclosure further includes: receiving input data, wherein the input data includes local server data and cross-server data, and the local server data is first data; the method further includes: performing routing decision calculation through a gating network to obtain routing information of the input data; sending the cross-server data to a second server of the computer system according to the routing information; and having the first data processed by the first server.
[0012] For example, in the data processing method provided in at least one embodiment of this disclosure, sending cross-server data to a second server of a computer system includes: sending the cross-server data via the PCIe bus through the network card of the first server and the network card of the second server.
[0013] For example, in the data processing method provided in at least one embodiment of this disclosure, a computer system is configured to deploy a hybrid expert model, multiple servers are configured to deploy multiple expert models, and multiple artificial intelligence processors in a first server are respectively configured to deploy at least one expert model to process the received data to be processed.
[0014] For example, in the data processing method provided in at least one embodiment of this disclosure, a first artificial intelligence processor is deployed with at least two expert models to process at least a first part of the first data to obtain first result data, including: processing at least a first part of the first data through at least two expert models to obtain first sub-result data and second sub-result data respectively, and aggregating the first sub-result data and second sub-result data to obtain the first result data.
[0015] At least one embodiment of this disclosure also provides a data processing apparatus for a central processing unit in a first server of a computer system including multiple servers, wherein the first server further includes multiple artificial intelligence processors, the data processing comprising: a receiving module configured to receive the first data from the first artificial intelligence processor to a shared memory of the central processing unit in response to the first artificial intelligence processor receiving input first data; and a distributing module configured to send at least a first portion of the first data from the shared memory of the central processing unit to the first artificial intelligence processor in response to computational processing required for the first data, and to send at least a second portion of the first data to at least one second artificial intelligence processor other than the first artificial intelligence processor among the multiple artificial intelligence processors of the first server.
[0016] At least one embodiment of this disclosure also provides another data processing apparatus for a first artificial intelligence processor among a plurality of artificial intelligence processors in a first server of a computer system including multiple servers, wherein the first server further includes a central processing unit. The data processing apparatus includes: a sending module configured to send first data input to the first artificial intelligence processor to a shared memory of the central processing unit; and a return module configured to, in response to receiving at least a first portion of the first data sent from the shared memory of the central processing unit, process at least a first portion of the first data to obtain first result data, and return the first result data back to the shared memory of the central processing unit for aggregation.
[0017] At least one embodiment of this disclosure also provides an electronic device, including the data processing apparatus provided in the at least one embodiment above.
[0018] At least one embodiment of this disclosure also provides another electronic device, including: at least one processor; and a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to perform a data processing method according to the at least one embodiment described above.
[0019] At least one embodiment of this disclosure also provides a non-transitory storage medium for storing computer-executable instructions, wherein when the computer-executable instructions are executed by a computer, the data processing method according to the at least one embodiment described above is performed. Attached Figure Description
[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.
[0021] Figure 1 This is a schematic diagram of EP communication in a MoE model of a computer system.
[0022] Figure 2 A flowchart illustrating a data processing method for the central processing unit side, provided for at least one embodiment of this disclosure;
[0023] Figure 3 A flowchart illustrating yet another data processing method for an artificial intelligence processor side, provided as at least one embodiment of this disclosure;
[0024] Figure 4 A schematic diagram illustrating the interaction of EP communication in a MoE model of a computer system provided for at least one embodiment of this disclosure;
[0025] Figure 5A schematic diagram of a data processing apparatus provided in at least one embodiment of this disclosure;
[0026] Figure 6 A schematic diagram of yet another data processing apparatus provided for at least one embodiment of the present disclosure;
[0027] Figure 7 A schematic diagram of an electronic device provided for at least one embodiment of this disclosure;
[0028] Figure 8 A schematic diagram of a storage medium provided for at least one embodiment of this disclosure; and
[0029] Figure 9 This is a schematic diagram of another electronic device provided in at least one embodiment of the present disclosure. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.
[0031] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.
[0032] Current MoE models are evolving towards a greater number of experts and smaller individual experts, with Expert Parallelism (EP) becoming the mainstream approach.
[0033] Figure 1 This is a schematic diagram of EP communication in a MoE model of a computer system.
[0034] like Figure 1As shown, in this MoE model, input data undergoes two processes: distribution and aggregation. These involve local communication (data distribution within the same GPU), cross-GPU communication (data distribution between GPUs within a server), and cross-server communication (GPU communication between different servers). Specifically, input data is sent to GPU0 in server 1. The gating network of GPU0 performs routing decision calculations on the input data to obtain its routing information. Then, based on this routing information, local data is sent to experts 0 and 1 in GPU0 for processing; cross-GPU data is sent to experts 2 and 3 in GPU1 in server 1 for processing; and cross-server data is sent to experts 4 in GPU2 and 6 in GPU3 in server 2 for processing. Finally, all the results processed by experts 0, 1, 2, 3, 4, and 6 are aggregated back to GPU0 (e.g., in video memory) for further aggregation processing to obtain the output results corresponding to the input data.
[0035] However, Figure 1 The data between the four onboard GPUs (Expert 0-Expert 7) shown directly relies on a high-speed interconnect system (such as the PCIe bus). Insufficient interconnect bandwidth or on a non-fully interconnected system increases data transmission latency, resulting in poor overall performance. Furthermore, during the data interaction between these eight cards, the GPUs' own computing resources are also consumed, further reducing the computing performance of the relevant GPUs.
[0036] In view of the above, at least one embodiment of the present disclosure provides a data processing method for a central processing unit in a first server of a computer system including multiple servers, wherein the first server further includes multiple artificial intelligence processors. The data processing method includes: receiving the first data from the first artificial intelligence processor to a shared memory of the central processing unit in response to the first artificial intelligence processor receiving the input first data; sending at least a first portion of the first data from the shared memory of the central processing unit to the first artificial intelligence processor in response to computational processing required for the first data; and sending at least a second portion of the first data to at least one second artificial intelligence processor other than the first artificial intelligence processor among the multiple artificial intelligence processors of the first server.
[0037] Meanwhile, at least one embodiment of this disclosure also provides a data processing apparatus corresponding to the above-described data processing method.
[0038] The data processing method provided in at least one embodiment of this disclosure moves input data from an artificial intelligence processor (e.g., a GPU) to CPU shared memory for distribution. Since the shared memory is shared by multiple artificial intelligence processors within a server, data communication between these processors is essentially data copying within the shared memory. The memory copying speed of 60-75 GB / s is much faster than the transfer speed between artificial intelligence processors via PCIe. Furthermore, the computing resources of the artificial intelligence processors can be freed up to perform other computing tasks, thus fully utilizing the resources of the CPU and the artificial intelligence processor (e.g., a GPU) and improving system resource utilization.
[0039] The data processing method of this disclosure will be described below through specific embodiments, specifically through two data processing methods in which the execution subject is a central processing unit and an artificial intelligence processor.
[0040] First, let's explain the case where the execution entity is the central processing unit.
[0041] Figure 2 The flowchart of a data processing method for the central processing unit side provided in at least one embodiment of the present disclosure specifically includes steps S110-S120.
[0042] Step S110: In response to the first artificial intelligence processor among a plurality of artificial intelligence processors receiving the first input data, the first data is received from the first artificial intelligence processor to the shared memory of the central processing unit.
[0043] Step S120: In response to the computational processing required for the first data, at least a first portion of the first data is sent from the shared memory of the central processing unit to a first artificial intelligence processor, and at least a second portion of the first data is sent to at least one second artificial intelligence processor other than the first artificial intelligence processor among the plurality of artificial intelligence processors of the first server.
[0044] For step S110, for example, the multiple artificial intelligence processors can be graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), tensor processing units (TPUs), neural processing units (NPUs), intelligence processing units (IPUs), or language processing units (LPUs). Unless otherwise specified, the following description uses GPUs as the artificial intelligence processors.
[0045] For example, when the first GPU receives the first input data (e.g., the first data is data that the first server where the first GPU resides needs to process), the CPU in the first server receives the first data from the first GPU and sends it to the CPU's shared memory for storage. For example, the first data can be received and sent to the CPU's shared memory via the PCIe bus.
[0046] For step S120, for example, since the first data needs to be processed by different GPUs in the first server, the CPU will send at least a first portion of the first data to the first GPU for processing, and at least a second portion of the first data to at least one second GPU other than the first GPU in the first server for processing. It should be noted that the CPU will send the target data corresponding to the first data to the corresponding GPU in the first server for processing according to the computational requirements carried in the first data, and the specific number of GPUs is not limited in this embodiment.
[0047] Thus, at least one embodiment of this disclosure moves the entire process of the first GPU distributing input data in the first server to the CPU in the first server for processing. By freeing up a shared memory area in the CPU to store the corresponding input data, efficient data distribution between the GPUs in the machine is achieved, saving the computing resources of the first GPU and improving the overall resource utilization of the CPU and the first GPU.
[0048] For example, in one possible implementation, the above data processing method further includes: receiving first result data after a first artificial intelligence processor has processed at least a first part and second result data after at least a second artificial intelligence processor has processed at least a second part via the shared memory of the central processing unit, and aggregating the first result data and the second result data.
[0049] refer to Figure 1 For example, after processing the data, GPU0 and GPU1 within the machine will both send the result data to GPU0 for unified aggregation. To avoid consuming GPU computing resources by performing the result data aggregation process in the GPU, for example, at least one embodiment of this disclosure still uses the CPU's shared memory to uniformly receive the first result data after the first GPU has processed at least a first part and the second result data after at least a second GPU has processed at least a second part, and performs aggregation processing in the CPU's shared memory, further realizing the release of the first GPU's computing resources.
[0050] It should be noted that the CPU's shared memory will receive the result data after all GPUs in the first server have processed the corresponding part of the data in the first data.
[0051] For example, in one possible implementation, at least a first portion of the first data is local data, and at least a second portion is cross-card data. Sending at least a first portion of the first data to a first artificial intelligence processor and sending at least a second artificial intelligence processor other than the first artificial intelligence processor to a plurality of artificial intelligence processors of the first server includes: performing routing decision calculations to obtain routing information of the first data; sending local data to the first artificial intelligence processor and cross-card data to at least one second artificial intelligence processor based on the routing information.
[0052] For example, after receiving the first data (i.e., local server data) from its shared memory, the CPU retrieves the first data from the shared memory and sends it to the distribution engine. The distribution engine then performs routing decisions to calculate the routing information for the first data. This routing information indicates the specific target location (i.e., the target GPU) where the local data and cross-card data in the first data need to be transmitted. Subsequently, the CPU sends the local data to the first GPU for processing based on the calculated routing information, and sends the cross-card data to at least one second GPU for processing.
[0053] For example, the data processing method mentioned in at least one of the above embodiments is used in a computer system that includes multiple servers, including the first server mentioned above.
[0054] For example, in one possible implementation, the computer system described above is configured to deploy a hybrid expert model (MoE), wherein multiple servers are configured to deploy multiple expert models, and multiple artificial intelligence processors in a first server are respectively configured to deploy at least one expert model to process the received data to be processed.
[0055] For example, when a computer system is configured to deploy a hybrid expert model (MoE), its multiple servers each deploy multiple expert models to perform EP communication (including local communication, cross-card communication, and cross-server communication). The multiple AI processors (e.g., a first GPU and at least one second GPU) in the first server are each configured to deploy at least one expert model to process the data distributed to them. For instance, if the expert models deployed on the first GPU include expert 0 and expert 1, then expert 0 and expert 1 will process the local data distributed to the first GPU; if the expert models deployed on the second GPU include expert 2 and expert 3, then expert 2 and expert 3 will process the cross-card data distributed to the second GPU.
[0056] In at least one embodiment of this disclosure, the distribution of input data of the first GPU in the first server is achieved by CPU shared memory copying. Since the speed of memory copying is higher than the bandwidth of PCIe data transmission and the latency of memory copying is also lower, the EP communication latency in the first server is reduced, thereby improving the computational performance of the MoE model.
[0057] The data processing method described above, provided in at least one embodiment of this disclosure, is applied to the CPU side. By moving input data from an artificial intelligence processor (e.g., a GPU) to the CPU's shared memory for on-demand unified data distribution and result aggregation, it significantly saves the computing resources of the artificial intelligence processor (e.g., a GPU) to perform other computing tasks while improving data transmission rate and reducing latency. This fully utilizes the resources of both the CPU and the artificial intelligence processor (e.g., a GPU), improving system resource utilization. Furthermore, it no longer relies on high-speed interconnect systems within servers.
[0058] The following section explains the implementation of artificial intelligence processors.
[0059] Figure 3 The flowchart of a data processing method for an artificial intelligence processor side provided in at least one embodiment of the present disclosure specifically includes steps S210-S220.
[0060] Step S210: Send the first data input to the first artificial intelligence processor to the shared memory of the central processing unit.
[0061] Step S220: In response to receiving at least a first portion of the first data sent from the shared memory of the central processing unit, the at least a first portion of the first data is processed to obtain first result data, and the first result data is sent back to the shared memory of the central processing unit for aggregation.
[0062] Similarly, in step S210, the first artificial intelligence processor can be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), an intelligence processing unit (IPU), or a language processing unit (LPU). Unless otherwise specified, the following description will use GPU as the artificial intelligence processor.
[0063] For example, the first GPU might send the first input data it receives (e.g., the first data is data that the first server, where the first GPU resides, needs to process) to the CPU's shared memory for storage. This first data could be sent to the CPU's shared memory via, for example, through the PCIe bus.
[0064] For step S220: For example, if the first GPU receives at least a first part of the first data sent by the CPU shared memory (e.g., the local data mentioned above), the first GPU will process the at least a first part of the data to obtain first result data, and finally send the first result data back to the CPU shared memory for aggregation processing.
[0065] It should be noted that the first GPU here can be any GPU in the first server, for example, the only difference being the specific part of the first data to be processed. The embodiments of this disclosure do not limit this.
[0066] The data processing method provided in at least one embodiment of this disclosure actively moves data distribution and result aggregation to the shared memory of the central processing unit by an artificial intelligence processor (e.g., GPU), saving the computing resources of the artificial intelligence processor (e.g., GPU) to perform other computing tasks and improving the utilization rate of computing resources.
[0067] For example, in one possible implementation, for cross-server data, the above data processing method further includes: receiving input data, wherein the input data includes local server data and cross-server data, and the local server data is the first data; the method further includes: performing routing decision calculation through a gating network to obtain routing information of the input data; sending the cross-server data to a second server of the computer system according to the routing information; and having the first data processed by the first server.
[0068] For example, before processing the first data, the first GPU first receives input data (Token data), which includes local server data (e.g., data to be processed by various GPUs within the first server) and cross-server data (e.g., data to be processed by some GPUs in the second server). In this case, the first GPU's gate network performs routing decision calculations on the input data to obtain overall routing information for the input data. This routing information represents the specific target locations (i.e., target servers) where the local server data and cross-server data in the input data need to be transmitted. Subsequently, the first GPU's gate network sends the local server data (i.e., the first data) to the first server corresponding to the first GPU for processing according to the calculated routing information, and sends the cross-server data to the second server among the multiple servers included in the computer system for processing.
[0069] It should be noted that after cross-server data is sent to the second server, it will also be processed in the same way as the data in the first server. The cross-server data will be distributed and aggregated on demand through the CPU shared memory in the second server. The specific operation process will not be described here.
[0070] For example, in one possible implementation, sending cross-server data to a second server in a computer system includes: sending the cross-server data via a PCIe bus through the network interface card (NIC) of the first server and the NIC of the second server.
[0071] For example, when sending cross-server data, the data is first sent to the network card of the first server via the PCIe bus, and then transmitted over the network to the network card of the second server for processing.
[0072] For example, the data processing method mentioned in at least one of the above embodiments is used in a computer system that includes multiple servers, including the first server and the second server mentioned above.
[0073] For example, in one possible implementation, the computer system described above is configured to deploy a hybrid expert model (MoE), wherein multiple servers are configured to deploy multiple expert models, and multiple artificial intelligence processors in a first server are respectively configured to deploy at least one expert model to process the received data to be processed.
[0074] For example, the first artificial intelligence processor is deployed with at least two expert models to process at least a first part of the first data to obtain first result data, including: processing at least a first part of the first data through at least two expert models to obtain first sub-result data and second sub-result data respectively, and aggregating the first sub-result data and second sub-result data to obtain the first result data.
[0075] For example, the expert model deployed on the first GPU includes at least expert 0 and expert 1. Expert 0 and expert 1 process at least a first portion (e.g., the aforementioned local data) of the first data distributed to the first GPU. Expert 0 processes the required data in this portion to obtain first sub-result data, and expert 1 processes the required data in this portion to obtain second sub-result data. The first and second sub-result data are pre-aggregated in the first GPU to obtain first result data, which is then sent back to the CPU shared memory for overall aggregation. In this way, without excessively consuming the computing resources of the first GPU, the first GPU performs pre-aggregation processing on the sub-result data calculated by its own expert model, reducing the aggregation pressure on the CPU shared memory and improving the overall resource utilization of the CPU and the first GPU.
[0076] However, it should be noted that the embodiments disclosed herein do not limit the number of experts included in the first GPU; the specific number depends on the actual situation.
[0077] The data processing method described above, provided in at least one embodiment of this disclosure, is applied to an artificial intelligence processor (e.g., a GPU). By actively moving data distribution and result aggregation to the shared memory of the central processing unit for processing, it significantly saves its own computing resources to perform other computing tasks while improving data transmission rate and reducing latency, thereby improving the utilization rate of computing resources. Furthermore, it no longer relies on the high-speed interconnect system within the server.
[0078] Figure 4 This is a schematic diagram of EP communication interaction in a MoE model of a computer system provided for at least one embodiment of the present disclosure.
[0079] like Figure 4 The computer system includes server 1 and server 2. Server 1 includes GPU0 and GPU1, and server 2 includes GPU2. The expert models deployed on GPU0 include expert 0 and expert 1, the expert models deployed on GPU1 include expert 2 and expert 3, and the expert models deployed on GPU2 include expert 4 and expert 5.
[0080] First, input data is transmitted to the gating network of GPU0. After routing calculation, the gating network sends the local server data (i.e., the data that each GPU in server 1 needs to process) to the CPU shared memory via Device to Host (D2H) over the PCIe bus. Then, the CPU reads the local server data and performs routing calculations on it through the CPU distribution engine. The local data is then sent to Expert 0 and Expert 1 on GPU0 for processing via Host to Device (H2D). Cross-card data is sent to Expert 2 and Expert 3 on GPU1 for processing via Host to Device (H2D). Finally, the results processed by Expert 0 and Expert 3 are sent back to the CPU shared memory for aggregation. Simultaneously, the gating network also sends cross-server data to server 2 for corresponding data processing.
[0081] The data processing method provided in at least one embodiment of this disclosure moves the data transmission process of the GPU in the MoE model to the CPU shared memory in the server. By uniformly executing on-demand data distribution and result aggregation through the CPU shared memory, the computing resources of the GPU are saved and the overall resource utilization of the CPU and GPU is improved.
[0082] At least one embodiment of this disclosure also provides a data processing apparatus. Figure 5 This is a schematic diagram of a data processing apparatus provided for at least one embodiment of the present disclosure.
[0083] like Figure 5 As shown, the data processing device 300 includes a receiving module 310 and a distributing module 320. The data processing device 300 can be used to implement the above-described data processing method for the CPU side.
[0084] The receiving module 310 is configured to receive the first data from the first artificial intelligence processor (one of a plurality of artificial intelligence processors) into the shared memory of the central processing unit (CPU) in response to the first artificial intelligence processor receiving the first input data. The specific operation process is described in the above-mentioned step S110, and will not be repeated here.
[0085] The distribution module 320 is configured to, in response to computational processing required for the first data, send at least a first portion of the first data from the shared memory of the central processing unit to a first artificial intelligence processor, and send at least a second portion of the first data to at least one second artificial intelligence processor other than the first artificial intelligence processor among a plurality of artificial intelligence processors of the first server. The specific operation process is described in the above description of step S120, and will not be repeated here.
[0086] At least one embodiment of this disclosure also provides yet another data processing apparatus. Figure 6 This is a schematic diagram of another data processing apparatus provided for at least one embodiment of the present disclosure.
[0087] like Figure 6 As shown, the data processing device 400 includes a sending module 410 and a return module 420. The data processing device 400 can be used to implement the above-mentioned data processing method for the artificial intelligence processor side.
[0088] The sending module 410 is configured to send the first data input to the first artificial intelligence processor to the shared memory of the central processing unit. The specific operation process is described in the above description of step S210, and will not be repeated here.
[0089] The return module 420 is configured to, in response to receiving at least a first portion of first data sent from the shared memory of the central processing unit, process at least a first portion of the first data to obtain first result data, and return the first result data to the shared memory of the central processing unit for aggregation. The specific operation process is described in the above description of step S220, and will not be repeated here.
[0090] It should be noted that, for clarity and brevity, the embodiments of this disclosure do not provide all the constituent units of the data processing device 300 or data processing device 400 described above. To achieve the necessary functions of the data processing device 300 or data processing device 400, those skilled in the art can provide or set other constituent units (not shown) according to specific needs, and the embodiments of this disclosure do not impose any limitations on this.
[0091] At least one embodiment of this disclosure also provides an electronic device, including the data processing apparatus 300 or 400 provided in any of the above embodiments. For example, the electronic device can be a computer system, a large computing server, or other computing equipment.
[0092] At least one embodiment of this disclosure also provides an electronic device. Figure 7 This is a schematic diagram of an electronic device provided in at least one embodiment of the present disclosure.
[0093] For example, such as Figure 7 As shown, the electronic device 500 includes at least one processor 510 and a memory 520. The memory 520 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 510 is used to execute the computer program instructions, which, when executed by the processor 510, perform the data processing method provided in any embodiment of this disclosure. The memory 520 and the processor 510 can be interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0094] Processor 510 can be a central processing unit (CPU), tensor processor (TPU), network processor (NP), or graphics processing unit (GPU) with data processing and / or program execution capabilities. It can also be a digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. For example, the CPU can be based on x86 or ARM architectures. Processor 510 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 500 to perform desired functions.
[0095] For example, memory 520 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 510 may run one or more computer program modules to implement various functions of electronic device 500. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.
[0096] At least one embodiment of this disclosure also provides a computer storage medium for storing non-transitory computer program executable code (e.g., computer executable instructions). When executed by a computer (e.g., including one or more processors), the non-transitory computer program executable code can implement the data processing method of any embodiment of this disclosure.
[0097] Figure 8 This is a schematic diagram of a storage medium provided in at least one embodiment of this disclosure. For example... Figure 8 As shown, storage medium 600 stores non-transitory computer-executable instructions 610.
[0098] For example, one or more computer instructions may be stored on the storage medium 600. Some of the computer instructions stored on the storage medium 600 may be, for example, instructions for implementing one or more steps in the data processing method described above.
[0099] For example, the storage medium may include the storage component of a tablet computer, the hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.
[0100] Figure 9 A schematic diagram of another electronic device provided in at least one embodiment of the present disclosure is shown. Figure 9 The illustrated electronic device 700 is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0101] like Figure 9 As shown, in some examples, electronic device 700 includes a processing unit (e.g., central processing unit, graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 702 or a program loaded from storage device 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the computer system. The processing unit 701, ROM 702, and RAM 703 are connected via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.
[0102] For example, the following components can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709, such as network interface cards like LAN cards and modems, etc. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data and perform communication processing via networks such as the Internet. Drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 710 as needed so that computer programs read from them can be installed into storage device 708 as needed.
[0103] Although Figure 9 An electronic device 700 including various devices is shown; however, it should be understood that implementation or inclusion of all shown devices is not required. More or fewer devices may be implemented or included alternatively.
[0104] For example, the electronic device 700 may further include a peripheral interface (not shown in the figure). This peripheral interface can be various types of interfaces, such as a USB interface, a Lightning interface, etc. The communication device 709 can communicate wirelessly with a network and other devices, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.
[0105] For example, the electronic device 700 may include any device such as a mobile phone, tablet computer, laptop computer, e-book, game console, television, digital photo frame, navigator, server, etc., or any combination of data processing device and hardware. The embodiments disclosed herein do not limit this.
[0106] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, such modifications or improvements made without departing from the spirit of the present disclosure are all within the scope of protection claimed by the present disclosure.
[0107] The following points should be noted regarding this disclosure:
[0108] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.
[0109] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.
[0110] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.
[0111] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.
Claims
1. A data processing method, characterized in that, A central processing unit in a first server of a computer system comprising multiple servers, wherein the first server further comprises multiple artificial intelligence processors, the computer system being configured to deploy a hybrid expert model, and the multiple servers being configured to deploy multiple expert models. The data processing method includes: In response to a first artificial intelligence processor among the plurality of artificial intelligence processors receiving input first data, the first data is received from the first artificial intelligence processor to the shared memory of the central processing unit, wherein the shared memory is shared by the plurality of artificial intelligence processors; In response to computational processing required for the first data, at least a first portion of the first data is sent from the shared memory of the central processing unit to the first artificial intelligence processor, and at least a second portion of the first data is sent to at least one second artificial intelligence processor among the plurality of artificial intelligence processors of the first server, excluding the first artificial intelligence processor. The data processing method further includes: The central processing unit receives first result data after at least one expert model in the first artificial intelligence processor has processed the at least first part, and second result data after at least one expert model in the at least one second artificial intelligence processor has processed the at least second part, via the shared memory of the central processing unit, and aggregates the first result data and the second result data. The at least second portion is cross-card data, and the step of sending at least the second portion of the first data to at least one second artificial intelligence processor other than the first artificial intelligence processor among the plurality of artificial intelligence processors of the first server includes: The routing information of the first data is obtained by performing routing decision calculations through the distribution engine of the central processing unit; The cross-card data is sent to the at least one second artificial intelligence processor according to the routing information.
2. The data processing method according to claim 1, characterized in that, The first part is local data. Sending at least a first portion of the first data to the first artificial intelligence processor includes: The local data is sent to the first artificial intelligence processor by the distribution engine based on the routing information.
3. The data processing method according to claim 1, characterized in that, The multiple artificial intelligence processors in the first server are each configured to deploy at least one expert model to process the received data to be processed.
4. The data processing method according to claim 1, characterized in that, The step of receiving the first data from the first artificial intelligence processor to the shared memory of the central processing unit includes: The first data is received from the first AI processor via the PCIe bus and sent to the shared memory of the central processing unit.
5. A data processing method, characterized in that, A first artificial intelligence processor is used in a first server of a computer system comprising multiple servers, wherein the first server further comprises a central processing unit, the computer system is configured to deploy a hybrid expert model, and the multiple servers are configured to deploy multiple expert models. The data processing method includes: The first data input to the first artificial intelligence processor is sent to the shared memory of the central processing unit, wherein the shared memory is shared by the plurality of artificial intelligence processors; In response to receiving at least a first portion of the first data sent from the shared memory of the central processing unit, the at least first portion of the first data is processed by at least one expert model in the first artificial intelligence processor to obtain first result data. The first result data is then sent back to the shared memory of the central processing unit and aggregated with second result data obtained by at least one expert model in at least one second artificial intelligence processor (excluding the first artificial intelligence processor) processing at least a second portion of the first data. The at least second part is cross-card data. The routing information of the first data is obtained by performing routing decision calculation through the distribution engine of the central processing unit, and the cross-card data is sent to the at least one second artificial intelligence processor according to the routing information.
6. The data processing method according to claim 5, characterized in that, The step of sending the first data input to the first artificial intelligence processor to the shared memory of the central processing unit includes: The first data is sent to the shared memory of the central processing unit via the PCIe bus.
7. The data processing method according to claim 5 or 6, characterized in that, The data processing method further includes: Receive input data, wherein the input data includes local server data and cross-server data, and the local server data is the first data. The data processing method further includes: The routing information of the input data is obtained by performing routing decision calculations through a gating network. The cross-server data is sent to the second server of the computer system according to the routing information, and the first data is processed by the first server.
8. The data processing method according to claim 7, characterized in that, The step of sending the cross-server data to the second server of the computer system includes: The cross-server data is sent via the PCIe bus through the network cards of the first server and the second server.
9. The data processing method according to claim 5 or 6, characterized in that, The multiple artificial intelligence processors in the first server are each configured to deploy at least one expert model to process the received data to be processed.
10. The data processing method according to claim 9, characterized in that, The first AI processor is equipped with at least two expert models. The process of processing at least a first portion of the first data to obtain the first result data includes: The first sub-result data and the second sub-result data are obtained by processing the at least first part of the first data through the at least two expert models respectively, and the first sub-result data and the second sub-result data are aggregated to obtain the first result data.
11. A data processing apparatus, characterized in that, A central processing unit in a first server of a computer system comprising multiple servers, wherein the first server further comprises multiple artificial intelligence processors, the computer system is configured to deploy a hybrid expert model, the multiple servers are configured to deploy multiple expert models, and the data processing apparatus comprises: A receiving module is configured to receive the first data from the first artificial intelligence processor to the shared memory of the central processing unit in response to the first artificial intelligence processor receiving the first input data, wherein the shared memory is shared by the plurality of artificial intelligence processors; The distribution module is configured to, in response to computational processing required for the first data, send at least a first portion of the first data from the shared memory of the central processing unit to the first artificial intelligence processor, and send at least a second portion of the first data to at least one second artificial intelligence processor other than the first artificial intelligence processor among a plurality of artificial intelligence processors of the first server. An aggregation module is configured to receive, via the shared memory of the central processing unit, first result data after at least one expert model in the first artificial intelligence processor has processed the at least first part, and second result data after at least one expert model in the at least one second artificial intelligence processor has processed the at least second part, and to aggregate the first result data and the second result data. The at least second portion is cross-card data, and the step of sending at least the second portion of the first data to at least one second artificial intelligence processor other than the first artificial intelligence processor among the plurality of artificial intelligence processors of the first server includes: The routing information of the first data is obtained by performing routing decision calculations through the distribution engine of the central processing unit; The cross-card data is sent to the at least one second artificial intelligence processor according to the routing information.
12. A data processing apparatus, characterized in that, A first artificial intelligence processor is used in a first server of a computer system comprising multiple servers, wherein the first server further comprises a central processing unit, the computer system is configured to deploy a hybrid expert model, the multiple servers are configured to deploy multiple expert models, and the data processing device comprises: The sending module is configured to send first data input to the first artificial intelligence processor to the shared memory of the central processing unit, wherein the shared memory is shared by the plurality of artificial intelligence processors; The feedback module is configured to, in response to receiving at least a first portion of the first data sent from the shared memory of the central processing unit, process the at least first portion of the first data using at least one expert model in the first artificial intelligence processor to obtain first result data, and then feed the first result data back to the shared memory of the central processing unit for aggregation with second result data obtained by at least one expert model in at least one second artificial intelligence processor (excluding the first artificial intelligence processor) processing at least a second portion of the first data. The at least second part is cross-card data. The routing information of the first data is obtained by performing routing decision calculation through the distribution engine of the central processing unit, and the cross-card data is sent to the at least one second artificial intelligence processor according to the routing information.
13. An electronic device, characterized in that, Includes the data processing apparatus as described in claim 11 or 12.
14. An electronic device, characterized in that, include: At least one processor; as well as Memory, including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to perform the data processing method according to any one of claims 1-4 or any one of claims 5-10.
15. A non-transitory storage medium, characterized in that, The system stores computer-executable instructions, wherein, when executed by a computer, the data processing method according to any one of claims 1-4 or any one of claims 5-10 is performed.