Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product, and particularly relates to the technical field of computers, in particular to the technical fields of artificial intelligence, deep learning models, hardware collaboration and the like. According to the specific implementation scheme, first data from a first processor are received, the first data are determined from to-be-processed data under the condition that the first processor determines overload based on the to-be-processed data, and the to-be-processed data are used for being processed by a target expert module in the hybrid expert model; based on the module parameters of the target expert module, operating the target expert module to process the first data to obtain a first processing result; and sending the first processing result to the first processor.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, deep learning models, and hardware collaboration, specifically to data processing methods, devices, electronic devices, storage media, and program products. Background Technology

[0002] With the advent of the artificial intelligence era, the continuous expansion of model size has become a key driver for performance improvement. To achieve high efficiency in single-task processing and simultaneous multi-task handling, Mixture-of-Experts (MoE) models have emerged. How to adapt and run MoE models with hardware devices has become a key research focus. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, electronic device, storage medium, and program product.

[0004] According to one aspect of this disclosure, a data processing method is provided, comprising: receiving first data from a first processor, wherein the first data is determined from the data to be processed when the first processor determines overload based on data to be processed, the data to be processed being used for processing by a target expert module in a hybrid expert model; running the target expert module to process the first data based on module parameters of the target expert module to obtain a first processing result; and sending the first processing result to the first processor so that the first processor determines a target processing result for the data to be processed based on the first processing result and a second processing result, wherein the second processing result is determined by the first processor running the target expert module to process second data based on the module parameters, the second data being determined based on the data to be processed and the first data.

[0005] According to another aspect of this disclosure, a data processing method is provided, comprising: acquiring data to be processed for processing by a target expert module in a hybrid expert model; determining first data and second data from the data to be processed when an overload is determined based on the data to be processed, the first data being sent to a second processor, the second processor storing module parameters of the target expert module; receiving a first processing result from the second processor, the first processing result being obtained by the second processor processing the first data by the target expert module based on the module parameters; and determining a target processing result for the data to be processed based on the first processing result and the second processing result, the second processing result being determined by processing the second data by the target expert module based on the module parameters.

[0006] According to another aspect of this disclosure, a data processing apparatus is provided, comprising: a first receiving module for receiving first data from a first processor, wherein the first data is determined from the data to be processed when the first processor determines overload based on data to be processed, and the data to be processed is used for processing by a target expert module in a hybrid expert model; a running module for running the target expert module to process the first data based on module parameters of the target expert module to obtain a first processing result; and a sending module for sending the first processing result to the first processor so that the first processor determines a target processing result of the data to be processed based on the first processing result and a second processing result, wherein the second processing result is determined by the first processor running the target expert module to process second data based on the module parameters, and the second data is determined based on the data to be processed and the first data.

[0007] According to another aspect of this disclosure, a data processing apparatus is provided, comprising: an acquisition module for acquiring data to be processed for processing by a target expert module in a hybrid expert model; a splitting module for determining first data and second data from the data to be processed when an overload is determined based on the data to be processed, the first data being sent to a second processor, the second processor storing module parameters of the target expert module; a second receiving module for receiving a first processing result from the second processor, the first processing result being obtained by the second processor processing the first data based on the module parameters using the target expert module; and a result summarizing module for determining a target processing result for the data to be processed based on the first processing result and the second processing result, the second processing result being determined by processing the second data based on the module parameters using the target expert module.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1A This illustration schematically depicts an exemplary scenario architecture where data processing methods and apparatus can be applied according to embodiments of this disclosure;

[0014] Figure 1B This illustration schematically shows an exemplary system architecture to which data processing methods and apparatus can be applied according to embodiments of the present disclosure;

[0015] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0016] Figure 3A A schematic diagram illustrating a data processing method according to a related example of this disclosure is shown.

[0017] Figure 3B A schematic diagram illustrating a data processing method according to another related example of this disclosure is shown.

[0018] Figure 4 A schematic diagram illustrating the structure of a hybrid expert model according to an embodiment of the present disclosure is shown.

[0019] Figure 5A A schematic diagram illustrating the acquisition of module parameters according to an embodiment of the present disclosure is shown.

[0020] Figure 5B A schematic diagram illustrating the acquisition of module parameters according to another embodiment of the present disclosure is shown;

[0021] Figure 6 A flowchart illustrating a data processing method according to another embodiment of the present disclosure is shown schematically;

[0022] Figure 7 A block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 8 A block diagram of a data processing apparatus according to another embodiment of the present disclosure is schematically shown; and

[0024] Figure 9 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] In the inference process using hybrid expert models, the number of tasks received by different expert modules varies significantly, directly leading to an imbalance in the computational load across these modules. When these expert modules are run on different processors of the computing device, this load difference can cause computational imbalance between processors.

[0027] For example, some processors in a computing device may be overloaded, while others may be idle. In other words, processors hosting popular expert modules may experience computing power strain, while processors hosting less popular expert modules may waste computing power. Furthermore, overloading processors with limited computing power can lead to latency issues.

[0028] In view of the above, embodiments of this disclosure provide a data processing method, comprising: receiving first data from a first processor, wherein the first data is determined from the data to be processed when the first processor determines that it is overloaded based on the data to be processed, and the data to be processed is used for processing by a target expert module in a hybrid expert model; running the target expert module to process the first data based on the module parameters of the target expert module to obtain a first processing result; and sending the first processing result to the first processor so that the first processor determines a target processing result for the data to be processed based on the first processing result and a second processing result, wherein the second processing result is determined by the first processor running the target expert module to process the second data based on the module parameters, and the second data is determined based on the data to be processed and the first data.

[0029] The data processing method provided in this embodiment can utilize the ability of the first processor to determine whether it is overloaded based on the data to be processed. In the case of overload of the first processor, some of the data to be processed can be actively forwarded to other processors for processing, thereby reducing latency and improving load balancing.

[0030] Figure 1A The illustration schematically depicts an exemplary scenario architecture in which data processing methods and apparatus can be applied according to embodiments of the present disclosure.

[0031] It should be noted that Figure 1 is only an example of a scenario architecture that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but it does not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0032] like Figure 1A As shown, the scenario architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0034] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server cluster 105 may include servers that provide various services, such as a backend management server (for example only) that supports the content browsed by users using terminal devices 101, 102, and 103. The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0036] For example, a server cluster can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It solves the problems of high management difficulty and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). A server cluster can also be a server cluster of a distributed system, or a server cluster that incorporates blockchain.

[0037] In one example embodiment, the terminal device can send a user request to the server cluster via a network. In response to the user request, the server cluster invokes a hybrid expert model to process the request, obtains a processing result, and then feeds the result back to the terminal device.

[0038] When multiple terminal devices simultaneously send user requests to the server cluster via the network, the number of tasks that the target expert module in the hybrid expert model needs to handle increases dramatically, leading to processor overload on the target expert module. The following will illustrate this through... Figure 1B The exemplary system architecture diagram shown illustrates how to solve the load imbalance problem.

[0039] Figure 1B An exemplary system architecture for which data processing methods and apparatus can be applied according to embodiments of this disclosure is illustrated.

[0040] like Figure 1B As shown, server cluster 105 can be a distributed system server cluster, including server nodes as computing devices 1051, ..., 105N and server nodes as auxiliary computing devices 105M. Each computing device may include a first processor and a second processor. The first and second processors of computing device 1051 may both store module parameters of the target expert module. The auxiliary computing device 105M may store module parameters of the gating selector.

[0041] The first processor and the second processor may each include one or more of the following: Graphics Processing Unit (GPU), Central Processing Unit (CPU), Tensor Processing Unit (TPU), Deep Learning Processing Unit (DPU), Neural Network Processing Unit (NPU), and Brain Processing Unit (BPU).

[0042] The auxiliary computing device 105M can operate the gating selector based on the module parameters of the gating selector to allocate intermediate data obtained in response to user requests to the first processor of the computing device 1051, which stores the module parameters of the target expert module. If the first processor determines overload based on the data to be processed, it determines first data and second data from the data to be processed. The first data is sent to the second processor of the computing device 1051, which, based on the module parameters of the target expert module, runs the target expert module to process the first data, obtaining a first processing result. The first processing result is then sent back to the first processor. Simultaneously, the first processor, based on the module parameters of the target expert module, runs the target expert module to process the second data, obtaining a second processing result. The first and second processing results are combined to determine the target processing result. The first processor can then feed back the target processing result to the auxiliary computing device 105M that distributed the data to be processed.

[0043] It should be understood that Figure 1B The number of server nodes, first processors, and second processors shown are merely illustrative. Depending on implementation requirements, any number of server nodes, first processors, and second processors can be used.

[0044] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of any type of information, such as user personal information, comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0045] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0046] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0047] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0048] like Figure 2 As shown, the method includes operations S210~S230.

[0049] In operation S210, first data is received from the first processor.

[0050] In operation S220, based on the module parameters of the target expert module, the target expert module is run to process the first data and obtain the first processing result.

[0051] In operation S230, the first processing result is sent to the first processor.

[0052] like Figure 2 The execution entity of the data processing method shown may include a second processor. The first processor and the second processor may belong to the same computing device, thereby reducing the transmission latency of the first data and the first processing result.

[0053] The first data may be determined from the data to be processed when the first processor determines an overload based on the data to be processed, and the data to be processed is used for processing by the target expert module in the hybrid expert model.

[0054] Optionally, the data to be processed can be in matrix form and can be obtained by processing at least one of image data, text data, and voice data.

[0055] Hybrid expert models can include gating selectors and multiple expert modules. Each expert module can include one or more neural networks or a portion of a neural network. For example, hybrid expert models can include large-scale neural network models based on deep learning, such as large language models or multimodal large models that include gating selectors and multiple expert modules. Compared to conventional deep learning models such as encoders and decoders, hybrid expert models have deeper layers, wider scope, and a larger number of parameters. Different expert modules can be run on different processors, thereby enabling multi-task concurrent processing and large-scale computational processing.

[0056] For example, the data to be processed can be assigned to the first processor to run the target expert module and obtain the corresponding target processing results. However, each computing device and processor has a rated load, and exceeding the rated load can be called overload. If the first processor is determined to be overloaded based on the data to be processed, continuing to use the first processor to process the data will lead to an imbalance in the processing time of multiple threads in parallel processing, thereby causing a delay in the entire processing flow due to the latency of individual processors.

[0057] A portion of the data to be processed can be identified as the first data and sent to the second processor. The second processor then uses the target expert module to process the first data based on the module parameters of the target expert module, thereby obtaining the first processing result.

[0058] While the second processor processes the first data, the first processor, based on the module parameters of the target expert module, runs the target expert module to process the second data, obtaining a second processing result. The second data is determined based on the data to be processed and the first data. For example, the first and second data are obtained by splitting the data to be processed. The first data is sent to the second processor, while the second data is left for processing by the first processor.

[0059] After processing the first data, the second processor can send the first processing result back to the first processor. This allows the first processor to determine the target processing result for the data to be processed based on the first and second processing results.

[0060] Optionally, the first processor and the second processor can be deployed on the same computing device to reduce the latency of transmitting the first data. There are no restrictions on the processor types of the first and second processors; they can be the same or different. As long as the module parameters of the expert module are stored in the storage space of the first processor, they can also be stored in the storage space of the second processor.

[0061] According to embodiments of this disclosure, by utilizing the ability of a first processor to determine whether it is overloaded based on the data to be processed, and in the event that the first processor is overloaded, a portion of the data to be processed is proactively forwarded to other processors for processing, thereby reducing latency.

[0062] Figure 3A A schematic diagram illustrating a data processing method according to a related example of this disclosure is shown.

[0063] like Figure 3A As shown, the module parameters of expert modules Expert1 and Expert2 are stored on processor GPU1, while the module parameters of expert modules Expert3 and Expert4 are stored on processor GPU2. Currently, there is a large amount of data to be processed using expert modules Expert1 and Expert2, accounting for 25% and 50% of the total data volume, respectively, causing processor GPU1 to be overloaded. To solve the load balancing problem, a global reordering scheme can be adopted. For example, a "high-low pairing" strategy can be used to balance the load, where the module parameters of expert module Expert2 can be transferred to processor GPU2, and the module parameters of expert module Expert3 can be transferred to processor GPU1.

[0064] A global reordering scheme can solve the processor load balancing problem without consuming additional memory. However, compared to... Figure 3A Compared to the data processing method shown, the data processing method provided in this embodiment can solve the processor load balancing problem while avoiding request pauses caused by the dynamic transfer of module parameters of the expert module.

[0065] Figure 3B A schematic diagram illustrating a data processing method according to another related example of this disclosure is shown.

[0066] With Figure 3A The difference is that, as shown, Figure 3BThe diagram illustrates a redundant replica scheme, also known as the Redundant Experts Strategy. Its core idea is to store the module parameters of the popular expert module Expert1 on the idle processor GPU2, acting as a replica of GPU1. When the server node distributes data to be processed, it directly distributes the two split sub-data sets to GPU1 and GPU2, allowing the expert modules Expert1 on GPU1 and GPU2 to process the sub-data sets separately, thus achieving load balancing.

[0067] Employing a redundant replica scheme can solve the processor load balancing problem and eliminates request lag issues caused by dynamic transfer of module parameters. However, compared to... Figure 3B Compared to the data processing method shown, the data processing method provided in this disclosure can solve the processor load balancing problem while allowing the processor to actively determine whether it is overloaded and then respond by dispatching tasks to other processors. This eliminates the need to collect the load information of each processor before determining the dispatch target, reducing the amount of data processing required for load judgment of each processor, and does not affect the latency and throughput of the critical path.

[0068] The foregoing examples further illustrate how the embodiments of this disclosure combine the first processor and the second processor to complete data processing. The following will describe, through embodiments and in conjunction with… Figure 4 This section explains how the first processor obtains the data to be processed.

[0069] According to embodiments of this disclosure, the data to be processed for the target expert module can be determined as follows: an auxiliary computing device in a server cluster calls a gating selector to process the output data of the target network layer in the hybrid expert model, obtaining an expert routing result. The server cluster includes computing devices and auxiliary computing devices, and the target network layer is located at the network layer above the target expert module in the hybrid expert model. If the expert routing result indicates that the output data should be routed to the target expert module, the data to be processed is determined based on the output data.

[0070] Figure 4 A schematic diagram illustrating the structure of a hybrid expert model according to an embodiment of the present disclosure is shown.

[0071] like Figure 4 As shown, a hybrid expert model can include multiple stacked hybrid expert groups and multiple general network layers. Each hybrid expert group includes a gating selector and multiple expert modules. The multiple expert modules reside at the same network layer level of the hybrid expert model.

[0072] Optionally, a general network layer may include, but is not limited to, convolutional layers, pooling layers, fully connected layers, etc.

[0073] Optionally, multiple expert modules at the same network layer can have the same network structure, but different module parameters, to match data of different modalities or adapt to processing requirements of different functions. The network structure of the expert modules is not limited; for example, it can include convolutional layers, pooling layers, fully connected layers, etc., but it is not limited to these. It can also include encoder-decoder, long short-term memory networks, self-attention mechanisms, etc.

[0074] A gating selector, also known as a gating network, can be a multi-classifier used to determine which expert modules should process the output data from the previous network layer.

[0075] like Figure 4 As shown, taking the hybrid expert group N as an example of a multilayer perceptron (MLP) network layer, the hybrid expert group includes a gating selector G and multiple expert modules MLP1, ..., MLPN. In the hybrid expert model, the network layer above the hybrid expert group is the self-attention mechanism layer, which is used as the target network layer.

[0076] like Figure 4 As shown, the self-attention mechanism layer processes the output data. The module parameters of the gating selector G are stored on the auxiliary computing device in the server cluster. The auxiliary computing device runs the gating selector based on the module parameters to process the output data in the hybrid expert model, obtaining the expert routing result. The expert routing result indicates the matching relationship between the expert module and the output data. Based on the expert routing result, the target expert module MLP1 can be determined from multiple expert modules MLP1, ..., MLPN. The output data can be used as the data to be processed. However, it is not limited to this. When the expert routing result indicates that there are two target expert modules, namely expert module MLP1 and expert module MLP3, the output data can be split to obtain the data to be processed for expert module MLP1 and the data to be processed for expert module MLP3, respectively.

[0077] According to embodiments of this disclosure, multiple hybrid expert groups with different network levels are configured for a hybrid expert model. The selection of the target expert module for each hybrid expert group is determined by a gating selector configured on an auxiliary computing device, thereby improving the selection accuracy of the target expert module. Furthermore, utilizing a combination of auxiliary computing devices and computing devices to perform data processing separates parallel processing from data allocation, improving processing efficiency and stability.

[0078] The preceding text explained how to determine the data to be processed by the first processor. The following text will explain how the second processor obtains the module parameters.

[0079] According to embodiments of this disclosure, for example, Figure 2 Before the operation S220 shown, the data processing method may further include: acquiring the module parameters of the target expert module according to the acquisition timing.

[0080] By acquiring the module parameters of the target expert module according to the acquisition timing, the reliability and effectiveness of the acquired module parameters can be improved.

[0081] According to embodiments of this disclosure, the timing of acquisition can be determined based on the storage space of the second processor and the target module parameters. The target module parameters include the module parameters of expert modules belonging to the same network layer as the target expert module in the hybrid expert model.

[0082] For example, based on the target module parameters stored in the computing device to which the first processor belongs, the storage space used to store the module parameters of the target expert module can be identified to obtain the identification result. The module parameters of the target expert module are then acquired according to the acquisition timing that matches the identification result.

[0083] Based on the target module parameters, the storage space is identified, and the identification result is obtained. The identification result can indicate that the storage space is sufficient to store the target module parameters, or it can indicate that the storage space is insufficient and it is difficult to store the target module parameters.

[0084] If the identification results indicate that there is sufficient storage space, the target module parameters can be pre-stored before the first processor is determined to be overloaded. This allows the target expert module to be run directly based on the module parameters in response to the receipt of the first data, reducing the latency caused by reading the module parameters.

[0085] If the identification results indicate insufficient storage space, the module parameters of the target expert module can be stored after the first processor is determined to be overloaded. This allows for targeted storage of module parameters and avoids the problem of incomplete storage caused by storing all module parameters, such as the target module parameters, into the storage space at once.

[0086] According to embodiments of this disclosure, determining the timing of acquiring module parameters of the target expert module based on the identification results of the storage space can improve the timeliness of acquiring module parameters while also improving the reliability and stability of module parameter storage.

[0087] Figure 5A A schematic diagram illustrating the acquisition of module parameters according to an embodiment of the present disclosure is shown.

[0088] like Figure 5AAs shown, the computing device may include GPU1 and GPU2 as the first processor and CPU as the second processor, and may also include memory, such as DIMM (Dual-Inline-Memory-Modules).

[0089] Alternatively, the second processor can directly access the memory. If the memory stores the module parameters, the second processor can directly read the module parameters from the memory.

[0090] like Figure 5A As shown, the module parameters of Expert1 and Expert2 are configured on GPU1, and the module parameters of Expert3 and Expert4 are configured on GPU2.

[0091] like Figure 5A As shown, the module parameters of Expert Modules Expert1 to Expert Modules Expert4 can be configured on the memory to ensure that the second processor can handle the load balancing pressure when GPU1 and GPU2 experience overload problems in different scenarios.

[0092] If the recognition result indicates that the storage space is greater than or equal to the space occupied by the target module parameters, the target module parameters are obtained in response to the first processor GPU1 acquiring the data to be processed.

[0093] like Figure 5A As shown, if the CPU's storage space is greater than or equal to the space occupied by the module parameters of Expert1 to Expert4, then in response to GPU1 acquiring the data to be processed, the CPU can read the module parameters of Expert1 to Expert4 from the DIMM and store them in the storage space.

[0094] According to embodiments of this disclosure, if the identification result indicates that the storage space is greater than or equal to the space occupied by the target module parameters, it means that the storage space is sufficient to store the target module parameters. In this case, the module parameters of all expert modules at the same network layer can be pre-stored. This ensures that the time for the second processor to access the memory for data processing is completely hidden, avoiding access latency and buying time for the CPU to perform calculations. Furthermore, since the GPU and CPU are on the same computing device, there is no need to transfer the first data corresponding to the target expert module across machines, reducing transmission latency.

[0095] Figure 5B A schematic diagram illustrating the acquisition of module parameters according to another embodiment of the present disclosure is shown.

[0096] With Figure 5A The difference shown is that, as Figure 5B As shown, if the identification result indicates that the CPU's storage space is less than the space occupied by the target module parameters, such as the module parameters of Expert1-Expert4, in response to determining that the first processor is overloaded, see as follows. Figure 5B The boxed line in bold indicates that the data to be processed for Expert1 is overloaded. The module parameters of the target Expert1 are obtained from the DIMM.

[0097] If the identification results indicate that the CPU's own storage space is insufficient, less than the space occupied by the target module parameters, then the storage space is determined to be inadequate, making it difficult to store the module parameters of all expert modules. In this case, it is impossible to pre-store the module parameters of all expert modules.

[0098] The first processor can determine the overload and send the overload information to the second processor. In response to receiving the overload information indicating that the first processor is overloaded, the module parameters of the target expert module are obtained. Since the overload information includes parameter identifiers of the module parameters that match the data to be processed, the module parameters can be read selectively. Simultaneously, the first processor determines the first data from the data to be processed and sends the first data to the second processor.

[0099] Since the first data still needs to be determined and transmitted from the first processor to the second processor, this time difference also reduces the latency of reading the module parameters.

[0100] According to embodiments of this disclosure, when the storage space is determined to be less than the space occupied by the target module parameters, the module parameters of the target expert module can be obtained in a targeted manner, which can ensure that the module parameters stored in the storage space are accurate and effective, and improve the stability and reliability of data processing.

[0101] According to embodiments of this disclosure, module parameters may include a matrix.

[0102] For example, retrieving module parameters may include calling a read program to read the row vector elements of the module parameters sequentially from memory.

[0103] The read program can be a written code program used to sequentially read row vector elements of module parameters from memory. Optionally, the read program may include an identifier field indicating how the row vector elements are compiled. The identifier field may include keywords such as "volatile," which indicates a "forced data loading, but no computation" compilation mode for the row vector elements, so that the central processing unit may modify or optimize away the stored row vector elements due to certain factors.

[0104] Another example of obtaining module parameters can be achieved by using a prefetch instruction from the CPU's cache.

[0105] Compared to directly obtaining module parameters using read instructions, using a written read program to access the module parameters to be used can automatically load this data into the cache of the second processor, while avoiding the optimization of pre-stored module parameters, thus improving the stability and reliability of write operations.

[0106] The preceding description used a second processor as the execution subject. The following description will use a first processor as the execution subject to describe embodiments of this disclosure.

[0107] Figure 6 A flowchart illustrating a data processing method according to another embodiment of the present disclosure is shown.

[0108] like Figure 6 As shown, the method includes operations S610~S640.

[0109] In operation S610, the data to be processed is obtained for the target expert module in the hybrid expert model.

[0110] like Figure 6 The execution entity of the data processing method shown may include a first processor. The first processor stores the module parameters of the target expert module.

[0111] In operation S620, if an overload is determined based on the data to be processed, first data and second data are determined from the data to be processed.

[0112] The first processor can send the first data to the second processor, which stores the module parameters of the target expert module. The second processor, based on the module parameters, can then run the target expert module to process the first data and obtain a first processing result. This first processing result is then sent back to the first processor via the second processor.

[0113] While the second processor processes the first data, the second processor can run the target expert module based on the module parameters to process the second data and determine the second processing result.

[0114] In operation S630, the first processing result is received from the second processor.

[0115] In operation S640, based on the first processing result and the second processing result, the target processing result of the data to be processed is determined.

[0116] According to embodiments of this disclosure, by utilizing the ability of a first processor to determine whether it is overloaded based on the data to be processed, and in the event that the first processor is overloaded, a portion of the data to be processed is proactively forwarded to other processors for processing, thereby reducing latency.

[0117] According to embodiments of this disclosure, the first processor and the second processor may belong to the same computing device, thereby reducing the transmission latency of the first data and the first processing result.

[0118] According to embodiments of this disclosure, when performing such Figure 6 Prior to the operation S620 shown, the data processing method may also include: determining whether there is an overload.

[0119] In one embodiment, it can be determined whether the system is overloaded based on the amount of data to be processed and the amount of reference data.

[0120] The reference data volume indicates the average processing volume of expert modules in the hybrid expert model that belong to the same network layer as the target expert module.

[0121] Alternatively, the data to be processed can also be called a token, and the amount of data to be processed can be measured in terms of the number of tokens.

[0122] For example, expert modules belonging to the same network layer in a hybrid expert model can also be called expert modules belonging to the same hybrid expert group, such as Expert1, Expert2, Expert3, and Expert4.

[0123] The amount of data to be processed routed to Expert1 is the number of tokens F1, the amount of data to be processed routed to Expert2 is the number of tokens F2, the amount of data to be processed routed to Expert3 is the number of tokens F3, and the amount of data to be processed routed to Expert4 is the number of tokens F4.

[0124] The amount of data to be processed for the target expert module Expert1, which is allocated to the first processor, is the number of tokens F1. The reference data amount is (number of tokens F2 + number of tokens F3 + number of tokens F4) / 3.

[0125] The first processor can be determined to be overloaded if the amount of data to be processed exceeds the amount of reference data. Conversely, it can be determined not to be overloaded if the amount of data to be processed is less than or equal to the amount of reference data. However, this is not the only possibility. The first processor can also be determined to be overloaded if the amount of data to be processed exceeds (1+ε)*the amount of reference data. ε is typically set to 0.2 to 0.3. By adding a weight greater than 1 to the reference data amount, the first processor can only be determined to be overloaded when the amount of data to be processed is significantly greater than the reference data amount. This raises the threshold for first processor overload, thereby improving the effectiveness and accuracy of the second processor's assistance in processing the data after overload, and avoiding misjudgments.

[0126] In another embodiment, it can be determined whether the system is overloaded based on the amount of data to be processed and hardware configuration information.

[0127] Hardware configuration information may include the load on the first processor, such as a maximum threshold for the number of tokens it can process. The first processor is determined to be overloaded if the ratio of the amount of data to be processed to the hardware configuration information is greater than or equal to the overload threshold. Conversely, the first processor is determined not to be overloaded if the ratio is less than or equal to the overload threshold. The overload threshold can be greater than or equal to 0.9, but is not limited to this; any threshold determined based on the actual situation is acceptable.

[0128] According to embodiments of this disclosure, a "hard overload" judgment can be directly triggered based on hardware configuration information, thereby triggering the processor's rated load and preventing the first processor from automatically discarding data to be processed under overload conditions, thus improving the operational stability of the hardware device.

[0129] According to a preferred embodiment of this disclosure, a first load identification result can be determined based on the amount of data to be processed and the amount of reference data. A second load identification result is determined based on the amount of data to be processed and hardware configuration information. Based on the first and second load identification results, it is determined whether the first processor is overloaded. For example, if either the first or second load identification result indicates that the first processor is overloaded, it is determined that the first processor is overloaded. Conversely, if both the first and second load identification results indicate that the first processor is not overloaded, it is determined that the first processor is not overloaded. This improves the sensitivity and accuracy of overload judgment by considering multiple dimensions.

[0130] According to embodiments of this disclosure, for example, Figure 6 The operation S620 shown, which determines the first data from the data to be processed, may include: determining the first data from the data to be processed based on the amount of data that can be processed by the second processor and the amount of data exceeding the processing limit. The amount of data exceeding the processing limit is the amount of data that exceeds a predetermined threshold.

[0131] The predetermined threshold can be understood as the upper limit of the number of tokens that the first processor can process.

[0132] The amount of data that is overloaded is the amount of data to be processed minus a predetermined threshold.

[0133] The amount of data that the second processor can process can be determined based on the tasks allocated to it by other processors in the same computing device. The amount of overloaded data can be compared with the amount of data that can be processed. If the amount of overloaded data is less than or equal to the amount of data that can be processed, the data to be processed in the overloaded data portion is directly used as the first data and allocated to the second processor. This achieves complete load balancing through the second processor.

[0134] When the amount of overloaded data exceeds the amount of data that can be processed, the unprocessed data is directly allocated as the first data to the second processor. This alleviates the load balancing pressure through the second processing.

[0135] According to embodiments of this disclosure, the amount of first data to be transferred is determined based on the amount of data that the second processor can process and the amount of data that the first processor is overloaded with, thereby avoiding lag caused by overload of the second processor and improving the stability and reliability of the second processor's operation.

[0136] Figure 7 A block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0137] like Figure 7 As shown, the data processing device 700 includes: a first receiving module 710, an operating module 720, and a sending module 730.

[0138] The first receiving module 710 is used to receive first data from the first processor. The first data is determined from the data to be processed when the first processor determines that there is an overload based on the data to be processed. The data to be processed is used for processing by the target expert module in the hybrid expert model.

[0139] The execution module 720 is used to run the target expert module to process the first data based on the module parameters of the target expert module and obtain the first processing result.

[0140] The sending module 730 is used to send the first processing result to the first processor so that the first processor can determine the target processing result of the data to be processed based on the first processing result and the second processing result. The second processing result is determined by the first processor running the target expert module to process the second data based on the module parameters. The second data is determined based on the data to be processed and the first data.

[0141] According to embodiments of this disclosure, the data processing apparatus further includes an identification module and a parameter acquisition module.

[0142] The identification module is used to identify the storage space used to store the module parameters of the target expert module based on the target module parameters stored in the computing device to which the first processor belongs, and to obtain the identification result. The target module parameters include the module parameters of the expert modules of the hybrid expert model that belong to the same network layer as the target expert module.

[0143] The parameter acquisition module is used to acquire the module parameters of the target expert module according to the acquisition timing that matches the recognition results.

[0144] According to an embodiment of this disclosure, the parameter acquisition module includes a first parameter acquisition submodule.

[0145] The first parameter acquisition submodule is used to acquire the target module parameters in response to the first processor acquiring the data to be processed when the recognition result indicates that the storage space is greater than or equal to the space occupied by the target module parameters.

[0146] According to embodiments of this disclosure, the parameter acquisition module includes a second parameter acquisition submodule.

[0147] The second parameter acquisition submodule is used to acquire the module parameters of the target expert module in response to determining that the first processor is overloaded when the recognition result indicates that the storage space is less than the space occupied by the target module parameters.

[0148] According to embodiments of this disclosure, the module parameters include a matrix.

[0149] The first parameter acquisition submodule or the second parameter acquisition submodule includes: a parameter acquisition unit.

[0150] The parameter acquisition unit is used to call the reading program to read the row vector elements of the module parameters sequentially from the memory. The reading program includes an identifier field, which indicates the compilation method of the row vector elements.

[0151] Figure 8 A block diagram of a data processing apparatus according to another embodiment of the present disclosure is shown schematically.

[0152] like Figure 8 As shown, the data processing device 800 includes: an acquisition module 810, a splitting module 820, a second receiving module 830, and a result summarization module 840.

[0153] The acquisition module 810 is used to acquire the data to be processed by the target expert module in the hybrid expert model.

[0154] The splitting module 820 is used to determine first data and second data from the data to be processed when an overload is determined based on the data to be processed. The first data is used to send to the second processor, which stores the module parameters of the target expert module.

[0155] The second receiving module 830 is used to receive the first processing result from the second processor. The first processing result is obtained by the second processor running the target expert module to process the first data based on the module parameters.

[0156] The result summary module 840 is used to determine the target processing result of the data to be processed based on the first processing result and the second processing result. The second processing result is determined by running the target expert module to process the second data based on the module parameters.

[0157] According to embodiments of this disclosure, the data processing apparatus further includes a first load determination module.

[0158] The first load determination module is used to determine whether the load is overloaded based on the amount of data to be processed and the amount of reference data. The amount of reference data indicates the average processing volume of expert modules in the hybrid expert model that belong to the same network layer as the target expert module.

[0159] According to embodiments of this disclosure, the data processing apparatus further includes a second load determination module.

[0160] The second load determination module is used to determine whether the system is overloaded based on the amount of data to be processed and hardware configuration information.

[0161] According to embodiments of this disclosure, the splitting module includes a splitting sub-module.

[0162] The splitting submodule is used to determine the first data from the data to be processed based on the amount of data that the second processor can process and the amount of data that is overloaded, wherein the amount of data that is overloaded is the amount of data that exceeds a predetermined threshold.

[0163] According to embodiments of this disclosure, the data to be processed for the target expert module is determined through the following modules:

[0164] The routing module is used by auxiliary computing devices in the server cluster to call the gating selector to process the output data of the target network layer in the hybrid expert model and obtain expert routing results. The server cluster includes computing devices and auxiliary computing devices, and the target network layer is the network layer above the network layer of the target expert module in the hybrid expert model.

[0165] The allocation module is used to determine the data to be processed based on the output data, when the expert routing result indicates that the output data should be routed to the target expert module.

[0166] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0167] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0168] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0169] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0170] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0172] Multiple components in device 900 are connected to input / output (I / O) interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0180] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Receive first data from a first processor, the first data being determined from the data to be processed when the first processor determines an overload based on the data to be processed, the data to be processed being used for processing by the target expert module in a hybrid expert model; Based on the module parameters of the target expert module, the target expert module is run to process the first data to obtain a first processing result; as well as The first processing result is sent to the first processor so that the first processor can determine the target processing result of the data to be processed based on the first processing result and the second processing result. The second processing result is determined by the first processor running the target expert module to process the second data based on the module parameters. The second data is determined based on the data to be processed and the first data.

2. The method according to claim 1, further comprising: Based on the target module parameters stored in the computing device to which the first processor belongs, the storage space used to store the module parameters of the target expert module is identified to obtain an identification result. The target module parameters include the module parameters of the expert modules of the hybrid expert model that belong to the same network layer as the target expert module; and The module parameters of the target expert module are obtained according to the acquisition timing that matches the recognition result.

3. The method according to claim 2, wherein, The step of acquiring the module parameters of the target expert module according to the acquisition timing that matches the recognition result includes: If the identification result indicates that the storage space is greater than or equal to the space occupied by the target module parameters, the target module parameters are obtained in response to the first processor obtaining the data to be processed.

4. The method according to claim 2, wherein, The step of acquiring the module parameters of the target expert module according to the acquisition timing that matches the recognition result includes: If the identification result indicates that the storage space is less than the space occupied by the target module parameters, in response to determining that the first processor is overloaded, the module parameters of the target expert module are obtained.

5. The method according to any one of claims 1 to 4, wherein, The module parameters include a matrix. Obtaining the module parameters includes: The reading program is invoked to sequentially read the row vector elements of the module parameters from the memory. The reading program includes an identifier field that indicates the compilation method of the row vector elements.

6. A data processing method, comprising: Obtain the data to be processed for the target expert module in the hybrid expert model; If an overload is determined based on the data to be processed, a first data and a second data are determined from the data to be processed. The first data is used to send to a second processor, and the second processor stores the module parameters of the target expert module. The system receives a first processing result from the second processor, which is obtained by the second processor processing the first data based on the module parameters by running the target expert module. as well as Based on the first processing result and the second processing result, the target processing result of the data to be processed is determined, and the second processing result is used to run the target expert module to process the second data based on the module parameters.

7. The method according to claim 6, further comprising: Based on the amount of data to be processed and the amount of reference data, it is determined whether the system is overloaded, wherein the amount of reference data indicates the average processing volume of expert modules in the hybrid expert model that belong to the same network layer as the target expert module.

8. The method according to claim 6, further comprising: Based on the amount of data to be processed and the hardware configuration information, it is determined whether the system is overloaded.

9. The method according to any one of claims 6 to 8, wherein, Determining the first data from the data to be processed includes: Based on the amount of data that the second processor can process and the amount of data that is overloaded, the first data is determined from the data to be processed, wherein the amount of data that is overloaded is the amount of data that exceeds a predetermined threshold.

10. The method according to any one of claims 1 to 9, wherein, The data to be processed for the target expert module is determined in the following manner: The auxiliary computing device in the server cluster calls the gating selector to process the output data of the target network layer in the hybrid expert model to obtain expert routing results. The server cluster includes the computing device and the auxiliary computing device, and the target network layer is located at the network layer above the target expert module in the hybrid expert model. If the expert routing result indicates that the output data should be routed to the target expert module, the data to be processed is determined based on the output data.

11. A data processing apparatus, comprising: A first receiving module is configured to receive first data from a first processor, wherein the first data is determined from the data to be processed when the first processor determines that it is overloaded based on the data to be processed, and the data to be processed is used for processing by the target expert module in a hybrid expert model. The running module is used to run the target expert module to process the first data based on the module parameters of the target expert module, and obtain a first processing result; as well as A sending module is used to send the first processing result to the first processor, so that the first processor can determine the target processing result of the data to be processed based on the first processing result and the second processing result. The second processing result is determined by the first processor running the target expert module to process the second data based on the module parameters. The second data is determined based on the data to be processed and the first data.

12. A data processing apparatus, comprising: The acquisition module is used to acquire the data to be processed by the target expert module in the hybrid expert model. A splitting module is used to determine first data and second data from the data to be processed when an overload is determined based on the data to be processed. The first data is used to send to a second processor, and the second processor stores the module parameters of the target expert module. The second receiving module is used to receive a first processing result from the second processor, wherein the first processing result is obtained by the second processor running the target expert module to process the first data based on the module parameters; as well as The result summary module is used to determine the target processing result of the data to be processed based on the first processing result and the second processing result, wherein the second processing result is used to run the target expert module to process the second data based on the module parameters.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images