Model scheduling method and device, computer device and storage medium

Abstract

The application relates to a model scheduling method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a to-be-inferred text, and determining input vectors of each word segmentation of the to-be-inferred text according to the to-be-inferred text; determining expert network groups of each input vector corresponding to each expert layer in a mixed expert model according to the input vectors of each word segmentation; calling a graphics processing unit of the mixed expert model, determining a first output vector group based on a first expert network of the input vectors and corresponding input vectors; calling a central processing unit of the mixed expert model, determining a second output vector group based on a second expert network of the input vectors and corresponding input vectors, and transmitting the second output vector group to the graphics processing unit; and calling the graphics processing unit of the mixed expert model, determining a target output vector corresponding to the to-be-inferred text according to the first output vector group and the second output vector group. The method can efficiently schedule the mixed expert model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model scheduling method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, hybrid expert model (MOE) technology has emerged. MOE includes hundreds of expert networks and selects the expert network associated with the input for computation based on different inputs.

[0003] Traditional hybrid expert model techniques can schedule hybrid expert networks to perform computations. However, during model inference, different expert networks need to be dynamically activated based on the input, resulting in a large amount of data transmission and low overall efficiency in resource-constrained environments.

[0004] In summary, there is an urgent need for an efficient scheme for scheduling hybrid expert models. Summary of the Invention

[0005] Therefore, it is necessary to provide a model scheduling method, apparatus, computer equipment, and storage medium for efficiently scheduling hybrid expert models, addressing the aforementioned technical problems.

[0006] In a first aspect, this application provides a model scheduling method for scheduling a hybrid expert model, the hybrid expert model comprising: a high-frequency expert network and a low-frequency expert network; the high-frequency expert network is deployed on a graphics processing unit (GPU); the low-frequency expert network is deployed on a central processing unit (CPU); the method comprises: acquiring text to be inferred, and determining the input vector of each word segment of the text to be inferred based on the text to be inferred; determining an expert network group for each input vector corresponding to each expert layer in the hybrid expert model based on the input vector of each word segment; the expert network group comprising: a first expert network and / or a second expert network; invoking the GPU of the hybrid expert model, determining a first output vector group based on the first expert network of the input vector and the corresponding input vector; invoking the CPU of the hybrid expert model, determining a second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmitting the second output vector group to the GPU; invoking the GPU of the hybrid expert model, determining a target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group.

[0007] In one embodiment, obtaining the text to be reasoned and determining the input vector of each word segment of the text to be reasoned based on the text to be reasoned includes: obtaining the text to be reasoned and performing word segmentation on the text to be reasoned to obtain multiple words; and performing vectorization processing on the multiple words to obtain the input vector of each word segment.

[0008] In one embodiment, determining the expert network group corresponding to each input vector of each expert layer in the hybrid expert model based on the input vector of each word segmentation includes: determining the expert network group corresponding to each input vector of each expert layer in the hybrid expert model based on the input vector of each word segmentation; and determining the network weights corresponding to each first expert network and / or each second expert network in the target expert network group based on the target expert network group.

[0009] In one embodiment, the step of calling the graphics processor of the hybrid expert model to determine the first output vector group based on the first expert network of the input vector and the corresponding input vector includes: for a target expert network group, calling the graphics processor of the hybrid expert model to perform parallel computation based on the first expert network of the input vector and the corresponding input vector to determine the first output vector group corresponding to the input vector.

[0010] In one embodiment, the step of invoking the central processing unit of the hybrid expert model, determining a second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmitting the second output vector group to the graphics processor includes: for a target expert network group, obtaining the historical usage frequency of multiple second expert networks in the target expert group; sorting the multiple second expert networks according to the historical usage frequency of the multiple second expert networks to determine a sorted set of multiple second expert networks; and sequentially invoking the central processing unit of the hybrid expert model in the order of the sorted set of multiple second expert networks, determining the second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmitting the second output vector group to the graphics processor.

[0011] In one embodiment, the step of sequentially calling the central processing unit (CPU) of the hybrid expert model in the order of the sorted plurality of second expert networks, determining a second output vector group based on the second expert network with the input vector and the corresponding input vector, and transmitting the second output vector group to the graphics processor includes: calling the CPU of the hybrid expert model according to the target second expert network, determining the second output vector corresponding to the target second expert network based on the target second expert network with the input vector and the corresponding input vector; caching the second output vector in the first cache of the CPU, and generating a first event identifier corresponding to the target second expert network; calling the CPU of the hybrid expert model according to the next second expert network of the target second expert network, determining the second output vector corresponding to the next second expert network based on the next second expert network with the input vector and the corresponding input vector; and synchronously detecting the first event identifier corresponding to the target second expert network, and if the first event identifier corresponding to the target second expert network exists, copying the second output vector corresponding to the target second expert network from the first cache of the CPU to the second cache of the graphics processor.

[0012] In one embodiment, the step of invoking the graphics processor of the hybrid expert model to determine the target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group includes: invoking the graphics processor of the hybrid expert model to determine the initial output vector of each input vector based on the first output vector group, the second output vector group of each input vector, and the network weights corresponding to each first expert network and / or each second expert network in the expert network group of each input vector; invoking the gated network of the hybrid expert model to determine the expert weights corresponding to the initial output vectors of each input vector based on the initial output vectors of each input vector; and determining the target output vector corresponding to the text to be inferred based on the initial output vectors of each input vector and the expert weights corresponding to the initial output vectors of each input vector.

[0013] In one embodiment, the method further includes: obtaining the number of word segments, a first delay time, a second delay time, and a third delay time for multiple word segments corresponding to the text to be inferred; the first delay time is the historical average time for calling the central processing unit (CPU) to determine the output vector group based on the expert network; the second delay time is the historical average time for calling the graphics processing unit (GPU) to determine the output vector group based on the expert network; the third delay time is the historical average time for migrating the expert network corresponding to the CPU to the GPU; determining a first inference duration based on the number of word segments and the first delay time; determining a second inference duration based on the second delay time and the third delay time; if the first inference duration is less than or equal to the second inference duration, then calling the CPU of the hybrid expert model, determining a second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmitting the second output vector group to the GPU; if the first inference duration is greater than the second inference duration, then migrating the second expert network corresponding to the CPU to the GPU, calling the GPU of the hybrid expert model, performing parallel computation based on the second expert network of the input vector and the corresponding input vector, and determining the second output vector group corresponding to the input vector.

[0014] Secondly, this application also provides a model scheduling apparatus. The apparatus includes:

[0015] The acquisition module is used to acquire the text to be inferred and determine the input vector of each word segment of the text to be inferred based on the text to be inferred.

[0016] The determination module is used to determine, based on the input vector of each word segmentation, the expert network group corresponding to each input vector of each expert layer in the hybrid expert model; the expert network group includes: a first expert network and / or a second expert network;

[0017] The first invocation module is used to invoke the graphics processor of the hybrid expert model to determine the first output vector group based on the first expert network of the input vector and the corresponding input vector;

[0018] The second calling module is used to call the central processing unit of the hybrid expert model, determine the second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmit the second output vector group to the graphics processor;

[0019] The output module is used to call the graphics processor of the hybrid expert model to determine the target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group.

[0020] Thirdly, this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the methods in the first aspect above.

[0021] The aforementioned model scheduling method, apparatus, computer equipment, and storage medium acquire the text to be inferred, determine the input vector of each word segment of the text based on the text to be inferred, and then determine the expert network group corresponding to each input vector of each expert layer in the hybrid expert model based on the input vector of each word segment; the expert network group includes: a first expert network and / or a second expert network; then, the graphics processor of the hybrid expert model is invoked, and a first output vector group is determined based on the first expert network and the corresponding input vector; then, the central processing unit of the hybrid expert model is invoked, and a second output vector group is determined based on the second expert network and the corresponding input vector, and the second output vector group is transmitted to the graphics processor; finally, the graphics processor of the hybrid expert model is invoked, and the target output vector corresponding to the text to be inferred is determined based on the first output vector group and the second output vector group, thereby achieving efficient scheduling of the hybrid expert model. Attached Figure Description

[0022] Figure 1 This is a diagram illustrating the application environment of the model scheduling method in one embodiment;

[0023] Figure 2 This is a flowchart illustrating a model scheduling method in one embodiment;

[0024] Figure 3 This is a flowchart illustrating the process of determining the second output vector group in one embodiment;

[0025] Figure 4 This is a structural block diagram of the model scheduling device in one embodiment;

[0026] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] With the rapid development of large-scale language models, hybrid expert models (MOEs) have attracted widespread attention due to their ability to significantly increase the number of model parameters while controlling computational costs. However, such models typically contain hundreds of expert networks, with the total number of parameters far exceeding the memory capacity of a single graphics processing unit (GPU), making it impossible to load them all onto the GPU for computation. Furthermore, during the inference process of MOE models, not all expert parameters participate in the computation; loading all parameters onto the GPU reduces GPU memory utilization.

[0029] Existing technologies typically address the memory bottleneck by storing a portion of the expert network in the CPU's main memory and migrating it to the GPU when needed. However, during model inference, different experts need to be dynamically activated based on the input, leading to frequent transfer of expert network weights between CPU memory and the GPU. This results in significant data transfer and fails to adequately consider the differences in computational characteristics between the CPU and GPU under varying input conditions, limiting the model's inference throughput and efficiency. Therefore, there is an urgent need for an efficient MOE inference scheme that effectively hides data transfer latency under resource constraints and utilizes efficient CPU offloading scheduling.

[0030] The model scheduling method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Server 104 is used to execute the model scheduling method. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0031] To address the aforementioned problems, in one embodiment of this application, such as Figure 2 As shown, a model scheduling method is provided for scheduling a hybrid expert model, which includes a high-frequency expert network and a low-frequency expert network; the high-frequency expert network is deployed on a graphics processing unit (GPU); the low-frequency expert network is deployed on a central processing unit (CPU), and the method includes the following steps:

[0032] Step 201: Obtain the text to be inferred, and determine the input vector of each word segment of the text to be inferred based on the text to be inferred.

[0033] After obtaining the text to be reasoned, the text is segmented into words to obtain multiple words. Then, each word is vectorized to obtain the input vector of each word in the text to be reasoned.

[0034] The text to be reasoned is the text input by the user that needs to be reasoned through a hybrid expert model. The input vector of each word segment of the text to be reasoned is a vector representing a certain actual word segment of the text to be reasoned, and the number of input vectors corresponds to the number of words in the text to be reasoned.

[0035] Step 202: Based on the input vector of each word segmentation, determine the expert network group corresponding to each input vector of each expert layer in the hybrid expert model.

[0036] The expert network group includes: the first expert network and / or the second expert network.

[0037] It should be noted that there are three scenarios for the expert network group: the first scenario is that the expert network group only includes the first expert network, and the number of first expert networks is one or more; the second scenario is that the expert network group only includes the second expert network, and the number of second expert networks is one or more; the third scenario is that the expert network group includes both the first and second expert networks, and the number of first expert networks is one or more, and the number of second expert networks is one or more.

[0038] The first expert network is a high-frequency expert network deployed on the graphics processing unit (GPU). The second expert network is a low-frequency expert network deployed on the central processing unit (CPU). High-frequency expert networks are those used frequently. Low-frequency expert networks are those used infrequently. A graphics processing unit (GPU) is a processor designed specifically for parallel processing of large-scale, regular datasets. A central processing unit (CPU) is a general-purpose processor capable only of serial computation.

[0039] It should be noted that during the computation of the hybrid expert model, not all expert networks are invoked for computation. Instead, for different inputs, the expert network associated with the input is selected for computation. Therefore, the usage frequency of each expert network in the hybrid expert model will be different. In this embodiment, the expert networks with high usage frequency in the hybrid expert model are deployed to the graphics processor, and the expert networks with low usage frequency in the hybrid expert model are deployed to the central processing unit.

[0040] It should be noted that the expert network group corresponding to each input vector in each expert layer is the individual expert network group in each expert layer corresponding to each input vector. Each expert layer will have an expert network group corresponding to each input vector.

[0041] Step 203: Call the graphics processor of the hybrid expert model, and determine the first output vector group based on the first expert network of the input vector and the corresponding input vector.

[0042] The graphics processor using the hybrid expert model inputs each input vector into the corresponding first expert network for parallel computation, thereby obtaining the first output vector group corresponding to each input vector.

[0043] The first expert network for the input vector is the first expert network associated with the input vector. It should be noted that, through the gating network in the hybrid expert model, the correlation between the input vector and each expert network in the hybrid expert model is calculated, and the first few expert networks most correlated with the input vector are taken as the expert networks corresponding to that input vector. There are multiple first output vector groups, each corresponding to a different input vector.

[0044] Step 204: Call the central processing unit of the hybrid expert model, determine the second output vector group based on the second expert network with input vectors and the corresponding input vectors, and transmit the second output vector group to the graphics processor.

[0045] Based on the historical usage frequency of each second expert network in the hybrid expert model, the second expert networks of the input vectors are sorted. Then, based on the order of word segmentation in the text to be predicted, the second expert networks are sorted together. By calling the central processing unit of the hybrid expert model, serial calculations are performed sequentially to obtain the second output vector group corresponding to each input vector. Then, the second output vector group corresponding to each input vector is transmitted to the graphics processor.

[0046] It should be noted that in other embodiments of this application, the second expert networks of the input vector are sorted in advance according to the historical usage frequency of each second expert network in the hybrid expert model, thereby improving the overall efficiency of the model scheduling method.

[0047] The second expert network for the input vectors is a second expert network associated with the input vectors. There are multiple sets of second output vectors, each corresponding to a different input vector.

[0048] Step 205: Call the graphics processor of the hybrid expert model to determine the target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group.

[0049] By invoking the graphics processor of the hybrid expert model, the initial output vector of each input vector is determined based on the first output vector group, the second output vector group, and the network weights corresponding to the first expert network and / or the second expert network in the expert network group of each input vector. Then, the target output vector corresponding to the text to be inferred is determined based on the initial output vector of each input vector and the expert weights corresponding to the initial output vector of each input vector.

[0050] The network weights are the weights corresponding to each first expert network and / or each second expert network, which are determined by the gating network in the hybrid expert model based on the expert network group for each input vector. The expert weights are the weights corresponding to the initial output vectors of each input vector in the hybrid expert model, and are determined by the gating network in the hybrid expert model based on the initial output vectors of each input vector.

[0051] It should be noted that the hybrid expert model includes multiple expert layers, and each expert layer includes multiple expert networks. The expert networks of different expert layers do not overlap.

[0052] In the above model scheduling method, the text to be inferred is obtained, and the input vectors of each word segment of the text to be inferred are determined based on the text to be inferred. Then, based on the input vectors of each word segment, the expert network group corresponding to each input vector of each expert layer in the hybrid expert model is determined. The expert network group includes a first expert network and / or a second expert network. Then, the graphics processor of the hybrid expert model is called, and based on the first expert network of the input vector and the corresponding input vector, a first output vector group is determined. Then, the central processing unit of the hybrid expert model is called, and based on the second expert network of the input vector and the corresponding input vector, a second output vector group is determined and transmitted to the graphics processor. Finally, the graphics processor of the hybrid expert model is called, and based on the first output vector group and the second output vector group, the target output vector corresponding to the text to be inferred is determined, thereby achieving efficient scheduling of the hybrid expert model.

[0053] In other embodiments of this application, obtaining the text to be reasoned and determining the input vectors for each segment of the text to be reasoned based on the text to be reasoned includes:

[0054] Step 1: Obtain the text to be reasoned and perform word segmentation on the text to be reasoned to obtain multiple words.

[0055] The system obtains the text to be inferred from the user input, and then performs word segmentation on the text to be inferred, resulting in multiple word segments.

[0056] The text to be inferred is the text entered by the user.

[0057] For example, the text to be inferred is "Artificial intelligence is changing the world.", with multiple word segments including "artificial", "intelligent", "is", "changing", "world" and ".".

[0058] Step 2: Vectorize multiple word segments to obtain the input vector for each word segment.

[0059] Each word segment is vectorized and converted into a vector, thus obtaining the input vector corresponding to each word segment.

[0060] The input vector is the vector obtained after word segmentation and vectorization. There is a one-to-one correspondence between the input vector and the word segmentation.

[0061] In other embodiments of this application, determining the expert network group corresponding to each input vector of each expert layer in the hybrid expert model based on the input vector of each word segmentation includes:

[0062] Step 1: Based on the input vector of each word segmentation, determine the expert network group corresponding to each input vector of each expert layer in the hybrid expert model.

[0063] The gating network of the hybrid expert model is invoked. Based on the input vector of each word segmentation, the gating network calculates the correlation between the expert network in each expert layer of the hybrid expert model and each input vector. For a specific input vector, the expert networks in each expert layer that are most relevant to the input vector are selected to obtain the expert network group corresponding to the input vector in each expert layer, thereby obtaining the expert network group for each input vector corresponding to each expert layer.

[0064] An expert network group consists of multiple expert networks associated with a given input vector within a specific expert layer.

[0065] Step 2: Invoke the gated network of the hybrid expert model, and determine the network weights corresponding to each first expert network and / or each second expert network in the target expert network group based on the target expert network group.

[0066] The target expert network group is any one of the expert network groups with multiple input vectors corresponding to multiple expert layers. Although this embodiment only performs the relevant steps on the target expert network group, in real-world scenarios, the relevant steps will be performed on each expert network group.

[0067] A gated network based on a hybrid expert model is invoked. This network, using a dynamic and learnable routing mechanism, calculates and determines the network weights for each first expert network and / or each second expert network within the target expert network group. In other words, weights are assigned to each expert network in the target expert network group.

[0068] It should be noted that the network weights corresponding to the first expert network and / or each of the second expert networks fall into three categories: the first category includes only the network weights corresponding to the first expert network, and the number of network weights corresponding to the first expert network is one or more; the second category includes only the network weights corresponding to the second expert network, and the number of network weights corresponding to the second expert network is one or more; the third category includes both the network weights corresponding to the first expert network and the network weights corresponding to the second expert network, with both categories having one or more network weights corresponding to the first expert network and one or more network weights corresponding to the second expert network.

[0069] It should be noted that the gating network is based on the target expert network group, and the determination of the network weights corresponding to each first expert network and / or each second expert network in the target expert network group is achieved by the gating network itself in the hybrid expert model.

[0070] In other embodiments of this application, the process of calling the graphics processor of the hybrid expert model to determine the first output vector group based on the first expert network of the input vector and the corresponding input vector specifically involves:

[0071] For the target expert network group, the graphics processor of the hybrid expert model is invoked to perform parallel computation based on the first expert network of the input vector and the corresponding input vector to determine the first output vector group corresponding to the input vector.

[0072] It should be noted that in this embodiment, there can be one or more first expert networks.

[0073] It should be noted that the target expert network group is any expert network group among the expert network groups corresponding to each input vector of each expert layer. Although this embodiment only performs the relevant steps on the target expert network group, in real-world scenarios, the relevant steps will be performed on each expert network group.

[0074] It should be noted that the graphics processing unit (GPU) is a processor capable of parallel computing. The first expert network for the input vector is the first expert network associated with the input vector. The first output vector group corresponding to the input vector consists of multiple output vectors obtained after the input vector is input to each of the first expert networks in the target expert network group.

[0075] The graphics processor inputs the input vectors corresponding to the target expert network groups into the first expert network in the target expert network group for parallel computation, thereby using the output of the first expert network in each target expert network group as the first output vector group corresponding to the input vector.

[0076] It should be noted that each word segment corresponds to a group of expert networks at each expert level.

[0077] In other embodiments of this application, such as Figure 3 As shown, the process of calling the central processing unit of the hybrid expert model, determining the second output vector group based on the second expert network and the corresponding input vectors, and transmitting the second output vector group to the graphics processor includes:

[0078] Step 301: For the target expert network group, obtain the historical usage frequency of multiple second expert networks in the target expert group.

[0079] It should be noted that the target expert network group is any expert network group among the expert network groups corresponding to each input vector of each expert layer. Although this embodiment only performs the relevant steps on the target expert network group, in real-world scenarios, the relevant steps will be performed on each expert network group.

[0080] It should be noted that during the computation of the hybrid expert model, not all expert networks are invoked. Instead, for different inputs, the expert network associated with the input is selected for computation. Therefore, the usage frequency of each expert network in the hybrid expert model will vary. The historical usage frequency refers to the historical usage frequency of multiple second expert networks in the target expert group.

[0081] Step 302: Sort the multiple second expert networks according to their historical usage frequency to determine the sorted multiple second expert networks.

[0082] Based on the historical usage frequency of multiple second expert networks in the target expert network group, the multiple second expert networks are sorted in descending order to obtain the sorted multiple second expert networks.

[0083] The sorted second expert networks are obtained by sorting the second expert networks according to their historical usage frequency.

[0084] Step 303: In the order of the sorted second expert networks, the central processing unit of the hybrid expert model is called sequentially. Based on the input vector of the second expert network and the corresponding input vector, the second output vector group is determined and the second output vector group is transmitted to the graphics processor.

[0085] Following the order of the sorted second expert networks, the central processing unit of the hybrid expert model is invoked to sequentially input the input vectors corresponding to the target expert network group into the sorted second expert networks, and the outputs of each second expert network are used as the second output vector group, which is then transmitted to the graphics processor.

[0086] It should be noted that, in other embodiments, the steps of obtaining the historical usage frequency of multiple second expert networks in the target expert group and sorting the multiple second expert networks according to their historical usage frequency can be performed in advance to determine the sorted multiple second expert networks, so as to improve the overall efficiency of the model scheduling method.

[0087] In other embodiments of this application, the central processing unit of the hybrid expert model is sequentially invoked in the order of the sorted plurality of second expert networks, and a second output vector group is determined based on the input vectors of the second expert networks and the corresponding input vectors, and the second output vector group is transmitted to the graphics processor, including:

[0088] Step 1: Based on the target second expert network, call the central processing unit of the hybrid expert model to determine the second output vector corresponding to the target second expert network based on the input vector of the target second expert network and the corresponding input vector.

[0089] The target second expert network is any one of the sorted second expert networks. Although this embodiment only performs the relevant steps on the target second expert network, in real-world scenarios, the relevant steps would be performed on each of the sorted second expert networks.

[0090] The second output vector corresponding to the target second expert network is the output vector obtained by inputting the corresponding input vector into the target second expert network and performing calculations through the target second expert network.

[0091] The central processing unit of the hybrid expert model is invoked to input the input vector into the target second expert network. Through the calculation of the target second expert network, the second output vector corresponding to the target second expert network is obtained.

[0092] Step 2: Cache the second output vector into the first cache of the central processing unit and generate the first event identifier corresponding to the target second expert network.

[0093] The first event identifier marks the end of the computation of the target second expert network. The first cache is a storage area pre-set in the central processing unit for storing the second output vectors corresponding to each second expert network.

[0094] The central processing unit caches the second output vector into the first cache area of ​​the central processing unit and generates a first event flag indicating the end of the computation of the target second expert network.

[0095] Step 3: Based on the next second expert network of the target second expert network, call the central processing unit of the hybrid expert model, determine the second output vector corresponding to the next second expert network based on the input vector of the next second expert network and the corresponding input vector; and simultaneously detect the first event identifier corresponding to the target second expert network. If the first event identifier corresponding to the target second expert network exists, copy the second output vector corresponding to the target second expert network from the first cache of the central processing unit to the second cache of the graphics processing unit.

[0096] The next second expert network after the target second expert network is the second expert network that follows the target second expert network in the ordered sequence of multiple second expert networks. The second output vector corresponding to the next second expert network is the output vector obtained by inputting the corresponding input vector into the next second second expert network and performing the operation. The second buffer is a storage area pre-set in the graphics processor to store the second output vectors corresponding to each second expert network.

[0097] While the CPU of the hybrid expert model inputs the input vector to the next second expert network for computation, the GPU of the hybrid expert model checks whether the first event identifier corresponding to the target second expert network exists. If the first event identifier corresponding to the target second expert network exists, the GPU copies the second output vector corresponding to the target second expert network from the first cache of the CPU to the second cache of the GPU.

[0098] It should be noted that while the next expert network is being computed, the first event identifier of the previous expert network is determined in parallel, thereby copying the computation result of the previous expert network from the first buffer to the second buffer in parallel.

[0099] In other embodiments of this application, the graphics processor of the hybrid expert model is invoked to determine the target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group, including:

[0100] Step 1: Call the graphics processor of the hybrid expert model to determine the initial output vector of each input vector based on the first output vector group, the second output vector group of each input vector, and the network weights corresponding to each first expert network and / or each second expert network in the expert network group of each input vector.

[0101] By calling the graphics processor of the hybrid expert model, the product of the output vector in the first output vector group and the second output vector group of each input vector and the corresponding network weight is calculated, and the product results belonging to the same input vector are added together to obtain the initial output vector of each input vector.

[0102] The network weights are the weights corresponding to each first expert network and / or each second expert network, and are determined by the gating network in the hybrid expert model based on the expert network group for each input vector. The initial output vector represents the sum of the outputs of the first expert network and / or each second expert network corresponding to the input vector.

[0103] Step 2: Invoke the gating network of the hybrid expert model and determine the expert weights corresponding to the initial output vectors of each input vector based on the initial output vectors of each input vector.

[0104] The system invokes a gated network based on a hybrid expert model. The gated network calculates and determines the expert weights corresponding to the initial output vectors of each input vector through a dynamic and learnable routing mechanism.

[0105] Expert weights are the expert weights corresponding to the initial output vectors of each input vector.

[0106] It should be noted that the gating network determines the expert weights corresponding to the initial output vectors of each input vector based on the initial output vectors of each input vector. This determination is achieved by the gating network itself within the hybrid expert model.

[0107] Step 3: Determine the target output vector corresponding to the text to be inferred based on the initial output vector of each input vector and the expert weights corresponding to the initial output vector of each input vector.

[0108] The initial output vector of each input vector is multiplied by the expert weight corresponding to the initial output vector of each input vector, and the results of each multiplication are added together to obtain the target output vector corresponding to the text to be inferred.

[0109] The target output vector is the vector obtained by the hybrid expert model based on the text to be reasoned.

[0110] Other embodiments of this application also include:

[0111] Step 1: Obtain the number of segments, the first delay time, the second delay time, and the third delay time for the multiple segments corresponding to the text to be inferred.

[0112] The first delay time is the historical average time for calling the central processing unit (CPU) to determine the output vector group based on the expert network; the second delay time is the historical average time for calling the graphics processing unit (GPU) to determine the output vector group based on the expert network; and the third delay time is the historical average time for migrating the expert network corresponding to the CPU to the GPU.

[0113] The number of words corresponding to the text to be inferred is the number of words obtained after the text to be inferred is segmented.

[0114] Step 2: Determine the first inference duration based on the number of segments in multiple word segments and the first delay time.

[0115] Multiply the number of segments by the first delay time to obtain the first inference time.

[0116] Step 3: Determine the second inference duration based on the second delay time and the third delay time.

[0117] Adding the second delay time to the third delay time yields the second inference time.

[0118] Step 4: If the first inference time is less than or equal to the second inference time, the central processing unit of the hybrid expert model is invoked to determine the second output vector group based on the second expert network with input vectors and the corresponding input vectors, and the second output vector group is transmitted to the graphics processor.

[0119] If the first inference time is less than or equal to the second inference time, the second expert networks of the input vectors are sorted according to the historical usage frequency of each second expert network in the hybrid expert model. Then, the second expert networks are sorted based on the order of word segmentation in the text to be predicted. The central processing unit of the hybrid expert model is called to perform serial calculations in sequence to obtain the second output vector group corresponding to each input vector. Then, the second output vector group corresponding to each input vector is transmitted to the graphics processor.

[0120] Step 5: If the first inference time is longer than the second inference time, then the second expert network corresponding to the central processing unit is migrated to the graphics processing unit, the graphics processing unit of the hybrid expert model is called, and the second expert network based on the input vector and the corresponding input vector are used for parallel computation to determine the second output vector group corresponding to the input vector.

[0121] When the first inference time is longer than the second inference time, the second expert network corresponding to the central processing unit is migrated to the graphics processing unit. The graphics processing unit with the hybrid expert model inputs each input vector into the corresponding first expert network for parallel computation, thereby obtaining the first output vector group corresponding to each input vector.

[0122] It should be noted that the model scheduling method deploys high-frequency experts in the image processor (GPU) and low-frequency experts in the central processing unit (CPU) through "model partitioning and heat-aware placement", and pre-sets the first and second buffers; combined with the linear latency model, it dynamically decides the path of "CPU local computation-transfer" or "direct GPU computation"; and uses an event-driven multi-level asynchronous pipeline to parallelize the computation of low-frequency experts, data copying, and accumulation through the image processor (GPU), thus hiding the transmission latency.

[0123] In other embodiments of this application, a CPU-GPU asynchronous pipeline scheduling method and system for efficient inference of large hybrid expert models under resource constraints is also provided. The system is deployed on a heterogeneous computing platform comprising at least one central processing unit (CPU) and at least one graphics processing unit (GPU), and includes:

[0124] The model partitioning and placement module is used to analyze the hybrid expert model structure and the historical call frequency of each expert network. It divides the model into non-expert layers and expert layers, loads the non-expert layers and high-frequency expert networks into GPU memory, stores low-frequency expert networks in CPU memory, and records the storage location information of each expert network.

[0125] The memory management module is used to pre-allocate a double-buffered memory region for each expert network stored in the CPU memory during the initialization phase. The double-buffered memory region includes a first buffer on the CPU side and a second buffer on the GPU side.

[0126] The compute stream management module is used to create and manage multiple CUDA streams on the GPU, including at least one compute stream for performing expert network computations and a copy stream for managing asynchronous data transfers.

[0127] The thread pool scheduling module is used to create and maintain a computing thread pool on the CPU side to receive and asynchronously execute computing tasks of expert networks located on the CPU.

[0128] An asynchronous pipeline scheduling module is used to coordinate the above modules and schedule the computation process of multiple expert networks activated within a single hybrid expert layer. The scheduling includes: grouping the activated experts into GPU expert groups and CPU expert groups according to the storage location of the expert networks; scheduling the computation of the GPU expert groups in parallel; for the CPU expert groups, starting a multi-level asynchronous pipeline based on double buffering and event-driven methods, sequentially executing the computation submission, asynchronous copying, and result accumulation stages of each CPU expert, and overlapping the computation and data transfer operations of continuous expert networks by alternately using double buffered memory.

[0129] In other embodiments of this application, in the double-buffered memory area pre-allocated by the memory management module, the first buffer on the CPU side is used to receive the calculation results of the corresponding CPU expert network and supports direct memory access (DMA) transfer; the second buffer on the GPU side is used to temporarily store the expert calculation results copied from the CPU side, waiting for the calculation stream to consume them.

[0130] In other embodiments of this application, the copy stream created by the computation stream management module is independent of the computation stream and is dedicated to performing asynchronous data copy tasks from the first buffer to the second buffer, thereby realizing the parallel execution of computation tasks and data transmission tasks.

[0131] In other embodiments of this application, the operation of the multi-stage asynchronous pipeline scheduled by the asynchronous pipeline scheduling module for the i-th CPU expert network includes:

[0132] In the first level, the computation task of the i-th CPU expert network is submitted to the thread pool scheduling module, specifying that the computation result be written to the first buffer and setting a CPU computation completion event.

[0133] In the second level, the CPU computation completion event of the (i-1)th CPU expert network is monitored. When the event is triggered, the operation of asynchronously copying the computation result of the (i-1)th expert network from the first buffer to the second buffer is initiated in the copy stream, and a GPU copy completion event is recorded.

[0134] In the third level, the GPU copy completion event of the (i-1)th CPU expert network is monitored in the computation flow. When the event is triggered, the computation result of the (i-1)th expert network is accumulated from the GPU buffer to the intermediate output tensor of the model.

[0135] After processing the i-th expert network, the pointers of the current buffer and the previous buffer are switched to prepare for processing the (i+1)-th expert network.

[0136] In other embodiments of this application, a CPU-GPU asynchronous pipeline scheduling method is also provided, applied to the above-mentioned system, including:

[0137] Step S1: Model analysis and initialization phase. Statistically analyze the historical call frequency of the expert network, preload the high-frequency expert network into the GPU memory, and store the low-frequency expert network in the CPU memory; pre-allocate double-buffered memory for the CPU experts; create GPU computing stream, copy stream, and CPU computing thread pool.

[0138] Step S2: For each expert layer of the model, perform routing calculations on the input tokens and select the top-k expert networks to be activated based on the output of the gating network.

[0139] Step S3: Based on the preset expert network storage location information, divide the activated expert network into GPU expert group and CPU expert group.

[0140] Step S4: Parallel processing of the GPU expert group, concurrently executing the forward computation of all GPU expert networks in the GPU computation flow, and accumulating the computation results into the intermediate output tensor.

[0141] Step S5: Pipeline process the CPU expert group, and start a multi-level asynchronous pipeline for each CPU expert network in sequence. The pipeline overlaps the three operations of the current expert network's calculation, the asynchronous copy of the previous expert network's calculation result, and the GPU-side accumulation of the copied expert network's calculation result.

[0142] Step S6: Input the intermediate output tensor processed by steps S4 and S5 into the subsequent layers of the model, and repeat steps S2 to S5 until the model inference is completed.

[0143] In other embodiments of this application, the multi-stage asynchronous pipeline described in step S5 specifically includes:

[0144] Step S51: Submit the current expert network's computation task to the CPU computation thread pool for asynchronous execution, write the computation result to the first buffer of the double-buffered memory, and record the computation completion event;

[0145] Step S52: If it is not the first expert network, wait for the completion event of the previous expert network's computation, and start the operation of asynchronously copying the computation result of the previous expert network from the first buffer of the double-buffered memory to the second buffer in the GPU copy stream, and record the copy completion event;

[0146] Step S53: If it is not the first expert network, wait for the copy completion event of the previous expert in the GPU computation flow, and accumulate the computation results of the previous expert network from the second buffer to the final output tensor;

[0147] Step S54: Switch the pointer to the double-buffered memory in preparation for processing the subsequent expert network.

[0148] This application constructs a latency model to pre-estimate the optimal computation method for the current input size, thereby achieving fast inference with limited GPU resources. Experts are placed based on their usage frequency, reducing data transfer frequency. Furthermore, computation and transmission can be organized into an asynchronous pipeline, maximizing parallel execution of these operations and avoiding resource idleness caused by synchronous waiting, significantly improving the overall system's computational efficiency and throughput. The method of this invention is not dependent on a specific model architecture and can be widely applied to various hybrid expert models.

[0149] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides a model scheduling apparatus for implementing the model scheduling method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more model scheduling apparatus embodiments provided below can be found in the limitations of the model scheduling method described above, and will not be repeated here.

[0151] In one embodiment of this application, such as Figure 4 As shown, a model scheduling device is provided, comprising:

[0152] The acquisition module 100 is used to acquire the text to be inferred and determine the input vector of each word segment of the text to be inferred based on the text to be inferred.

[0153] The determining module 200 is configured to determine, based on the input vector of each word segmentation, an expert network group corresponding to each input vector of each expert layer in the hybrid expert model; the expert network group includes: a first expert network and / or a second expert network;

[0154] The first calling module 300 is used to call the graphics processor of the hybrid expert model to determine the first output vector group based on the first expert network of the input vector and the corresponding input vector.

[0155] The second calling module 400 is used to call the central processing unit of the hybrid expert model, determine the second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmit the second output vector group to the graphics processor;

[0156] The output module 500 is used to call the graphics processor of the hybrid expert model to determine the target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group.

[0157] In one embodiment of this application, the acquisition module 100 is further configured to acquire the text to be inferred, and to perform word segmentation on the text to be inferred to obtain multiple word segments; and to perform vectorization on the multiple word segments to obtain the input vector of each word segment.

[0158] In one embodiment of this application, the determining module 200 is further configured to determine, based on the input vector of each word segmentation, the expert network group corresponding to each input vector of each expert layer in the hybrid expert model; and invoke the gated network of the hybrid expert model to determine the network weights corresponding to each first expert network and / or each second expert network in the target expert network group based on the target expert network group.

[0159] In one embodiment of this application, the first invocation module 300 is further configured to invoke the graphics processor of the hybrid expert model for the target expert network group, perform parallel computation based on the first expert network of the input vector and the corresponding input vector, and determine the first output vector group corresponding to the input vector.

[0160] In one embodiment of this application, the second invocation module 400 is further configured to: obtain the historical usage frequency of multiple second expert networks in the target expert network group; sort the multiple second expert networks according to their historical usage frequency to determine the sorted multiple second expert networks; sequentially invoke the central processing unit of the hybrid expert model in the order of the sorted multiple second expert networks; determine the second output vector group based on the second expert network with the input vector and the corresponding input vector; and transmit the second output vector group to the graphics processor.

[0161] In one embodiment of this application, the second invocation module 400 is further configured to: invoke the central processing unit (CPU) of the hybrid expert model according to the target second expert network; determine the second output vector corresponding to the target second expert network based on the target second expert network and the corresponding input vector; cache the second output vector in the first cache of the CPU and generate a first event identifier corresponding to the target second expert network; invoke the CPU of the hybrid expert model according to the next second expert network of the target second expert network; determine the second output vector corresponding to the next second expert network based on the next second expert network and the corresponding input vector; and synchronously detect the first event identifier corresponding to the target second expert network. If the first event identifier corresponding to the target second expert network exists, copy the second output vector corresponding to the target second expert network from the first cache of the CPU to the second cache of the graphics processor.

[0162] In one embodiment of this application, the output module 500 is further configured to invoke the graphics processor of the hybrid expert model to determine the initial output vector of each input vector based on the first output vector group, the second output vector group, and the network weights corresponding to each first expert network and / or each second expert network in the expert network group of each input vector; invoke the gating network of the hybrid expert model to determine the expert weights corresponding to the initial output vectors of each input vector based on the initial output vectors of each input vector; and determine the target output vector corresponding to the text to be inferred based on the initial output vectors of each input vector and the expert weights corresponding to the initial output vectors of each input vector.

[0163] In one embodiment of this application, the model scheduling method further includes: obtaining the number of word segments, a first delay time, a second delay time, and a third delay time for multiple word segments corresponding to the text to be inferred; the first delay time is the historical average time for calling the central processing unit (CPU) to determine the first output vector group based on the expert network; the second delay time is the historical average time for calling the graphics processing unit (GPU) to determine the second output vector group based on the expert network; the third delay time is the historical average time for migrating the expert network corresponding to the CPU to the GPU; determining a first inference duration based on the number of word segments and the first delay time; determining a second inference duration based on the second delay time and the third delay time; if the first inference duration is less than or equal to the second inference duration, then calling the CPU of the hybrid expert model, determining the second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmitting the second output vector group to the GPU; if the first inference duration is greater than the second inference duration, then migrating the second expert network corresponding to the CPU to the GPU, calling the GPU of the hybrid expert model, performing parallel computation based on the second expert network of the input vector and the corresponding input vector, and determining the second output vector group corresponding to the input vector.

[0164] Each module in the aforementioned model scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0165] In one embodiment of this application, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 5As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores all relevant data for executing a model scheduling method. The network interface communicates with external terminals via a network connection. The computer program, when executed by the processor, implements a model scheduling method.

[0166] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0167] In one embodiment of this application, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the model scheduling method in the above embodiment.

[0168] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, the computer program being executed by a processor to implement the steps of the model scheduling method in the above-described method embodiments.

[0169] In one embodiment of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the model scheduling method in the above-described method embodiments.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0172] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model scheduling method, characterized in that, The model scheduling method is used to schedule a hybrid expert model, which includes a high-frequency expert network and a low-frequency expert network; the high-frequency expert network is deployed on a graphics processing unit (GPU); the low-frequency expert network is deployed on a central processing unit (CPU); the method includes: Obtain the text to be reasoned, and determine the input vector of each word segment of the text based on the text to be reasoned; Based on the input vector of each word segmentation, an expert network group corresponding to each input vector of each expert layer in the hybrid expert model is determined; the expert network group includes: a first expert network and / or a second expert network; The graphics processor of the hybrid expert model is invoked to determine the first output vector group based on the first expert network of the input vector and the corresponding input vector; The central processing unit of the hybrid expert model is invoked to determine the second output vector group based on the second expert network of the input vector and the corresponding input vector, and the second output vector group is transmitted to the graphics processor. The graphics processor of the hybrid expert model is invoked to determine the target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group.

2. The model scheduling method according to claim 1, characterized in that, Obtaining the text to be inferred, and determining the input vector for each word segment of the text based on the text to be inferred, includes: Obtain the text to be inferred, and perform word segmentation on the text to be inferred to obtain multiple words; The multiple word segments are vectorized to obtain the input vector of each word segment.

3. The model scheduling method according to claim 1, characterized in that, The step of determining the expert network group corresponding to each input vector of each expert layer in the hybrid expert model based on the input vector of each word segmentation includes: Based on the input vector of each word segmentation, determine the expert network group corresponding to each input vector of each expert layer in the hybrid expert model; The gating network of the hybrid expert model is invoked, and the network weights corresponding to each first expert network and / or each second expert network in the target expert network group are determined based on the target expert network group.

4. The model scheduling method according to claim 1, characterized in that, The graphics processor that invokes the hybrid expert model determines the first output vector group based on the first expert network of the input vector and the corresponding input vector, including: For the target expert network group, the graphics processor of the hybrid expert model is invoked to perform parallel computation based on the first expert network of the input vector and the corresponding input vector to determine the first output vector group corresponding to the input vector.

5. The model scheduling method according to claim 1, characterized in that, The process of calling the hybrid expert model by the central processing unit, determining the second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmitting the second output vector group to the graphics processor includes: For the target expert network group, obtain the historical usage frequency of multiple second expert networks in the target expert group; Based on the historical usage frequency of the plurality of second expert networks, the plurality of second expert networks are sorted to determine the sorted plurality of second expert networks; The central processing unit of the hybrid expert model is called sequentially in the order of the sorted second expert networks. Based on the second expert networks with the input vectors and the corresponding input vectors, the second output vector group is determined and transmitted to the graphics processor.

6. The model scheduling method according to claim 5, characterized in that, The step of sequentially calling the central processing unit of the hybrid expert model in the order of the sorted second expert networks, determining the second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmitting the second output vector group to the graphics processor includes: Based on the target second expert network, the central processing unit of the hybrid expert model is invoked to determine the second output vector corresponding to the target second expert network based on the input vector and the corresponding input vector; The second output vector is cached in the first cache area of ​​the central processing unit, and a first event identifier corresponding to the target second expert network is generated; Based on the next second expert network of the target second expert network, the central processing unit of the hybrid expert model is invoked to determine the second output vector corresponding to the next second expert network based on the input vector and the corresponding input vector; and simultaneously detects the first event identifier corresponding to the target second expert network. If the first event identifier corresponding to the target second expert network exists, the second output vector corresponding to the target second expert network is copied from the first cache of the central processing unit to the second cache of the graphics processing unit.

7. The model scheduling method according to claim 1, characterized in that, The process of calling the graphics processor of the hybrid expert model to determine the target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group includes: The graphics processor of the hybrid expert model is invoked to determine the initial output vector of each input vector based on the first output vector group, the second output vector group of each input vector, and the network weights corresponding to each first expert network and / or each second expert network in the expert network group of each input vector. The gating network of the hybrid expert model is invoked to determine the expert weights corresponding to the initial output vectors of each input vector based on the initial output vectors of each input vector. Based on the initial output vector of each input vector and the expert weights corresponding to the initial output vector of each input vector, the target output vector corresponding to the text to be inferred is determined.

8. The model scheduling method according to claim 4, characterized in that, The method further includes: Obtain the number of words segmented for the text to be inferred, the first delay time, the second delay time, and the third delay time; the first delay time is the historical average time for calling the central processing unit to determine the output vector group based on the expert network; the second delay time is the historical average time for calling the graphics processing unit to determine the output vector group based on the expert network; the third delay time is the historical average time for migrating the expert network corresponding to the central processing unit to the graphics processing unit. The first inference duration is determined based on the number of words segmented and the first delay time. The second inference duration is determined based on the second delay time and the third delay time; If the first inference time is less than or equal to the second inference time, then the central processing unit of the hybrid expert model is invoked to determine the second output vector group based on the second expert network of the input vector and the corresponding input vector, and the second output vector group is transmitted to the graphics processor. If the first inference time is longer than the second inference time, the second expert network corresponding to the central processing unit is migrated to the graphics processing unit, and the graphics processing unit of the hybrid expert model is called to perform parallel computation based on the second expert network of the input vector and the corresponding input vector to determine the second output vector group corresponding to the input vector.

9. A model scheduling device, characterized in that, The device includes: The acquisition module is used to acquire the text to be inferred and determine the input vector of each word segment of the text to be inferred based on the text to be inferred. The determination module is used to determine the expert network group corresponding to each input vector of each expert layer in the hybrid expert model based on the input vector of each word segmentation; the expert network group includes: a first expert network and / or a second expert network; The first invocation module is used to invoke the graphics processor of the hybrid expert model to determine the first output vector group based on the first expert network of the input vector and the corresponding input vector; The second calling module is used to call the central processing unit of the hybrid expert model, determine the second output vector group based on the second expert network of the input vector and the corresponding input vector, and transmit the second output vector group to the graphics processor; The output module is used to call the graphics processor of the hybrid expert model and determine the target output vector corresponding to the text to be inferred based on the first output vector group and the second output vector group.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images