Hybrid expert model reasoning acceleration method, device, equipment, medium and program

Abstract

The embodiment of the present invention discloses a hybrid expert model reasoning acceleration method, device, equipment, medium and program, wherein the method includes: loading the target association parameters of the hybrid expert model; wherein the target association parameters include the non-expert network parameters and the expert network reference vector of the hybrid expert model; predicting the target expert network of the network layer to be run according to the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run; preloading the target expert network of the network layer to be run to the device storage running the hybrid expert model to perform the model reasoning process. The technical solution of the embodiment of the present invention can reduce the runtime behaviors such as data movement and memory access of the hybrid expert model, save memory usage, and improve the computing performance and operation speed of the model, thereby improving the utilization of hardware resources and the execution efficiency of the model.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of computer and artificial intelligence technology, and in particular, to a hybrid expert model reasoning acceleration method, device, equipment, medium and program. Background Art

[0002] Mixture of Experts (MoE) is an integrated learning method that combines multiple specialized sub-models, or "experts", to form an overall model. Each "expert" contributes in his or her area of ​​expertise, and the mechanism that determines which "expert" participates in answering a specific question is a "gated network". The core idea of ​​MoE is to divide the input data into multiple regions according to the task type, and assign one or more expert models to process the data in each region, thereby improving the overall performance of the model. The mixed expert model is a sparse neural network that can greatly increase the number of model parameters without increasing the amount of model calculation, thereby improving the model's capabilities.

[0003] In the process of implementing the present invention, the inventors found that due to the sparsity of the hybrid expert model, usually one to two experts are selected for processing for each token (the smallest unit of input data used by the model), and different data are processed by different experts. During the model operation, loading all the expert parameters of the model will bring great storage pressure and bandwidth pressure, reducing the model calculation efficiency. Summary of the invention

[0004] The embodiments of the present invention provide a hybrid expert model reasoning acceleration method, device, equipment, medium and program, which can reduce the runtime behaviors such as data movement and memory access of the hybrid expert model, save memory usage, and improve the computing performance and operation speed of the model, thereby improving the utilization of hardware resources and the execution efficiency of the model.

[0005] According to one aspect of the present invention, a hybrid expert model reasoning acceleration method is provided, comprising:

[0006] Loading target-related parameters of the hybrid expert model; wherein the target-related parameters include non-expert network parameters and expert network reference vectors of the hybrid expert model;

[0007] Predicting a target expert network of the network layer to be run according to a current layer output feature vector of the current running network layer of the hybrid expert model and an expert network reference vector of the network layer to be run;

[0008] The target expert network of the network layer to be run is preloaded into the storage of the device running the hybrid expert model to perform the model reasoning process.

[0009] According to another aspect of the present invention, a hybrid expert model reasoning acceleration device is provided, comprising:

[0010] A target-associated parameter loading module, used to load target-associated parameters of the hybrid expert model; wherein the target-associated parameters include non-expert network parameters and expert network reference vectors of the hybrid expert model;

[0011] A target expert network prediction module, used for predicting a target expert network of the network layer to be run according to a current layer output feature vector of the current running network layer of the hybrid expert model and an expert network reference vector of the network layer to be run;

[0012] The target expert network preloading module is used to preload the target expert network of the network layer to be run into the device storage running the hybrid expert model to perform the model reasoning process.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the hybrid expert model reasoning acceleration method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the hybrid expert model reasoning acceleration method described in any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the hybrid expert model reasoning acceleration method according to any embodiment of the present invention is implemented.

[0019] The embodiment of the present invention loads target-related parameters such as non-expert network parameters and expert network reference vectors of the hybrid expert model to predict the target expert network of the network layer to be run according to the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run, thereby preloading the target expert network of the network layer to be run into the device storage running the hybrid expert model to perform the model reasoning process. The preloading process of the expert network can be realized, thereby accelerating the reasoning process of the hybrid expert model, solving the problems of large storage and bandwidth pressure caused by the need to load all expert networks during the operation of the existing hybrid expert model, and low model calculation efficiency. The runtime behaviors such as data movement and memory access of the hybrid expert model can be reduced, memory usage can be saved, and the calculation performance and operation speed of the model can be improved, thereby improving the utilization of hardware resources and the execution efficiency of the model.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a hybrid expert model reasoning acceleration method provided in Embodiment 1 of the present invention;

[0023] Figure 2 It is a schematic diagram of the effect of a certain layer of network structure in a hybrid expert model provided in the first embodiment of the present invention;

[0024] Figure 3 is a flow chart of a hybrid expert model reasoning acceleration method provided by Embodiment 2 of the present invention;

[0025] Figure 4 It is a schematic diagram of a process of training an expert prediction perception model provided by the second embodiment of the present invention;

[0026] Figure 5 It is a structural diagram of an input layer of an expert prediction perception model provided in the second embodiment of the present invention;

[0027] Figure 6 1 is a schematic diagram of the effect of clustering a reference vector provided by the second embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of a process for optimizing and updating an expert prediction perception model provided by the second embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram of the time comparison effect of a baseline solution and a Predict solution provided in the second embodiment of the present invention;

[0030] Fig. 9 is a schematic diagram of a hybrid expert model reasoning acceleration device provided in Embodiment 3 of the present invention;

[0031] Fig.10 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "include", "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatus.

[0034] Embodiment 1

[0035] Figure 1 This is a flow chart of a hybrid expert model reasoning acceleration method provided by the first embodiment of the present invention. This embodiment is applicable to the case where the expert network in the hybrid expert model is preloaded to accelerate the model reasoning process. The method can be executed by a hybrid expert model reasoning acceleration device, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as it can execute the hybrid expert model reasoning acceleration method, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0036] S110, loading target association parameters of the hybrid expert model; wherein the target association parameters include non-expert network parameters and an expert network reference vector of the hybrid expert model.

[0037] Among them, the target-related parameters can be related parameters that can be loaded once during the operation of the hybrid expert model. Non-expert network parameters can be network parameters in the hybrid expert model that are not related to the expert network, such as but not limited to related parameters of the Attention module. The expert network reference vector can be the reference vector of each expert network in the hybrid expert model, that is, the reference vector of the expert network involved in the prediction.

[0038] Before the hybrid expert model is loaded into the device for operation, the non-expert network parameters and expert network benchmark vectors of the hybrid expert model can be loaded at one time. Since the parameter scale of the non-expert network parameters and the expert network benchmark vectors is generally relatively small, and they are required in each token reasoning process, the one-time loading of the non-expert network parameters and the expert network benchmark vectors will not have a significant impact on the overall operation efficiency of the hybrid expert model. The expert network benchmark vector can be a benchmark vector of each expert network in the hybrid expert model, and can characterize the benchmark characteristics of each expert network in the hybrid expert model. It can be understood that different types of expert networks have different vector contents of the corresponding calculated expert network benchmark vectors. By loading the non-expert network parameters and the expert network benchmark vectors of the hybrid expert model, the embodiment of the present invention can ensure the operation process of the hybrid expert model through the loaded non-expert network parameters, and predict and preload the expert network of each layer through the loaded expert network benchmark vectors.

[0039] S120, predicting a target expert network of the network layer to be run according to the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run.

[0040] Among them, the current running network layer can be a network layer structure in the running state of the hybrid expert model during operation. The network layer to be run can be the network layer structure of the next layer or the next layers of the hybrid expert model waiting to enter the running state. That is, the network layer to be run can be the next layer of network structure or the next layers of network structure of the current running network layer. Exemplarily, assuming that the current running network layer is the Nth network layer of the hybrid expert model, the network layer to be run can be the N+1th network layer of the hybrid expert model, or the network layer to be run can be the N+1th network layer or the N+2th network layer or the N+3th network layer of the hybrid expert model...etc. The current layer output feature vector can be the output feature vector of the current network layer. Optionally, for the current running network layer, the current layer output feature vector can be the output feature vector of the current network layer; for the network layer to be run, the current layer output feature vector can be the input feature vector referenced when predicting the expert network for the current network layer. The target expert network can be the expert network predicted by the network layer to be run.

[0041] In an embodiment of the present invention, after the hybrid expert model runs to the currently running network layer, the output feature vector of the currently running network layer can be obtained. Optionally, the current layer output feature vector can be the output feature vector of the currently running network layer, or it can also be the weighted value of the output feature vectors of each network layer structure before the currently running network layer, or it can also be the weighted value of the output feature vectors of a certain number of network layer structures before the currently running network layer. The embodiment of the present invention does not limit the specific calculation method of the current layer output feature vector. It can be understood that when the current layer output feature vector is calculated using the weighted value of the output feature vectors of each network layer structure before the currently running network layer, the accuracy of the predicted target expert network is higher.

[0042] Figure 2 Schematic diagram of the effect of a certain layer of network structure in a hybrid expert model provided in the first embodiment of the present invention. Figure 2 As shown, assuming that the N-1th layer network structure of the hybrid expert model is the current running network layer of the hybrid expert model, and the Nth layer network structure is one of the network layers to be run of the hybrid expert model, then the output feature vector of the N-1th layer of the hybrid expert model, that is, Tensor 1, is the input feature vector referenced when the Nth layer network predicts the expert network, and serves as the reference vector value for predicting the expert network of the Nth layer network. Alternatively, the weighted values ​​of Tensor1 of each network layer from the 1st layer to the N-1th layer can be obtained to obtain the input feature vector referenced when the Nth layer network structure predicts the expert network, and serves as the reference vector value for predicting the expert network of the Nth layer network.

[0043] It is understandable that, since each layer of the hybrid expert model may be configured with multiple expert networks, the network layer to be run may also be configured with multiple expert networks. When the hybrid expert model runs to the current running network layer, the expert network reference vectors corresponding to each expert network in the network layer to be run can be extracted and obtained, so as to calculate the expert network reference vector that best meets the requirements based on the expert network reference vectors corresponding to each expert network in the network layer to be run, combined with the current layer output feature vector of the current running network layer of the hybrid expert model, and then determine the expert network corresponding to the calculated expert network reference vector as the target expert network, as the expert network preloaded for the network layer to be run.

[0044] S130: preload the target expert network of the network layer to be run into the storage of the device running the hybrid expert model to perform a model reasoning process.

[0045] Correspondingly, after the target expert network is predicted for the network layer to be run of the hybrid expert model, the target expert network of the network layer to be run can be preloaded into the device storage running the hybrid expert model to perform the model reasoning process, thereby achieving the effect of hiding the expert network's transfer time in the device storage, saving the expert network's memory usage, improving the hardware storage and bandwidth utilization during the operation of the hybrid expert model, and at the same time improving the hardware storage and bandwidth utilization during the model reasoning process, thereby running larger-scale models.

[0046] The embodiment of the present invention loads target-related parameters such as non-expert network parameters and expert network reference vectors of the hybrid expert model to predict the target expert network of the network layer to be run according to the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run, thereby preloading the target expert network of the network layer to be run into the device storage running the hybrid expert model to perform the model reasoning process. The preloading process of the expert network can be realized, thereby accelerating the reasoning process of the hybrid expert model, solving the problems of large storage and bandwidth pressure caused by the need to load all expert networks during the operation of the existing hybrid expert model, and low model calculation efficiency. The runtime behaviors such as data movement and memory access of the hybrid expert model can be reduced, memory usage can be saved, and the calculation performance and operation speed of the model can be improved, thereby improving the utilization of hardware resources and the execution efficiency of the model.

[0047] Embodiment 2

[0048] Figure 3is a flow chart of a hybrid expert model reasoning acceleration method provided by the second embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, a training process of an expert prediction perception model, updating and calculating an expert network reference vector using an expert prediction perception model, and predicting a target expert network of a network layer to be run are provided. Accordingly, as Figure 3 As shown, the method of this embodiment may include:

[0049] S210, obtaining expert network input vector sample data.

[0050] The expert network input vector sample data may be vector data outputted during data processing by the expert networks of each layer in the hybrid expert model.

[0051] In an embodiment of the present invention, the expert network reference vector of each expert network in the hybrid expert model can be calculated by the expert prediction perception model. Before calculating the expert network reference vector of each expert network in the hybrid expert model by the expert prediction perception model, the expert prediction perception model needs to be trained first. Optionally, an expert prediction perception model can be trained for all the expert networks included in each network layer, and each expert prediction perception model can calculate the expert network reference vector for each expert network of the corresponding network layer. Alternatively, a unified expert prediction perception model can be trained for all the expert networks of all layers, and the expert prediction perception model calculates the expert network reference vector for all the expert networks of all layers. The embodiment of the present invention does not limit the number of expert prediction perception models and the method of calculating the expert network reference vector.

[0052] In an optional embodiment of the present invention, the step of obtaining the expert network input vector sample data may include: obtaining the original model training sample data, and inputting the original model training sample data into the hybrid expert model to obtain the input tensor data of each block layer in the hybrid expert model; and combining the input tensor data of each block layer in the hybrid expert model in sequence to obtain the expert network input vector sample data.

[0053] Among them, the original model training sample data can be sample data input to the hybrid expert model for data processing. Optionally, the original model training sample data can be collected according to the application scenario of the hybrid expert model. Exemplarily, when the hybrid expert model is applied to a data classification scenario, the original model training sample data can be sample data for data classification, such as sample data for image classification, etc., or the original model training sample data can also be detection task data in a detection scenario, target segmentation data in a target segmentation scenario, key point distribution data in a key point detection scenario, and evaluation data in an evaluation scenario, etc. That is, any data type that can be applied to a hybrid expert model for data processing can be used as the original model training sample data, and the embodiment of the present invention does not limit the data type of the original model training sample data.

[0054] In the embodiment of the present invention, the original model training sample data may be sample data used to train the hybrid expert model. Optionally, part or all of the original model training sample data may be used and input into the hybrid expert model. The hybrid expert model processes the received original model training sample data layer by layer, and each block layer may correspond to the obtained input tensor data, such as Figure 2 The Tensor1 data shown. Furthermore, the input tensor data of each block layer in the hybrid expert model can be combined in the order of the block layer to obtain the expert network input vector sample data. That is, the original model training sample data input to the hybrid expert model can be used as a sequence. Each token in the original model training sample data is forwarded by the hybrid expert model, and the tensor output by each block layer can be output, which is then saved as the input history vector. For the first block layer of the hybrid expert model, it only contains one vector, namely the input tensor1 of the first block layer.

[0055] S220: Training an expert prediction perception model according to the expert network input vector sample data.

[0056] The expert prediction perception model is used to update and calculate the expert network benchmark vector. That is, the expert prediction perception model can calculate the initial expert network benchmark vector, and during the operation of the hybrid expert model, it can also continuously collect sample data related to the hybrid expert model to update the expert network benchmark vector according to the collected sample data, thereby continuously improving the accuracy of the expert network benchmark vector, and further improving the accuracy of expert network prediction and preloading.

[0057] Correspondingly, after the original model training sample data is processed by the hybrid expert model to obtain the expert network input vector sample data, the expert network input vector sample data can be input into the expert prediction perception model of the expert network of the corresponding network layer to perform the model training process for the expert prediction perception model.

[0058] Exemplarily, when an expert prediction perception model is trained for all the expert networks included in each network layer, the input tensor data of the first layer in the hybrid expert model can be used as the expert network input vector sample data, input into the expert prediction perception model corresponding to the first layer of the expert network, and the model training is performed on it, and the expert network input vector sample data from the 1st to the Qth layers are input into the expert prediction perception model corresponding to the Nth layer of the expert network, and the model training is performed on it, where Q is greater than 1 and Q is less than or equal to N.

[0059] In an optional embodiment of the present invention, the training of the expert prediction perception model according to the expert network input vector sample data may include: inputting the expert network input vector sample data into the input layer of the expert prediction perception model; performing weighted processing on the expert network input vector sample data through the input layer of the expert prediction perception model to obtain input layer output data; inputting the input layer output data into the MLP (Multilayer Perceptron) layer of the expert prediction perception model to process the input layer output data through the MLP layer and the output projection layer to obtain the expert prediction network output multidimensional vector of the corresponding block layer; sending the expert prediction network output multidimensional vector to the Softmax (normalization function) layer, and outputting the expert network classification probability of the expert prediction network output multidimensional vector through the Softmax layer; determining the expert label of the expert prediction network output multidimensional vector through the Gate (gating function) layer in the hybrid expert model; calculating the loss value of the expert network classification probability of the expert prediction network output multidimensional vector through the loss function, and cyclically executing the training process of the expert prediction perception model according to the loss value until it is determined that the expert prediction perception model training is completed.

[0060] The input layer output data may be data outputted by the input layer of the expert prediction perception model after processing the expert network input vector sample data. The expert prediction network output multidimensional vector may be a multidimensional vector obtained by the expert network corresponding to the input layer output data being analyzed and processed by the MLP layer and the output projection layer of the expert prediction perception model. The expert network classification probability may be the probability of classification prediction of the expert network. The expert label may be used to label the expert network.

[0061] In an embodiment of the present invention, the model structure of the expert prediction perception model may include an input layer, an MLP layer, an output projection layer, and a Softmax layer. Figure 4 It is a flowchart of training an expert prediction perception model provided by Embodiment 2 of the present invention. Figure 5 is a schematic diagram of the structure of the input layer of an expert prediction perception model provided by the second embodiment of the present invention. In a specific example, Figure 4 and Figure 5 As shown, when the expert prediction perception model is trained, the expert network input vector sample data output by the hybrid expert model can be input into the input layer of the expert prediction perception model. Optionally, the input layer of the expert prediction perception model can be a linear layer, and the parameter scale of the input layer can be configured according to the number of block layers in the hybrid expert model. For example, the parameter scale can be 1*(N-1), where N is the number of block layers in the hybrid expert model. Different weights can be assigned to each layer of expert network input vector sample data according to the distance from the block layer of the current hybrid expert model. Figure 5 As shown, the weights of the input layer of the expert prediction perception model can be W1, W2...WN-1, and the expert network input vector sample data tensorN-1 is multiplied by the corresponding weights, and the corresponding elements are added to obtain the input layer output data. The weights of the input layer can be updated through training learning.

[0062] Exemplarily, for the input layer of the expert prediction perception model corresponding to the Nth layer network of the hybrid expert model, the following formula can be used to perform weighted processing on the expert network input vector sample data to obtain the input layer output data Output of the expert prediction perception model corresponding to the Nth layer network of the hybrid expert model: Output=tensor1*W1+tensor2*W2…+tensorN*WN. Among them, tensorN represents the Nth input tensor data in the expert network input vector sample data, that is, the input tensor data of the Nth layer block layer of the hybrid expert model, and WN represents the weight of the Nth layer of the input layer. The above-mentioned method of weighted processing of the expert network input vector sample data through the input layer of the expert prediction perception model can improve the accuracy of the expert prediction perception model, thereby improving the calculation accuracy of the expert network reference vector. Alternatively, for the input layer of the expert prediction perception model corresponding to the Nth layer network of the hybrid expert model, the Nth input tensor data in the expert network input vector sample data can also be directly obtained, and the input layer output data Output corresponding to the Nth layer is calculated in combination with the weight of the Nth layer of the input layer: Output=tensorN*WN. The advantage of this processing is that it can improve the data processing efficiency of the expert prediction perception model, thereby improving the calculation efficiency of the expert network benchmark vector.

[0063] In an embodiment of the present invention, the MLP layer of the expert prediction perception model may include one or more layers of MLP, with a nonlinear activation function superimposed in the middle, for performing nonlinear fitting on the data. The output projection layer of the expert prediction perception model may also be a linear layer. The output projection layer may be combined with the MLP layer to determine the number of output categories. Figure 4 As shown, after the input layer of the expert prediction perception model outputs the input layer output data, the input layer output data can be input into the MLP layer of the expert prediction perception model to process the input layer output data through the MLP layer and the output projection layer to obtain the expert prediction network output multi-dimensional vector corresponding to each expert network of the corresponding block layer. It can be understood that the expert prediction network output multi-dimensional vector is the data obtained by vector parsing of the expert network. Furthermore, the expert prediction network output multi-dimensional vector can be sent to the Softmax layer, thereby outputting the expert network classification probability of the expert prediction network output multi-dimensional vector through the Softmax layer. Among them, the expert network classification probability can reflect the hit probability of each expert network. At the same time, the Gate layer in the hybrid expert model (that is, as shown in Figure 4 The gating function of the MoE shown in the figure) determines the expert label label of the expert prediction network output multidimensional vector to save the selection result of the TopK expert network. The expert label label can be applied in the subsequent calculation of the expert network benchmark vector and in the training of the expert prediction perception model. Correspondingly, after obtaining the expert network classification probability of the expert prediction network output multidimensional vector, the loss value of the expert network classification probability of the expert prediction network output multidimensional vector can be calculated by the loss function, and the training process of the expert prediction perception model is cyclically executed according to the loss value, such as judging whether the loss value reaches a stable value or is less than a set value, or judging whether the number of training times of the expert prediction perception model reaches a set number. If it is determined that the expert prediction perception model has not been trained, it can return to execute the operation of inputting the sample data of the expert network input vector into the input layer of the expert prediction perception model until it is determined that the expert prediction perception model training is completed.

[0064] S230. Determine an initial reference vector of each expert network in the hybrid expert model according to the expert prediction perception model.

[0065] The initial reference vector may be a vector obtained by processing and calculating the expert network input vector sample data through the trained expert prediction perception model. The vector may be a vector output by the MLP layer of the expert prediction perception model, such as Figure 4 shown.

[0066] S240. Calculate an expert network reference vector of each expert network in the hybrid expert model according to the initial reference vector of each expert network.

[0067] In the embodiment of the present invention, after the expert prediction perception model is trained, the expert prediction perception model can be used to calculate the expert network reference vector of each expert network. First, the initial reference vector corresponding to each expert network in the hybrid expert model is determined by the expert prediction perception model. Optionally, Figure 4 As shown, after the expert prediction perception model training is completed, the expert network input vector sample data can be traversed again through the expert prediction perception model, and the expert network input vector sample data can be saved after passing through the initial reference vector output by the MLP layer of the expert prediction perception model. Optionally, each token in the expert network input vector sample data can output a corresponding initial reference vector through the MLP layer of the expert prediction perception model. Therefore, when an expert network input vector sample data includes multiple tokens, since each token can correspond to an initial reference vector output by the MLP layer of the expert prediction perception model, an expert network input vector sample data can output multiple initial reference vectors. Since an expert network can calculate multiple initial reference vectors, and each initial reference vector cannot accurately represent the characteristics of the expert network, the final expert network reference vector of each expert network can be further calculated based on the initial reference vectors of each expert network in the hybrid expert model.

[0068] In an optional embodiment of the present invention, the calculation of the expert network benchmark vector of each expert network according to the initial benchmark vector of each expert network in the hybrid expert model may include: clustering multiple initial benchmark vectors of the current expert network to obtain cluster association information; wherein the cluster association information includes initial cluster centers and cluster labels; performing data cleaning on the initial benchmark vectors according to the cluster labels and expert labels in the cluster association information to obtain preprocessed benchmark vectors; re-clustering the preprocessed benchmark vectors to obtain target cluster centers; and using the target cluster centers as the expert network benchmark vector of the current expert network.

[0069] Among them, the current expert network may be the expert network currently being processed to calculate the expert network benchmark vector. The cluster association information may be the clustering-related information generated after clustering multiple initial benchmark vectors of the expert network. The initial cluster center may be the cluster center obtained after clustering multiple initial benchmark vectors of the current expert network for the first time. The cluster label may be the label of the cluster obtained after clustering multiple initial benchmark vectors of the current expert network for the first time. The preprocessed benchmark vector may be the benchmark vector obtained after data cleaning of the initial benchmark vector. The target cluster center may be the cluster center obtained after re-clustering the preprocessed benchmark vector.

[0070] Specifically, after determining multiple initial reference vectors corresponding to each expert network in the hybrid expert model through the expert prediction perception model, for the current expert network, the multiple initial reference vectors of the current expert network can be first clustered to obtain the initial clustering center and cluster label corresponding to the current expert network. Figure 6 is a schematic diagram of the effect of clustering the reference vectors provided by the second embodiment of the present invention. In a specific example, Figure 6 As shown, for each expert network, multiple initial reference vectors can be clustered, and each expert network can obtain a corresponding initial cluster center and cluster label through clustering. For each expert network, multiple initial reference vectors included therein can be clustered separately, such as using clustering algorithms such as Kmeans (K-means clustering algorithm) for clustering. Furthermore, for a cluster, the cluster label corresponding to the cluster can be compared with the expert label label corresponding to each initial reference vector in the cluster, and the singular value vector that does not conform to the expert label can be removed to achieve data cleaning of the initial reference vector and obtain the preprocessing reference vector. After obtaining the preprocessing reference vector, each target cluster center can be recalculated for the preprocessing reference vector of each expert network, so that the target cluster center can be used as the expert network reference vector of each expert network. Correspondingly, after calculating the target cluster center, the target cluster center can be used as the expert network reference vector of the current expert network.

[0071] S250, running the hybrid expert model, and loading target association parameters of the hybrid expert model; wherein the target association parameters include non-expert network parameters and expert network reference vectors of the hybrid expert model.

[0072] S260, predicting the target expert network of the network layer to be run according to the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run, and preloading the target expert network of the network layer to be run into the device storage running the hybrid expert model to perform the model inference process.

[0073] When the hybrid expert model runs to the current running network layer, the current layer output feature vector executes the Attention module, and can use the predictor to use the contrast learning method to calculate the similarity between the current layer output feature vector and the expert network benchmark vectors of the network layer to be run, and select the expert network corresponding to the expert network benchmark vector with the greatest similarity as the target expert network. Among them, the predictor is also the expert prediction perception model. Exemplarily, the similarity between the current layer output feature vector and the expert network benchmark vectors of the network layer to be run can be calculated by calculating the cosine distance or the Euclidean distance. The embodiment of the present invention does not limit the specific calculation method of the similarity between the current layer output feature vector and the expert network benchmark vectors of the network layer to be run. After predicting the target expert network of the network layer to be run, the target expert network of the network layer to be run can be preloaded into the device storage running the hybrid expert model to perform the model reasoning process.

[0074] S270, determining whether the target expert network of the network layer to be run is a hit expert network, if so, executing S280, otherwise, executing S290.

[0075] S280: Execute expert calculation through the target expert network.

[0076] The hit expert network may be an expert network selected by executing a Gate function during normal operation of the hybrid expert model.

[0077] When the hybrid expert model runs to the network layer to be run, the hybrid expert model can execute the Gate function to determine whether the target expert network is the hit expert network. If it is determined that the target expert network is the hit expert network selected by the Gate function, the expert calculation can be directly performed through the target expert network.

[0078] S290: reselect an updated target expert network from each expert network of the to-be-run network layer, and perform expert calculation through the updated target expert network.

[0079] The updated target expert network may be an expert network reselected by executing a Gate function during normal operation of the hybrid expert model.

[0080] Correspondingly, if it is determined that the target expert network is not the hit expert network selected by the Gate function, the Gate function can be used to reselect and update the target expert network from the expert networks of the network layer to be run, and the expert calculation can be performed through the reselected updated target expert network.

[0081] In an optional embodiment of the present invention, after performing expert calculations through the updated target expert network, it may also include: constructing model optimization sample data based on the expert labels of the updated target expert network and the current layer output feature vector of the currently running network layer; optimizing and updating the expert prediction perception model based on the model optimization sample data.

[0082] Among them, the model optimization sample data can be sample data generated in real time according to the operation process data of the hybrid expert model, and can be used to update and optimize the expert prediction perception model.

[0083] It is understandable that if the target expert network is the hit expert network selected by the Gate function, it indicates that the expert network benchmark vector of the expert network has a high accuracy, and there is no need to update the optimized expert network benchmark vector. If the target expert network is not the hit expert network selected by the Gate function, it indicates that the expert network benchmark vector of the expert network has a low accuracy, and the optimized expert network benchmark vector needs to be updated. Figure 7 is a flow chart of optimizing and updating the expert prediction perception model provided by the second embodiment of the present invention. In a specific example, Figure 7 As shown, the model optimization sample data can be constructed based on the expert label of the updated target expert network and the current layer output feature vector of the current running network layer, so as to update the optimized expert prediction perception model based on the constructed model optimization sample data. Then, the updated expert prediction perception model is used to recalculate the expert network benchmark vectors corresponding to each expert network.

[0084] Figure 8 FIG. 1 is a schematic diagram of the time comparison effect of a baseline solution and a Predict solution provided in the second embodiment of the present invention. In a specific example, Figure 8 As shown in FIG. 1 , the baseline solution is a conventional operation solution of the hybrid expert model, that is, a solution for loading the expert network after executing the Gate function, and the Predict solution is a solution for preloading the expert network for the hybrid expert model provided by the embodiment of the present invention. Compared with the Baseline solution, the Predictor method provided by the embodiment of the present invention can save t s time, in the case of not hitting the expert network, only the time t to judge whether the expert prediction hits is increased v , which can greatly improve the model operation speed of the hybrid expert model as a whole, thereby improving the utilization rate of hardware computing units.

[0085] At present, the existing hybrid expert model reasoning optimization method obtains the expert network sequence number by inferring the model's gated layer and then loads the expert network. By saving storage, the terminal device deployment of the hybrid expert model is realized, which will increase the reasoning time. At the same time, the existing LLM (Large Language Model) reasoning acceleration methods such as Speculative Decoding cannot update the weights of small models, cannot count historical activation features for learning, and especially cannot provide prediction experts for each layer of structure for input, and cannot flexibly realize the pre-loading of each layer of expert network.

[0086] The above technical solution establishes an expert prediction perception model, uses the expert prediction perception model to traverse the expert network input vector sample data, learns the prediction perception network parameters corresponding to each layer of the expert network, and obtains the expert network reference vector of each expert network after multi-stage clustering. During the operation of the hybrid expert model, for each layer of the block of the hybrid expert model, the output feature vector of the block can be compared and matched with the expert network reference vector of each expert network in the next layer, so as to predict the expert network adapted to the next layer in advance, and preload the predicted expert network of the next layer into the device storage, thereby hiding the transfer time of the expert network, realizing the mutual concealment of the expert network loading process and the calculation of the previous layer network, avoiding the waiting of computing resources during the transfer of the expert network, so as to reduce the calculation delay of the hybrid expert model. In addition, the parameters of the expert prediction perception model and the expert network reference vector can be updated according to the accuracy of the prediction result of the expert network. If the prediction result of the expert network is consistent with the result of the expert network calculated by the hybrid expert model inference, there is no need to update the reference vector; if not, new features are learned for the expert prediction perception model and the expert network reference vector is updated. The entire hybrid expert model reasoning acceleration process can be completed through hardware. The expert network benchmark vectors of each layer of the expert network can be stored in the on-chip memory, and the comparison learning function hardware acceleration instructions can be configured to realize the calculation of the similarity of the expert network benchmark vectors, which can achieve rapid prediction. The above technical solution can reduce runtime behaviors such as data movement and memory access, improve the computing performance of deep neural networks, save the memory usage of experts, and improve the computing speed and reasoning efficiency of the hybrid expert model, thereby improving the utilization of hardware resources and the execution efficiency of AI models.

[0087] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0088] It should be noted that in the embodiments of the present invention, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solutions in the embodiments of the present invention, but it does not mean that the applicant has or will necessarily use the solution.

[0089] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0090] Embodiment 3

[0091] Fig. 9 is a schematic diagram of a hybrid expert model reasoning acceleration device provided in the third embodiment of the present invention, such as Fig. 9 As shown, the device includes: a target association parameter loading module 310, a target expert network prediction module 320 and a target expert network preloading module 330, wherein:

[0092] The target-associated parameter loading module 310 is used to load the target-associated parameters of the hybrid expert model; wherein the target-associated parameters include the non-expert network parameters and the expert network reference vector of the hybrid expert model;

[0093] A target expert network prediction module 320 is used to predict a target expert network of the network layer to be run according to the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run;

[0094] The target expert network preloading module 330 is used to preload the target expert network of the network layer to be run into the device storage running the hybrid expert model to perform the model reasoning process.

[0095] The embodiment of the present invention loads target-related parameters such as non-expert network parameters and expert network reference vectors of the hybrid expert model to predict the target expert network of the network layer to be run according to the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run, thereby preloading the target expert network of the network layer to be run into the device storage running the hybrid expert model to perform the model reasoning process, realize the preloading process of the expert network, thereby accelerating the reasoning process of the hybrid expert model, solving the problems of large storage and bandwidth pressure caused by the need to load all expert networks during the operation of the existing hybrid expert model, and low model calculation efficiency. The hybrid expert model can reduce runtime behaviors such as data movement and memory access, save memory usage, and improve the computing performance and operation speed of the model, thereby improving the utilization of hardware resources and the execution efficiency of the model.

[0096] Optionally, the above-mentioned device may also include an expert calculation module, which is used to: when it is determined that the target expert network of the network layer to be run is a hit expert network, perform expert calculation through the target expert network; when it is determined that the target expert network of the network layer to be run is a non-hit expert network, reselect and update the target expert network from the expert networks of the network layer to be run, and perform expert calculation through the updated target expert network.

[0097] Optionally, the above-mentioned device may also include an expert prediction perception model updating module, which is used to: construct model optimization sample data based on the expert label of the update target expert network and the current layer output feature vector of the currently running network layer; optimize and update the expert prediction perception model based on the model optimization sample data; wherein, the expert prediction perception model is used to update and calculate the expert network benchmark vector.

[0098] Optionally, the above-mentioned device may also include an expert prediction perception model training module, which is used to: obtain expert network input vector sample data; and train the expert prediction perception model according to the expert network input vector sample data.

[0099] Optionally, the expert prediction perception model training module is also used to: obtain original model training sample data, and input the original model training sample data into the hybrid expert model to obtain input tensor data of each block layer in the hybrid expert model; combine the input tensor data of each block layer in the hybrid expert model in sequence to obtain the expert network input vector sample data.

[0100] Optionally, the expert prediction perception model training module is also used to: input the expert network input vector sample data into the input layer of the expert prediction perception model; perform weighted processing on the expert network input vector sample data through the input layer of the expert prediction perception model to obtain input layer output data; input the input layer output data into the MLP layer of the expert prediction perception model to process the input layer output data through the MLP layer and the output projection layer to obtain the expert prediction network output multidimensional vector of the corresponding block layer; send the expert prediction network output multidimensional vector to the Softmax layer, and output the expert network classification probability of the expert prediction network output multidimensional vector through the Softmax layer; determine the expert label of the expert prediction network output multidimensional vector through the Gate layer in the hybrid expert model; calculate the loss value of the expert network classification probability of the expert prediction network output multidimensional vector through the loss function, and cyclically execute the training process of the expert prediction perception model according to the loss value until it is determined that the expert prediction perception model training is completed.

[0101] Optionally, the above-mentioned device also includes an expert network benchmark vector calculation module, which is used to determine the initial benchmark vector of each expert network in the hybrid expert model according to the expert prediction perception model; and calculate the expert network benchmark vector of each expert network according to the initial benchmark vector of each expert network in the hybrid expert model.

[0102] Optionally, the expert network benchmark vector calculation module is also used to: cluster multiple initial benchmark vectors of the current expert network to obtain cluster association information; wherein the cluster association information includes initial cluster centers and cluster labels; perform data cleaning on the initial benchmark vectors according to the cluster labels and expert labels in the cluster association information to obtain preprocessed benchmark vectors; re-cluster the preprocessed benchmark vectors to obtain target cluster centers; and use the target cluster centers as the expert network benchmark vectors of the current expert network.

[0103] The hybrid expert model reasoning acceleration device can execute the hybrid expert model reasoning acceleration method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the hybrid expert model reasoning acceleration method provided by any embodiment of the present invention.

[0104] Since the hybrid expert model reasoning acceleration device introduced above is a device that can execute the hybrid expert model reasoning acceleration method in the embodiment of the present invention, based on the hybrid expert model reasoning acceleration method introduced in the embodiment of the present invention, the technical personnel of the field can understand the specific implementation of the hybrid expert model reasoning acceleration device of the present embodiment and its various variations, so how the hybrid expert model reasoning acceleration device implements the hybrid expert model reasoning acceleration method in the embodiment of the present invention is not introduced in detail here. As long as the technical personnel of the field implement the device adopted by the hybrid expert model reasoning acceleration method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0105] Embodiment 4

[0106] Fig.10 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0107] like Fig.10 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0108] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the hybrid expert model reasoning acceleration method.

[0110] Optionally, the hybrid expert model reasoning acceleration method may include: loading target association parameters of the hybrid expert model; wherein the target association parameters include non-expert network parameters and expert network reference vectors of the hybrid expert model; predicting the target expert network of the network layer to be run based on the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run; and preloading the target expert network of the network layer to be run into the device storage running the hybrid expert model to perform the model reasoning process.

[0111] In some embodiments, the hybrid expert model reasoning acceleration method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the hybrid expert model reasoning acceleration method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the hybrid expert model reasoning acceleration method in any other appropriate manner (e.g., by means of firmware).

[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0114] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0117] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A hybrid expert model reasoning acceleration method, characterized in that: include: Loading target-related parameters of the hybrid expert model; wherein the target-related parameters include non-expert network parameters and expert network reference vectors of the hybrid expert model; Predicting a target expert network of the network layer to be run according to a current layer output feature vector of the current running network layer of the hybrid expert model and an expert network reference vector of the network layer to be run; Preloading the target expert network of the network layer to be run into the storage of the device running the hybrid expert model to perform a model reasoning process; The step of predicting a target expert network of the network layer to be run according to the current layer output feature vector of the current running network layer of the hybrid expert model and the expert network reference vector of the network layer to be run comprises: When the hybrid expert model runs to the currently running network layer, obtaining the expert network reference vector corresponding to each expert network in the network layer to be run; Taking the expert network reference vectors corresponding to each of the expert networks in the network layer to be run as a reference, and combining the output feature vector of the current layer, an expert network reference vector that meets the requirements is calculated; The expert network corresponding to the calculated expert network reference vector is determined as the target expert network.

2. The method according to claim 1, characterized in that After preloading the target expert network of the network layer to be run into the device storage for running the hybrid expert model, the method further includes: When it is determined that the target expert network of the network layer to be run is a hit expert network, performing expert calculation through the target expert network; When it is determined that the target expert network of the network layer to be run is a non-hit expert network, an updated target expert network is reselected from each expert network of the network layer to be run, and expert calculation is performed through the updated target expert network.

3. The method according to claim 2, characterized in that After performing expert calculation by updating the target expert network, the method further includes: Constructing model optimization sample data according to the expert label of the updated target expert network and the current layer output feature vector of the current running network layer; Optimizing and updating the expert prediction perception model based on the model optimization sample data; Wherein, the expert prediction perception model is used to update and calculate the expert network benchmark vector.

4. The method according to claim 1, characterized in that Before the target association parameters of the hybrid expert model are loaded, the method further includes: Obtain expert network input vector sample data; The expert prediction perception model is trained according to the expert network input vector sample data.

5. The method according to claim 4, characterized in that The step of obtaining expert network input vector sample data includes: Acquire original model training sample data, and input the original model training sample data into the hybrid expert model to obtain input tensor data of each block layer in the hybrid expert model; The input tensor data of each block layer in the hybrid expert model are combined in sequence to obtain the expert network input vector sample data.

6. The method according to claim 4, characterized in that The step of training the expert prediction perception model according to the expert network input vector sample data includes: Inputting the expert network input vector sample data into the input layer of the expert prediction perception model; Performing weighted processing on the expert network input vector sample data through the input layer of the expert prediction perception model to obtain input layer output data; Inputting the output data of the input layer into the multi-layer perceptron MLP layer of the expert prediction perception model, so as to process the output data of the input layer through the MLP layer and the output projection layer to obtain the expert prediction network output multi-dimensional vector of the corresponding block layer; Sending the expert prediction network output multidimensional vector to the normalization function Softmax layer, and outputting the expert network classification probability of the expert prediction network output multidimensional vector through the Softmax layer; Determining the expert label of the multidimensional vector output by the expert prediction network through the gate function Gate layer in the hybrid expert model; The loss value of the expert network classification probability of the expert prediction network output multidimensional vector is calculated by the loss function, and the training process of the expert prediction perception model is cyclically executed according to the loss value until it is determined that the training of the expert prediction perception model is completed.

7. The method according to any one of claims 4 to 6, characterized in that: After training the expert prediction perception model according to the expert network input vector sample data, the method further includes: Determining an initial reference vector of each expert network in the hybrid expert model according to the expert prediction perception model; The expert network reference vector of each expert network in the hybrid expert model is calculated according to the initial reference vector of each expert network.

8. The method according to claim 7, characterized in that The step of calculating the expert network reference vector of each expert network according to the initial reference vector of each expert network in the hybrid expert model comprises: Clustering multiple initial reference vectors of the current expert network to obtain cluster association information; wherein the cluster association information includes initial cluster centers and cluster labels; Performing data cleaning on the initial reference vector according to the cluster labels and expert labels in the cluster association information to obtain a preprocessed reference vector; Re-clustering the pre-processed reference vectors to obtain target cluster centers; The target cluster center is used as the expert network reference vector of the current expert network.

9. A hybrid expert model reasoning acceleration device, characterized in that: include: A target-associated parameter loading module, used to load target-associated parameters of the hybrid expert model; wherein the target-associated parameters include non-expert network parameters and expert network reference vectors of the hybrid expert model; A target expert network prediction module, used for predicting a target expert network of the network layer to be run according to a current layer output feature vector of the current running network layer of the hybrid expert model and an expert network reference vector of the network layer to be run; A target expert network preloading module, used for preloading the target expert network of the network layer to be run into the storage of the device running the hybrid expert model, so as to perform a model reasoning process; Wherein, the target expert network prediction module is also used for: When the hybrid expert model runs to the currently running network layer, obtaining the expert network reference vector corresponding to each expert network in the network layer to be run; Taking the expert network reference vectors corresponding to each of the expert networks in the network layer to be run as a reference, and combining the output feature vector of the current layer, an expert network reference vector that meets the requirements is calculated; The expert network corresponding to the calculated expert network reference vector is determined as the target expert network.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the hybrid expert model reasoning acceleration method described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the hybrid expert model reasoning acceleration method described in any one of claims 1-8 when executed.

12. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the hybrid expert model reasoning acceleration method described in any one of claims 1-8 is implemented.

Images