Expert model preloading method and device, chip, electronic equipment, storage medium and computer program product

Abstract

This application provides an expert model preloading method, apparatus, chip, electronic device, storage medium, and computer program product, relating to the field of data processing. The method includes: in response to an expert model being invoked, acquiring historical data and model parameters of multiple expert models, wherein the historical data includes at least invocation information and performance information of the expert models, and the multiple expert models are respectively loaded into video memory, RAM, or hard disk; determining the invocation prediction value of the expert model based on the historical data and model parameters of the expert model; sorting the multiple expert models according to the invocation prediction value; loading the first target number of expert models in the sorting result into video memory, loading the second target number of expert models in the sorting result into RAM, and loading the third target number of expert models in the sorting result into hard disk.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to an expert model preloading method, apparatus, chip, electronic device, storage medium, and computer program product. Background Technology

[0002] The Mixture of Experts (MoE) architecture is a typical expert model architecture commonly used in large-scale deep learning models. In MoE, the model is divided into multiple expert models, each processing a portion of the input data. Traditional MoE models typically use gating networks to select a few expert models for inference to reduce computational overhead and improve efficiency. MoE architecture improves computational efficiency and saves resources by only calling a subset of expert models during each inference iteration. Compared to traditional fully connected networks, MoE offers higher computational efficiency and lower resource consumption when processing large-scale data. However, expert models are typically only categorized as active or inactive, i.e., whether or not to load an expert model into GPU memory, and historical data of the expert models is not considered when switching their states. This leads to a waste of memory and disk storage resources, and the loading effect of expert models is also poor. Summary of the Invention

[0003] This application provides an expert model preloading method, apparatus, chip, electronic device, storage medium, and computer program product.

[0004] This application provides an expert model preloading method, the method comprising:

[0005] In response to the invocation of an expert model, historical data and model parameters of multiple expert models are obtained. The historical data includes at least the invocation information and performance information of the expert models. The multiple expert models are loaded into video memory, memory or hard disk respectively.

[0006] The invoked prediction value of the expert model is determined based on the historical data and model parameters of the expert model;

[0007] The multiple expert models are sorted according to the called prediction values;

[0008] The expert model for the first number of objectives in the sorting results is loaded into the video memory, the expert model for the second number of objectives in the sorting results is loaded into the memory, and the expert model for the third number of objectives in the sorting results is loaded into the hard disk.

[0009] The step of determining the invoked prediction value of the expert model based on its historical data and model parameters includes:

[0010] Extract the number of times the expert model was invoked and the invocation time within a preset time period from the invoked information;

[0011] The loading time information and cache hit information of the expert model are extracted from the performance information;

[0012] The predicted value of the expert model is determined based on the number of times the expert model is invoked, the invocation time, the loading time, the cache hit information, and the model parameters within a preset time period.

[0013] The step of determining the invocation prediction value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, and the model parameters within a preset time period includes:

[0014] Obtain the business information and user information of the expert model;

[0015] The first predicted value of the expert model is determined based on the business information and the user information;

[0016] The invocation prediction value of the expert model is determined based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, and the first prediction value within a preset time period.

[0017] The step of determining the first predicted value of the expert model based on the business information and the user information includes:

[0018] Based on the business information, determine the business importance information and business key information of the expert model;

[0019] The user distribution information of the expert model is determined based on the user information, and the user distribution information includes at least the proportion of paying users;

[0020] The first predicted value is determined based on the business importance information, business key information, and user distribution information of the expert model.

[0021] The step of determining the invocation prediction value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, and the model parameters within a preset time period includes:

[0022] Obtain correlation information between the expert model and other expert models;

[0023] The second predicted value of the expert model is determined based on the correlation information and the invoked information;

[0024] The invocation prediction value of the expert model is determined based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, and the second prediction value within a preset time period.

[0025] Alternatively, the invocation prediction value of the expert model can be determined based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, the first prediction value, and the second prediction value within a preset time period.

[0026] The step of determining the second predicted value of the expert model based on the correlation information and the invoked information includes:

[0027] The co-occurrence call count and call sequence of the expert model are determined based on the call information, wherein the co-occurrence call count is the number of times the expert model is called together with any other expert model;

[0028] The similarity between the expert model and the other expert models is determined based on the correlation information;

[0029] The second prediction value of the expert model is determined based on the number of co-occurrence calls, the call sequence, and the similarity between the expert model and the other expert models.

[0030] The step of loading the expert model for the number of top-ranked targets in the sorting results into the video memory includes:

[0031] The first target number of expert models are loaded into multiple expert memory regions in the video memory. The video memory is divided into multiple expert memory regions, and expert models in different expert memory regions can be called simultaneously.

[0032] After loading the expert model of the first number of targets in the sorting results into the video memory, the method further includes:

[0033] The expert model for the fourth objective among the expert models for the first objective quantity is identified as the commonly used expert model;

[0034] The commonly used expert model is copied and loaded into the commonly used expert video memory region, which is divided into multiple expert video memory regions and at least one commonly used expert video memory region.

[0035] Another embodiment of this application provides an expert model preloading device, the device comprising:

[0036] The acquisition module is used to acquire historical data and model parameters of multiple expert models when the expert model is invoked. The historical data includes at least the invocation information and performance information of the expert model. The multiple expert models are loaded into video memory, memory or hard disk respectively.

[0037] The calculation module is used to determine the called prediction value of the expert model based on the historical data and model parameters of the expert model;

[0038] The processing module is used to sort the multiple expert models according to the called prediction values;

[0039] The processing module is further configured to load the expert model of the first number of target items in the sorting result into the video memory, load the expert model of the first second number of target items in the sorting result into the memory, and load the expert model of the first third number of target items in the sorting result into the hard disk.

[0040] The processing module is further configured to extract the number of times the expert model was invoked and the invocation time within a preset time period from the invoked information;

[0041] The processing module is also used to extract the loading time information and cache hit information of the expert model from the performance information;

[0042] The calculation module is also used to determine the predicted value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, and the model parameters within a preset time period.

[0043] The acquisition module is also used to obtain the business information and user information of the expert model;

[0044] The calculation module is further configured to determine the first predicted value of the expert model based on the business information and the user information;

[0045] The calculation module is further configured to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters, and the first prediction value within a preset time period.

[0046] The calculation module is further configured to determine the business importance information and business key information of the expert model based on the business information.

[0047] The calculation module is further configured to determine the user distribution information of the expert model based on the user information, wherein the user distribution information includes at least the proportion of paying users;

[0048] The calculation module is further configured to determine the first predicted value based on the business importance information, business key information, user distribution information, and the first preset weight of the expert model.

[0049] The acquisition module is also used to obtain correlation information between the expert model and other expert models;

[0050] The calculation module is further configured to determine a second predicted value of the expert model based on the correlation information and the invoked information;

[0051] The calculation module is also used to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters and the second prediction value within a preset time period;

[0052] Alternatively, the calculation module is further configured to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters, the first prediction value, and the second prediction value within a preset time period.

[0053] The calculation module is further configured to determine the number of co-occurrence calls and the call sequence of the expert model based on the call information, wherein the number of co-occurrence calls is the number of times the expert model is called together with any other expert model;

[0054] The calculation module is further configured to determine the similarity between the expert model and the other expert models based on the correlation information;

[0055] The calculation module is further configured to determine a second predicted value of the expert model based on the number of co-occurrence calls, the call sequence, and the similarity between the expert model and the other expert models.

[0056] The processing module is further configured to load the first target number of expert models into multiple expert memory regions in the video memory, wherein the video memory is divided into multiple expert memory regions, and expert models in different expert memory regions can be called simultaneously.

[0057] The processing module is further configured to identify the expert model for the fourth target among the expert models for the first target number as a commonly used expert model.

[0058] The processing module is also used to copy the commonly used expert model and load it into the commonly used expert video memory area, wherein the video memory is divided into multiple expert video memory areas and at least one commonly used expert video memory area.

[0059] Another embodiment of this application provides a chip, the chip including a processor, the processor being capable of executing the expert model preloading method.

[0060] Another embodiment of this application provides an electronic device, the electronic device including a chip, the chip including a processor, the processor being capable of executing the expert model preloading method.

[0061] Another embodiment of this application provides a computer-readable storage medium storing a computer program for executing the expert model preloading method described above.

[0062] Another aspect of this application provides a computer program product, including a computer program or instructions, for inducing a processor to execute and implement the expert model preloading method provided in this application.

[0063] In the above solution, after an expert model is invoked, its historical data and model parameters are obtained. Based on this data, the invoked prediction values ​​of the expert models are determined and sorted. Then, the top-ranked expert models (the first target number) are loaded into GPU memory, the second-ranked into RAM, and the third-ranked into disk. This effectively solves the problem of poor expert model loading performance in traditional MoE architectures, where expert models are only divided into active and inactive states, and historical data is not considered during switching, leading to wasted memory and disk storage resources. By combining the invoked and performance information of expert models from historical data to determine the invoked prediction values, the allocation of storage resources is more aligned with the actual invocation needs of the expert models. GPU memory access is the fastest, and data transmission latency is the lowest. Therefore, it is used to load expert models with high invoked prediction values, ensuring fast response in high-frequency invocation scenarios and avoiding unnecessary occupation of GPU memory resources. RAM and disk respectively host expert models with medium and low invoked prediction values, making full use of the characteristics of different storage media and improving the utilization rate of storage resources such as memory and disk. Meanwhile, based on the hierarchical loading method according to the sorting results and the number of targets, the distribution of expert models across different storage media was optimized, reducing latency and resource consumption during model loading, thereby improving the overall loading effect of expert models.

[0064] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0065] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0066] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0067] Figure 1 A flowchart of an expert model preloading method according to an embodiment of this application is shown;

[0068] Figure 2 A schematic diagram of the structure of a storage device of a system according to an embodiment of this application is shown;

[0069] Figure 3 A flowchart of an expert model preloading method according to another embodiment of this application is shown;

[0070] Figure 4 A flowchart of an expert model preloading method according to another embodiment of this application is shown;

[0071] Figure 5 A flowchart of an expert model preloading method according to another embodiment of this application is shown;

[0072] Figure 6 A flowchart of an expert model preloading method according to another embodiment of this application is shown;

[0073] Figure 7 A flowchart of an expert model preloading method according to another embodiment of this application is shown;

[0074] Figure 8 A flowchart of an expert model preloading method according to another embodiment of this application is shown;

[0075] Figure 9 A schematic diagram of the system architecture according to an embodiment of this application is shown;

[0076] Figure 10 A schematic diagram of the structure of an expert model preloading device according to an embodiment of this application is shown;

[0077] Figure 11 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0078] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] To improve the utilization of storage resources such as memory and hard disk, and to enhance the loading efficiency of expert models, one embodiment of this application provides an expert model preloading method, such as... Figure 1 As shown, the method includes:

[0080] Step 101: In response to the invocation of an expert model, obtain historical data and model parameters of multiple expert models. The historical data includes at least the invocation information and performance information of the expert models. The multiple expert models are loaded into video memory, memory or hard disk respectively.

[0081] In this embodiment, the expert model is invoked in the following ways: when a user initiates a business request through the client, such as intelligent question answering, data analysis, or image recognition, the corresponding expert model is invoked; when the system executes a pre-set periodic batch processing task, such as batch data verification or historical result review, the relevant expert model is invoked; after an expert model completes its own reasoning steps, it invokes subsequent expert models based on task dependencies. In other embodiments, it may also include cases where any other arbitrary expert model is invoked.

[0082] The expert model's invocation information includes at least the actual number of times the expert model was invoked in the past 1 hour, 24 hours, and 7 days, the timestamp of the last invocation, the hourly invocation distribution in the past 24 hours, the invocation count in the most recent 1 hour and the previous 5 hours, the number of times it co-occurred (i.e. was jointly invoked) with other expert models in the same request, and the invocation sequence.

[0083] The performance information of the expert model includes at least the average time and loading speed of the expert model from the storage medium to the video memory, as well as the number of hits, misses, average hit access time and average miss access time in the cache system.

[0084] The model parameters of an expert model must include at least the actual storage size of the expert model file.

[0085] In the initial stage, expert models can be allocated according to importance or randomly, and loaded into video memory, main memory, or hard disk respectively.

[0086] When expert models are loaded into video memory, their high bandwidth allows for the fastest access speed and lowest data transfer latency, and they can be directly invoked by the GPU for inference without additional intermediary operations. However, due to the cost and physical capacity of video memory, the number of expert models that can be loaded into video memory is limited. Therefore, other expert models can be loaded into RAM or hard disk to improve the utilization of other storage resources.

[0087] When expert models are loaded into memory, their access speed and cost fall between those of video memory and hard disk, with loading latency higher than video memory but lower than hard disk. When loaded into hard disk, access speed is the slowest and loading latency is the highest, but its cost is lower, making it a superior storage resource.

[0088] Step 102: Determine the called prediction value of the expert model based on the historical data and model parameters of the expert model.

[0089] The predicted value of the expert model can be determined by using historical data and model parameters of the expert model, through methods such as entropy weighting, preset weights, or pre-trained time series models.

[0090] Step 103: Sort the multiple expert models according to the called prediction values.

[0091] The numerical range of the called prediction value is within [0, 1], and all expert models are sorted according to the called prediction value.

[0092] Step 104: Load the expert model of the first number of targets in the sorting results into the video memory, load the expert model of the second number of targets in the sorting results into the memory, and load the expert model of the third number of targets in the sorting results into the hard disk.

[0093] The first target number is the maximum number of expert models that can be loaded into video memory. This can be a preset value or determined based on the maximum video memory and the file size of the expert models ranked higher in the sorting results. Similarly, the second target number is the maximum number of expert models that can be loaded into memory. This can be a preset value or determined based on the maximum memory and the file size of the expert models ranked after the ones loaded into video memory in the sorting results. The third target number is the maximum number of expert models that can be loaded into hard disk. This can be a preset value or determined based on the maximum hard disk storage space and the file size of the expert models ranked after the ones loaded into memory in the sorting results.

[0094] For example, such as Figure 2As shown, a system deploys 10 expert models. Based on historical data and model parameters, the predicted values ​​of these 10 expert models are determined and sorted. The resulting ranking is: Expert Model 1: 0.95 (predicted value), Expert Model 2: 0.91, Expert Model 3: 0.87, Expert Model 4: 0.83, Expert Model 5: 0.76, Expert Model 6: 0.71, Expert Model 7: 0.64, Expert Model 8: 0.59, Expert Model 9: 0.48, Expert Model 10: 0.35. The number of first objectives is determined to be 2, and the number of second and third objectives are both 4. Therefore, Expert Model 1 and Expert Model 2 are loaded into video memory, Expert Model 3, Expert Model 4, Expert Model 5, and Expert Model 6 are loaded into system memory, and Expert Model 7, Expert Model 8, Expert Model 9, and Expert Model 10 are loaded into the hard disk.

[0095] In the above solution, after an expert model is invoked, its historical data and model parameters are obtained. Based on this data, the invoked prediction values ​​of the expert models are determined and sorted. Then, the top-ranked expert models (the first target number) are loaded into GPU memory, the second-ranked into RAM, and the third-ranked into disk. This effectively solves the problem of poor expert model loading performance in traditional MoE architectures, where expert models are only divided into active and inactive states, and historical data is not considered during switching, leading to wasted memory and disk storage resources. By combining the invoked and performance information of expert models from historical data to determine the invoked prediction values, the allocation of storage resources is more aligned with the actual invocation needs of the expert models. GPU memory access is the fastest, and data transmission latency is the lowest. Therefore, it is used to load expert models with high invoked prediction values, ensuring fast response in high-frequency invocation scenarios and avoiding unnecessary occupation of GPU memory resources. RAM and disk respectively host expert models with medium and low invoked prediction values, making full use of the characteristics of different storage media and improving the utilization rate of storage resources such as memory and disk. Meanwhile, based on the hierarchical loading method according to the sorting results and the number of targets, the distribution of expert models across different storage media was optimized, reducing latency and resource consumption during model loading, thereby improving the overall loading effect of expert models.

[0096] This application also provides an example of an expert model preloading method, such as... Figure 3 As shown, determining the invoked prediction value of the expert model based on its historical data and model parameters includes:

[0097] Step 201: Extract the number of times the expert model was invoked and the invocation time within a preset time period from the invoked information.

[0098] Extract the number of times the expert model was invoked within a preset time period from the invoked information, including the actual number of times the expert model was invoked in the past 1 hour, 24 hours, and 7 days. Extract the invoke time of the expert model, including the timestamp of the last invoked expert model, the hourly invocation distribution in the past 24 hours, and the invocation count in the most recent 1 hour and the previous 5 hours.

[0099] Step 202: Extract the loading time information and cache hit information of the expert model from the performance information.

[0100] The loading time information of the expert model is extracted from the performance information, including the average time for the expert model to be loaded from the storage medium to the video memory, as well as the number of hits, misses, average hit access time and average miss access time of the expert model in the cache system.

[0101] Step 203: Determine the predicted value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, and the model parameters within the preset time period.

[0102] The model parameters include the model size of the expert model.

[0103] Based on the above data and the actual storage size of the expert model file in the model parameters, the predicted value of the expert model to be called is determined by means of entropy weighting, preset weights, or pre-trained time series models.

[0104] In this embodiment, the invoked prediction value of the expert model can also be determined using the following formula:

[0105] First, determine the predicted value of the expert model in the dimension of call count using the following formula and the actual number of times the expert model has been called in the past 1 hour, 24 hours, and 7 days. :

[0106]

[0107] in, The weights correspond to the actual number of times the expert model was invoked in the past hour. The weights correspond to the actual number of times the expert model was invoked in the past 24 hours. The weights correspond to the actual number of times the expert model was invoked in the past 7 days. This represents the actual number of times the expert model was invoked in the past hour. This represents the actual number of times all expert models were invoked in the past hour. This represents the actual number of times the expert model was invoked in the past 24 hours. This represents the actual number of times all expert models were invoked in the past 24 hours. This represents the actual number of times the expert model was invoked in the past 7 days. This represents the actual number of times all expert models have been invoked in the past 7 days. , , The weights can be preset. Since the prediction of the probability of being called is highly real-time, the weights of more recent data can be set higher.

[0108] The predicted value of the expert model in the time dimension of the invocation is determined by the following formula, the timestamp of the last invocation of the expert model, the hourly invocation distribution (i.e., mean and standard deviation) over the past 24 hours, the invocation counts of the most recent hour and the previous 5 hours, and the model size of the expert model. :

[0109]

[0110] in, For the most recent score, To score based on regularity, For sudden scoring, The weights corresponding to the recentity scores. The weights corresponding to the regularity scores. The weights corresponding to sudden scores. , , All of these are preset weights.

[0111] The recentity score is determined using the following formulas. Regularity score and sudden score :

[0112]

[0113] in, This is the time difference between the last time the expert model was invoked and the current time.

[0114]

[0115] in, This represents the standard deviation of the number of calls to the expert model per hour over the past 24 hours. This represents the average number of calls made per hour over the past 24 hours to the expert model.

[0116]

[0117] in, This represents the actual number of times the expert model was invoked in the past hour. This represents the actual number of times the expert model was invoked in the past 5 hours.

[0118] The following formulas and the average time to load from storage medium to video memory, along with the expert model's hit count, miss count, average hit access time, and average miss access time in the cache system, determine the expert model's performance predictions. :

[0119]

[0120] in, Score the model size. The loading efficiency score is given. For cache hit score, The weights corresponding to the model size score. The weights corresponding to the loading efficiency score. The weight corresponding to the cache hit score. , , All of these are preset weights.

[0121] The model size score is determined using the following formulas. Loading efficiency score Cache hit score :

[0122]

[0123] in, This refers to the model size of the expert model.

[0124]

[0125] in, This refers to the loading speed of the expert model.

[0126]

[0127] in, This represents the average time it takes for the expert model to be loaded from the storage medium to the video memory.

[0128]

[0129] in, This represents the number of hits by the expert model in the cache system. This represents the total number of accesses to the expert model in the caching system (i.e., the sum of hits and misses). This represents the average hit access time of the expert model in the caching system. This represents the average time-miss for expert models in the caching system.

[0130] Finally, the predicted value of the expert model is determined according to the following formula. :

[0131]

[0132] in, This represents the predicted value of the expert model in terms of the number of times it is invoked. This represents the predicted value of the expert model at the time of invocation. This represents the predicted value of the expert model in the performance dimension. These are the weights corresponding to the predicted values ​​of the expert model in terms of the number of times it is invoked. These are the weights corresponding to the predicted values ​​of the expert model at the time dimension of the call. These are the weights corresponding to the expert model's predicted values ​​in the performance dimension. , , All of these are preset weights.

[0133] In the above solution, the number of calls and call times within a preset time period are extracted from the expert model's call information, and loading time and cache hit information are extracted from performance information. Combined with the actual storage size of the expert model file in the model parameters, the predicted values ​​to be called are determined using methods such as entropy weighting, preset weights, or pre-trained time-series models, and then sorted and allocated to different storage media. This makes the determination of the predicted values ​​to be called more comprehensive and accurate. It better reflects the actual calling situation and performance characteristics of the expert model, thus making the allocation of storage resources more reasonable. This fully leverages the advantages of different storage media, reduces storage resource waste, further optimizes the loading effect of the expert model, and improves the response efficiency of model calls.

[0134] This application also provides an example of an expert model preloading method, such as... Figure 4 As shown, determining the invocation prediction value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, and the model parameters within a preset time period includes:

[0135] Step 301: Obtain the business information and user information of the expert model.

[0136] Business information includes business importance information and business key information from expert models.

[0137] User information includes the percentage of paying users for expert models.

[0138] Step 302: Determine the first predicted value of the expert model based on the business information and the user information.

[0139] Based on business and user information, the first prediction value of the expert model can be determined through methods such as entropy weighting, preset weights, or pre-trained time series models. .

[0140] Step 303: Determine the predicted value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, and the first predicted value within a preset time period.

[0141] Specifically, the predicted value of the expert model can be determined using the following formula. :

[0142]

[0143] in, This represents the predicted value of the expert model in terms of the number of times it is invoked. This represents the predicted value of the expert model at the time of invocation. This represents the predicted value of the expert model in the performance dimension. This is the first predicted value from the expert model. These are the weights corresponding to the predicted values ​​of the expert model in terms of the number of times it is invoked. These are the weights corresponding to the predicted values ​​of the expert model at the time dimension of the call. These are the weights corresponding to the expert model's predicted values ​​in the performance dimension. The weights are the values ​​corresponding to the first predicted value of the expert model. , , , All of these are preset weights.

[0144] In the above solution, business information and user information dimensions are added to the existing dimensions of acquiring expert model call information, performance information, and model parameters. First, the primary predicted value reflecting business importance and user value is determined using business and user information. Then, combined with predicted values ​​from the dimensions of call count, call time, and performance, the final predicted call value is determined and allocated to different storage media. This results in a wider range of dimensions in the predicted call value, not only aligning with the call patterns and performance characteristics of the expert model but also fully considering the priority of business needs and differences in user value. This makes the allocation of storage resources more targeted, avoiding waste of storage resources for core business and high-value user-related models, while ensuring that expert models of different business types receive appropriate storage support. Ultimately, this more effectively improves storage resource utilization and the loading response of expert models.

[0145] This application also provides an example of an expert model preloading method, such as... Figure 5 As shown, determining the first predicted value of the expert model based on the business information and the user information includes:

[0146] Step 401: Determine the business importance information and business key information of the expert model based on the business information.

[0147] Business importance information is the SLA (Service Level Agreement) of the expert model, and business criticality information is the business type of the expert model.

[0148] Step 402: Determine the user distribution information of the expert model based on the user information, wherein the user distribution information includes at least the proportion of paying users.

[0149] The proportion of paying users can be determined based on user distribution information, or by dividing the number of calls made by paying users by the total number of calls.

[0150] Step 403: Determine the first predicted value based on the business importance information, business key information, and user distribution information of the expert model.

[0151] Based on the business importance information, business key information, and user distribution information of the expert model, the first prediction value of the expert model can be determined by methods such as entropy weighting, preset weights, or pre-trained time series models.

[0152] In this embodiment, the first predicted value of the expert model can be determined using the following formula. :

[0153]

[0154] in, For service level score, Score users based on their priority. For business criticality scores, The weights corresponding to the service level score, The weights corresponding to the user priority scores. The weights corresponding to the business criticality score.

[0155] The service level score is determined using the following formulas. User priority score Business Keyness Score :

[0156]

[0157] in, Availability corresponding to the SLA service level.

[0158]

[0159] in, This represents the percentage of paying users.

[0160]

[0161] in, This refers to the business type of the expert model.

[0162] In the above solution, the business importance and criticality information of the expert model are identified from business information, and user distribution information is determined from user information. Then, a first predicted value is determined by combining the first preset weight. Finally, the predicted values ​​of the call count, call time, and performance dimensions are weighted to obtain the call prediction value, which is then sorted and allocated to different storage media. This makes the calculation of the first predicted value more targeted and accurate, and can more clearly reflect the differences between business importance, business criticality, and user value. This allows the allocation of storage resources to not only conform to the model's own calling patterns and performance characteristics, but also accurately match business needs and user value priorities, effectively reducing the ineffective occupation of various storage resources and further improving the loading efficiency and response effect of the expert model.

[0163] This application also provides an example of an expert model preloading method, such as... Figure 6 As shown, determining the invocation prediction value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, and the first prediction value within a preset time period includes:

[0164] Step 501: Obtain the correlation information between the expert model and other expert models.

[0165] Relevance information includes data such as the expert model's function labels and task types.

[0166] Step 502: Determine the second predicted value of the expert model based on the correlation information and the invoked information.

[0167] Based on relevance information and the information being invoked, the second predicted value of the expert model can be determined using methods such as entropy weighting, preset weights, or pre-trained time-series models. .

[0168] Step 503: Determine the predicted value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, and the second predicted value within a preset time period.

[0169] Specifically, the predicted value of the expert model can be determined using the following formula. :

[0170]

[0171] in, This represents the predicted value of the expert model in terms of the number of times it is invoked. This represents the predicted value of the expert model at the time of invocation. This represents the predicted value of the expert model in the performance dimension. This is the second predicted value from the expert model. These are the weights corresponding to the predicted values ​​of the expert model in terms of the number of times it is invoked. These are the weights corresponding to the predicted values ​​of the expert model at the time dimension of the call. These are the weights corresponding to the expert model's predicted values ​​in the performance dimension. The weights are the weights corresponding to the second predicted value of the expert model. , , , All of these are preset weights.

[0172] Step 504, or, determine the called prediction value of the expert model based on the number of times the expert model is called within a preset time period, the call time, the loading time information, the cache hit information, the model parameters, the first prediction value, and the second prediction value.

[0173] Specifically, the predicted value of the expert model can be determined using the following formula. :

[0174]

[0175] in, This represents the predicted value of the expert model in terms of the number of times it is invoked. This represents the predicted value of the expert model at the time of invocation. This represents the predicted value of the expert model in the performance dimension. This is the first predicted value from the expert model. This is the second predicted value from the expert model. These are the weights corresponding to the predicted values ​​of the expert model in terms of the number of times it is invoked. These are the weights corresponding to the predicted values ​​of the expert model at the time dimension of the call. These are the weights corresponding to the expert model's predicted values ​​in the performance dimension. The weights corresponding to the first predicted value of the expert model. The weights are the weights corresponding to the second predicted value of the expert model. , , , , All of these are preset weights.

[0176] In the above solution, the second predicted value is determined by further increasing the correlation information dimension between the expert model and other expert models. Subsequently, the predicted value of the called model can be determined by combining the predicted values ​​of the call count dimension, call time dimension, and performance dimension with the second predicted value, or the first predicted value can be added to determine the predicted value of the called model. This makes the consideration of the predicted value of the called model not only cover the calling pattern, performance characteristics, and business and user value of the model itself, but also fully take into account the correlation with other expert models. This makes the prediction results more comprehensive and accurate, and the allocation of storage resources can better adapt to the linkage call requirements between models. It effectively reduces the problem of untimely loading or resource waste caused by ignoring the correlation between models, and further improves the utilization rate of storage resources and the loading response efficiency of expert models.

[0177] This application also provides an example of an expert model preloading method, such as... Figure 7 As shown, determining the second predicted value of the expert model based on the correlation information and the invoked information includes:

[0178] Step 601: Determine the co-occurrence call count and call sequence of the expert model based on the call information, wherein the co-occurrence call count is the number of times the expert model is called together with any other expert model.

[0179] The call information is analyzed to obtain the number of times the expert model appears in the same request as other expert models, thus determining the co-occurrence call count. The call sequence of each expert model is then extracted from the call information, i.e., all call sequences containing that expert model.

[0180] Step 602: Determine the similarity between the expert model and the other expert models based on the correlation information.

[0181] Based on data such as functional labels and task types in the relevance information, the similarity between expert models and other expert models is determined by methods such as cosine similarity, Euclidean distance, or Pearson correlation coefficient.

[0182] Step 603: Determine the second prediction value of the expert model based on the number of co-occurrence calls, the call sequence, and the similarity between the expert model and the other expert models.

[0183] Based on the number of co-occurrence calls of the expert model, the call sequence, and the similarity between the expert model and other expert models, the second prediction value of the expert model can be determined by methods such as entropy weighting, preset weights, or pre-trained time series models.

[0184] In this embodiment, the second predicted value of the expert model can be determined using the following formula. :

[0185]

[0186] in, To score the intensity of co-occurrence calls, Assign a score to the importance of sequence position. The score is based on the degree of functional relevance. The weights corresponding to the co-occurrence call intensity scores, The weights corresponding to the importance scores of sequence positions. The weights corresponding to the functional relevance scores.

[0187] The co-occurrence call intensity score is determined using the following formulas. Sequence position importance score Functional relevance score :

[0188]

[0189] in, This refers to the number of other expert models that appeared together with the expert model in the same request. For the first The number of times each other expert model is invoked in conjunction with the expert model. For the first The predicted values ​​of other expert models in the dimension of the number of times they are invoked. This is the sum of the number of times the expert model appears together with all other expert models in the same request.

[0190]

[0191] in, This represents the number of sequences invoked by the expert model. For expert models in the first The position in the called sequence For the first The total length of the sequence of each called sequence.

[0192]

[0193] in, To compare with the number of all other expert models, For expert models and the first Similarity to other expert models.

[0194] It should be noted that, in this embodiment, other expert models with similar functions can be identified first by using the overlap between the expert model's functional labels and task types. Then, the intersection and union of the functional labels of the expert model and each of the other expert models are determined. The number of labels in the intersection is divided by the number of labels in the union to obtain the similarity between the expert model and each of the other expert models. Other expert models with similarity scores lower than a preset similarity score (e.g., 0.3) are removed, and the similarity scores are then substituted into the above formula to determine the functional relevance score of the expert model.

[0195] In the above scheme, the number of co-occurrence calls and the call sequence are first determined based on the call information. Then, the similarity to other expert models is determined using the corresponding similarity method, combining data such as function tags and task types from the relevance information. Next, a second predicted value is obtained by weighting the co-occurrence call intensity score, sequence position importance score, and functional relevance score with corresponding weights. Finally, the called predicted value can be calculated by combining the original predicted values ​​from each dimension or by further incorporating the first predicted value. This makes the calculation of the second predicted value more logical and accurate, more comprehensively reflecting the correlation characteristics between the expert model and other models. It makes the consideration dimensions of the called predicted value more complete, and the allocation of storage resources not only aligns with the model's own calling patterns, performance characteristics, and business user value, but also accurately adapts to the co-occurrence calls, sequence calls, and functional relevance requirements between models. This effectively reduces resource waste or loading delays caused by insufficient consideration of model relevance, further improving storage resource utilization and the loading response effect of the expert model.

[0196] In one example of this application, an expert model preloading method is also provided, wherein loading the expert models of the first number of target items in the ranking results into the video memory includes:

[0197] The first target number of expert models are loaded into multiple expert memory regions in the video memory. The video memory is divided into multiple expert memory regions, and expert models in different expert memory regions can be called simultaneously.

[0198] For example, as shown in Table 1, a certain graphics card has 96GB of video memory, which is divided into 32 expert video memory regions, each with 3GB of video memory. Among the 32 expert video memory regions, one expert video memory region loads one shared expert model, while the other 31 expert video memory regions each load one professional expert model.

[0199] Table 1

[0200]

[0201] Expert models in different expert memory regions can be accessed simultaneously.

[0202] In the above scheme, by dividing the video memory into multiple expert video memory regions, the expert models with the highest number of targets in the sorting results are loaded into the expert video memory regions respectively, and the expert models in different expert video memory regions can be called simultaneously. This not only makes the allocation of video memory resources more refined and reasonable, fully adapting to the storage needs of high-priority expert models, but also supports the parallel calling of multiple high-priority models, avoiding loading delays or calling conflicts caused by video memory resource contention, further improving the calling response speed of high-priority expert models, and optimizing the utilization efficiency of video memory, significantly improving the utilization rate of storage resources and the loading effect of expert models.

[0203] This application also provides an example of an expert model preloading method, such as... Figure 8 As shown, after loading the expert model of the number of the first target items in the sorting results into the video memory, the method further includes:

[0204] Step 701: Determine the expert model for the fourth target among the expert models for the first target as the commonly used expert model.

[0205] For example, if the top 10 expert models in the ranking results need to be loaded into video memory, then the top 4 of these top 10 expert models are identified as commonly used expert models.

[0206] Step 702: Copy the commonly used expert model and load it into the commonly used expert video memory area, wherein the video memory is divided into multiple expert video memory areas and at least one commonly used expert video memory area.

[0207] The video memory is divided into multiple expert video memory regions and at least one commonly used expert video memory region.

[0208] For example, as shown in Table 1, a certain graphics card has 96GB of video memory, which is divided into 32 expert video memory regions, each with 3GB of video memory. Four of these 32 expert video memory regions are designated as commonly used expert video memory regions. Then, the four commonly used expert models are loaded into these four commonly used expert video memory regions respectively.

[0209] In the above solution, by dividing the video memory into multiple expert video memory regions and at least one frequently used expert video memory region, and determining the top fourth-ranked number of frequently used expert models from the first-ranked number of expert models loaded into video memory, these models are copied and loaded into the frequently used expert video memory region, increasing the number of frequently used expert models deployed. This allows frequently used expert models that are called frequently in high-frequency scenarios to respond quickly directly from the dedicated video memory region, avoiding resource contention and call waiting in high-frequency call scenarios, further improving the call response efficiency of frequently used expert models. At the same time, by dividing the dedicated region and deploying multiple models, the allocation of video memory resources is more in line with the needs of high-frequency use, which not only optimizes the utilization of video memory resources, but also better ensures the loading and calling effect of expert models in high-frequency scenarios.

[0210] This application also provides an example of an expert model preloading method, including:

[0211] Horizontal scaling is achieved using Nginx (a reverse proxy web server).

[0212] To ensure the scalability of the expert model inference architecture, horizontal scaling can be achieved by deploying multiple identical inference nodes when the system faces performance bottlenecks. Utilizing Nginx reverse proxy, the system can intelligently distribute external requests across multiple backend nodes. Nginx employs load balancing strategies (such as round-robin and least connections) to ensure even distribution of requests across different server nodes, avoiding overload on a single node. As request volume increases, new inference nodes can be dynamically added, and Nginx automatically load balances traffic to the new nodes, thereby improving overall processing capacity. Nginx's reverse proxy forwards requests from the client to different backend inference nodes by configuring virtual hosts and proxy rules, and supports features such as caching and SSL encryption, further enhancing system performance and security. Simultaneously, Nginx's health check mechanism periodically monitors the status of backend nodes, removing faulty nodes to ensure that requests are only distributed to healthy server nodes.

[0213] Large-scale model inference tasks can also be handled using OLLamaServer (an inference service for deploying and running large language models in local or private environments), combined with batch processing mechanisms to improve system performance. Specifically, OLLamaServer can receive requests in batches under high concurrency, combine multiple requests into batches, send them to the lower-level inference engine for parallel processing, and finally return the inference results to the user in batches. This mechanism can significantly improve inference efficiency and reduce the processing time for each request.

[0214] When the number of requests exceeds the system's processing capacity, OLLamaServer implements a request queuing mechanism. New requests are added to the queue, waiting for the system to have sufficient resources to process them. The system calculates the estimated waiting time based on the number of requests currently in the queue and the processing capacity of each batch, and then sends this information back to the user. The user can then decide whether to continue waiting or choose another processing method based on the waiting time indication.

[0215] Furthermore, to handle load fluctuations in high-concurrency scenarios, the system can be used in conjunction with Nginx's load balancing mechanism. Through Nginx's reverse proxy configuration, the system can dynamically add service nodes during peak periods. When it detects excessively long request queuing times or high system load, Nginx automatically distributes traffic to newly added OLLamaServer instances, thereby alleviating pressure and improving processing capacity. This dynamic scaling mechanism ensures stable system operation under high concurrency. Combining Nginx reverse proxy and load balancing strategies, the system can dynamically add new OLLamaServer instances during peak load periods, automatically allocating traffic to ensure high availability and high throughput during high-concurrency periods.

[0216] Furthermore, the AsyncLLMEngine (a core component of vLLM) provided by vLLM (a high-performance inference engine for large language models) is used to replace the inference engine provided by OLLama.

[0217] Although Ollam provides an asynchronous model inference engine, its inference speed is slower than vLLM, often failing to fully utilize the computing power of the GPU. Therefore, we decided to use AsyncLLMEngine provided by vLLM to replace the Ollam inference engine.

[0218] AsyncLLMEngine enables parallel processing of CPU and GPU, fully leveraging the synergy between CPU and GPU resources. After processing the first request, the GPU can immediately process the next request without waiting for the CPU to complete data retrieval for the first request. Furthermore, AsyncLLMEngine has an efficient task queue management mechanism that ensures each inference request is processed promptly when the system load is low, and dynamically adjusts task priorities and resource allocation to avoid bottlenecks when the load is high. Through this mechanism, the system maintains efficient resource utilization even in large-scale request environments.

[0219] For example, such as Figure 9 As shown, Figure 9The system shown can distribute requests across multiple OllamaServer nodes using the Nginx complex load balancer. On one hand, batch requests are distributed to nodes such as OllamaServer1 and OllamaServer2, which then sequentially invoke the GPU cluster via the AsyncLLMEngine asynchronous inference engine to perform large language model inference. On the other hand, it supports horizontal scaling of multiple OllamaServer nodes, thereby improving the overall processing capacity and scalability of the system.

[0220] Furthermore, a higher-performance PCIe (bus) standard (such as PCIe 5.0) will be adopted.

[0221] During testing, the PCIe connection between the CPU and GPU operated at full capacity throughout data transfer. This made data exchange between the CPU and GPU a bottleneck for overall system performance. As the primary communication channel between the CPU and GPU, the bandwidth limitations of PCIe affect the speed of data transmission. When the model requires frequent data exchange, the PCIe bandwidth may be insufficient to support high throughput demands, resulting in the GPU's computing power not being fully utilized. Therefore, adopting a higher-performance PCIe standard can significantly improve the data transfer rate between the CPU and GPU, avoiding GPU performance limitations caused by insufficient PCIe bandwidth.

[0222] Furthermore, it adopts DDR5 high-bandwidth memory.

[0223] By increasing memory bandwidth, data can be transferred from host memory to the GPU more quickly, reducing latency caused by memory bottlenecks.

[0224] To implement the aforementioned expert model preloading method, such as Figure 10 As shown, an example of this application provides an expert model preloading device, comprising:

[0225] The acquisition module 801 is used to acquire historical data and model parameters of multiple expert models when an expert model is invoked. The historical data includes at least the invocation information and performance information of the expert models. The multiple expert models are loaded into video memory, memory or hard disk respectively.

[0226] The calculation module 802 is used to determine the called prediction value of the expert model based on the historical data and model parameters of the expert model;

[0227] Processing module 803 is used to sort the multiple expert models according to the called prediction values;

[0228] The processing module 803 is further configured to load the expert model of the first number of target items in the sorting result into the video memory, load the expert model of the first second number of target items in the sorting result into the memory, and load the expert model of the first third number of target items in the sorting result into the hard disk.

[0229] The processing module 803 is further configured to extract the number of times the expert model was invoked and the invocation time within a preset time period from the invoked information;

[0230] The processing module 803 is further configured to extract the loading time information and cache hit information of the expert model from the performance information;

[0231] The calculation module 802 is further configured to determine the predicted value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, and the model parameters within a preset time period.

[0232] The acquisition module 801 is also used to obtain the business information and user information of the expert model;

[0233] The calculation module 802 is further configured to determine the first predicted value of the expert model based on the business information and the user information;

[0234] The calculation module 802 is further configured to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters, and the first prediction value within a preset time period.

[0235] The calculation module 802 is further configured to determine the business importance information and business key information of the expert model based on the business information.

[0236] The calculation module 802 is further configured to determine the user distribution information of the expert model based on the user information, wherein the user distribution information includes at least the proportion of paying users;

[0237] The calculation module 802 is further configured to determine the first predicted value based on the business importance information, business key information, user distribution information, and first preset weight of the expert model.

[0238] The acquisition module 801 is also used to obtain correlation information between the expert model and other expert models;

[0239] The calculation module 802 is further configured to determine a second predicted value of the expert model based on the correlation information and the invoked information;

[0240] The calculation module 802 is further configured to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters and the second prediction value within a preset time period;

[0241] Alternatively, the calculation module 802 is further configured to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters, the first prediction value, and the second prediction value within a preset time period.

[0242] The calculation module 802 is further configured to determine the co-occurrence call count and call sequence of the expert model based on the call information, wherein the co-occurrence call count is the number of times the expert model is called together with any other expert model;

[0243] The calculation module 802 is further configured to determine the similarity between the expert model and the other expert models based on the correlation information;

[0244] The calculation module 802 is further configured to determine a second predicted value of the expert model based on the number of co-occurrence calls, the called sequence, and the similarity between the expert model and the other expert models.

[0245] The processing module 803 is further configured to load the first target number of expert models into multiple expert memory regions in the video memory, wherein the video memory is divided into multiple expert memory regions, and expert models in different expert memory regions can be called simultaneously.

[0246] The processing module 803 is further configured to determine the expert model for the fourth target number among the expert models for the first target number as a commonly used expert model.

[0247] The processing module 803 is also used to copy the commonly used expert model and load it into the commonly used expert video memory area, wherein the video memory is divided into multiple expert video memory areas and at least one commonly used expert video memory area.

[0248] This application also provides a chip, which includes a processor capable of executing the expert model preloading method provided in this application.

[0249] This application also provides an electronic device.

[0250] Figure 11A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0251] like Figure 11 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded into a random access memory (RAM) 903 from a storage unit 908. The RAM 903 may also store various programs and data required for the operation of the device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0252] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0253] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the expert model preloading method. For example, in some embodiments, the expert model preloading method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the expert model preloading method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the expert model preloading method by any other suitable means (e.g., by means of firmware).

[0254] This application provides a computer-readable storage medium storing executable instructions, wherein a computer program is stored, the computer program being used to execute the expert model preloading method provided in this application.

[0255] This application provides a computer program product comprising a computer program or instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or instructions from the computer-readable storage medium and executes the computer program or instructions, causing the computer device to perform the expert model preloading method described above in this application.

[0256] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0257] In some embodiments, a computer program may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0258] As an example, a computer program may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0259] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0260] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0261] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0262] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0263] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0264] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0265] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0266] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0267] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for preloading expert models, characterized in that, The method includes: In response to the invocation of an expert model, historical data and model parameters of multiple expert models are obtained. The historical data includes at least the invocation information and performance information of the expert models. The multiple expert models are loaded into video memory, memory or hard disk respectively. Extract the number of times the expert model was invoked and the invocation time within a preset time period from the invoked information; The loading time information and cache hit information of the expert model are extracted from the performance information; Obtain the business information and user information of the expert model; The first predicted value of the expert model is determined based on the business information and the user information; The invocation prediction value of the expert model is determined based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, and the first prediction value within a preset time period. The multiple expert models are sorted according to the called prediction values; The expert model for the first number of objectives in the sorting results is loaded into the video memory, the expert model for the second number of objectives in the sorting results is loaded into the memory, and the expert model for the third number of objectives in the sorting results is loaded into the hard disk.

2. The method according to claim 1, characterized in that, Determining the first predicted value of the expert model based on the business information and the user information includes: Based on the business information, determine the business importance information and business key information of the expert model; The user distribution information of the expert model is determined based on the user information, and the user distribution information includes at least the proportion of paying users; The first predicted value is determined based on the business importance information, business key information, and user distribution information of the expert model.

3. The method according to claim 1 or 2, characterized in that, The step of determining the invocation prediction value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, and the model parameters within a preset time period includes: Obtain correlation information between the expert model and other expert models; The second predicted value of the expert model is determined based on the correlation information and the invoked information; The invocation prediction value of the expert model is determined based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, and the second prediction value within a preset time period. Alternatively, the invocation prediction value of the expert model can be determined based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information, the model parameters, the first prediction value, and the second prediction value within a preset time period.

4. The method according to claim 3, characterized in that, Determining the second predicted value of the expert model based on the correlation information and the invoked information includes: The co-occurrence call count and call sequence of the expert model are determined based on the call information, wherein the co-occurrence call count is the number of times the expert model is called together with any other expert model; The similarity between the expert model and the other expert models is determined based on the correlation information; The second prediction value of the expert model is determined based on the number of co-occurrence calls, the call sequence, and the similarity between the expert model and the other expert models.

5. The method according to claim 1, characterized in that, The step of loading the expert model of the number of top-ranked targets in the sorting results into the video memory includes: The first target number of expert models are loaded into multiple expert memory regions in the video memory. The video memory is divided into multiple expert memory regions, and expert models in different expert memory regions can be called simultaneously.

6. The method according to claim 1 or 5, characterized in that, After loading the expert model of the number of top-ranked targets in the sorting results into the video memory, the method further includes: The expert model for the fourth objective among the expert models for the first objective quantity is identified as the commonly used expert model; The commonly used expert model is copied and loaded into the commonly used expert video memory region, which is divided into multiple expert video memory regions and at least one commonly used expert video memory region.

7. An expert model preloading device, characterized in that, The device includes: The acquisition module is used to acquire historical data and model parameters of multiple expert models when the expert model is invoked. The historical data includes at least the invocation information and performance information of the expert model. The multiple expert models are loaded into video memory, memory or hard disk respectively. The calculation module is used to determine the called prediction value of the expert model based on the historical data and model parameters of the expert model; The processing module is used to sort the multiple expert models according to the called prediction values; The processing module is further configured to load the expert model of the first number of target items in the sorting result into the video memory, load the expert model of the first number of target items in the sorting result into the memory, and load the expert model of the first number of target items in the sorting result into the hard disk; The processing module is also used to extract the number of times the expert model was invoked and the invocation time within a preset time period from the invoked information; The processing module is also used to extract the loading time information and cache hit information of the expert model from the performance information; The calculation module is also used to determine the predicted value of the expert model based on the number of times the expert model is invoked, the invocation time, the loading time information, the cache hit information and the model parameters within a preset time period. The acquisition module is also used to obtain the business information and user information of the expert model; The calculation module is further configured to determine the first predicted value of the expert model based on the business information and the user information; The calculation module is further configured to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters, and the first prediction value within a preset time period.

8. The apparatus according to claim 7, characterized in that, include: The calculation module is also used to determine the business importance information and business key information of the expert model based on the business information; The calculation module is further configured to determine the user distribution information of the expert model based on the user information, wherein the user distribution information includes at least the proportion of paying users; The calculation module is further configured to determine the first predicted value based on the business importance information, business key information, user distribution information, and the first preset weight of the expert model.

9. The apparatus according to claim 7 or 8, characterized in that, include: The acquisition module is also used to obtain correlation information between the expert model and other expert models; The calculation module is further configured to determine a second predicted value of the expert model based on the correlation information and the invoked information; The calculation module is also used to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters and the second prediction value within a preset time period; Alternatively, the calculation module is further configured to determine the called prediction value of the expert model based on the number of times the expert model is called, the call time, the loading time information, the cache hit information, the model parameters, the first prediction value, and the second prediction value within a preset time period.

10. The apparatus according to claim 9, characterized in that, include: The calculation module is further configured to determine the number of co-occurrence calls and the call sequence of the expert model based on the call information, wherein the number of co-occurrence calls is the number of times the expert model is called together with any other expert model; The calculation module is further configured to determine the similarity between the expert model and the other expert models based on the correlation information; The calculation module is further configured to determine a second predicted value of the expert model based on the number of co-occurrence calls, the call sequence, and the similarity between the expert model and the other expert models.

11. The apparatus according to claim 7, characterized in that, include: The processing module is further configured to load the first target number of expert models into multiple expert memory regions in the video memory, wherein the video memory is divided into multiple expert memory regions, and expert models in different expert memory regions can be called simultaneously.

12. The apparatus according to claim 7 or 11, characterized in that, include: The processing module is further configured to determine the expert model for the fourth target among the expert models for the first first target number as a commonly used expert model; The processing module is also used to copy the commonly used expert model and load it into the commonly used expert video memory area, wherein the video memory is divided into multiple expert video memory areas and at least one commonly used expert video memory area.

13. A chip, characterized in that, The chip includes a processor capable of executing the expert model preloading method according to any one of claims 1 to 6.

14. An electronic device, characterized in that, The electronic device includes a chip, the chip including a processor, the processor being capable of executing the expert model preloading method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the expert model preloading method according to any one of claims 1 to 6.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, the expert model preloading method according to any one of claims 1 to 6 is implemented.

Images