Computing power scheduling method of intelligent computing center and related device

Abstract

The invention provides a computing power scheduling method of an intelligent computing center and a related device. The method comprises the following steps: S1, acquiring historical prediction data and historical real computing data of each large language model in a large language model set; determining the quality score of each large language model according to the error between the historical prediction data and the historical real calculation data; s2, determining at least one model as a target large language model from the large language model set based on a target function according to the quality score and a task demand of the to-be-executed calculation task; the target function is determined according to a quality score of a model matched with a task requirement in the large language model set and a segmentation calculation task; s3, according to the matching execution relation between the target large language model and the segmented calculation task, the intelligent calculation center is indicated to call the target large language model to execute the calculation task to be executed, and the target large language model for executing the calculation task to be executed with high accuracy can be selected and determined.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computing power infrastructure, and in particular, to a computing power scheduling method and related devices for an intelligent computing center. Background Art

[0002] With the development of artificial intelligence technology and computing power technology, the concept of intelligent computing center has emerged. "Intelligent computing center" refers to the use of large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, mainly for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning and other scenarios) to provide the required computing power, data and algorithms. Intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0003] When faced with computing tasks, it is necessary to specifically determine which large language model to use in the intelligent computing center to perform the computing tasks. The models determined by existing methods have low accuracy when performing computing tasks. Summary of the invention

[0004] The embodiments of the present invention provide a computing power scheduling method and related devices for an intelligent computing center to solve the problem that when faced with computing tasks, it is necessary to specifically determine which large language model to use in the intelligent computing center to perform computing tasks, and the model determined by the existing method has low accuracy when performing computing tasks.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a computing power scheduling method for an intelligent computing center, including:

[0007] Step S1: obtaining historical prediction data of each of the large language models in the large language model set, and obtaining historical real calculation data of the historical calculation tasks corresponding to the historical prediction data in actual scenarios; determining the quality score of each of the large language models according to the error between the historical prediction data and the historical real calculation data; wherein each of the large language models in the large language model set is deployed in an intelligent computing center;

[0008] Step S2: According to the quality score and the task requirements of the computing task to be executed by the intelligent computing center, at least one model is determined as a target large language model from the large language model set based on the objective function; each of the target large language models matches and executes at least one segmented computing task in the computing task to be executed, and the segmented computing tasks matched by all the target large language models are added together to obtain the computing task to be executed; the objective function is determined according to the quality score of the model matching the task requirements in the large language model set and the segmented computing task;

[0009] Step S3: According to the matching execution relationship between the target large language model and the segmented computing task, instruct the intelligent computing center to call the target large language model to execute the computing task to be executed.

[0010] Optionally, the step S3 includes:

[0011] Step S301: During the execution of the computing task to be executed by the target large language model, the intelligent computing center calls the query large language model deployed in the intelligent computing center to identify the query requirements of the computing task to be executed, obtains an SQL query statement that meets the query requirements, and queries the SQL database with the SQL query statement.

[0012] Optionally, the step S301 includes:

[0013] Step S3011: optimizing the model reasoning process of the query large language model identifying the query requirement to improve reasoning efficiency;

[0014] The optimization process includes at least one of the following: prefix sharing optimization, deduplication optimization, query optimization, and batch processing optimization.

[0015] Optionally, the objective function is expressed as follows:

[0016]

[0017] Among them, S i,j Perform segment calculation task d for matching j The large language model M i The quality score of i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirements.

[0018] Optionally, the constraint condition of the target large language model includes at least one of the following constraint conditions: a first constraint condition, a second constraint condition;

[0019] The expression of the first constraint is as follows:

[0020]

[0021] Among them, C i,j For model M i Processing segmented computing tasks d j The cost required; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirement; B is the preset total cost;

[0022] The expression of the second constraint is as follows:

[0023]

[0024] Among them, l i,j For model M i Processing segmented computing tasks d j delay; L is the preset delay threshold; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirements.

[0025] Optionally, the step S3 includes:

[0026] Step S302: Optimizing the number of tokens required to call the target large language model.

[0027] In a second aspect, an embodiment of the present invention provides a computing power scheduling device for an intelligent computing center, including:

[0028] An acquisition module is used to acquire historical prediction data of each of the large language models in the large language model set, and to acquire historical real calculation data of the historical calculation tasks corresponding to the historical prediction data in actual scenarios; and to determine the quality score of each of the large language models according to the error between the historical prediction data and the historical real calculation data; wherein each of the large language models in the large language model set is deployed in an intelligent computing center;

[0029] An analysis module is used to determine at least one model from the large language model set as a target large language model based on an objective function according to the quality score and the task requirements of the computing task to be executed by the intelligent computing center; each of the target large language models matches and executes at least one segmented computing task in the computing task to be executed, and the segmented computing tasks matched by all the target large language models are added together to obtain the computing task to be executed; the objective function is determined according to the quality score of the model matching the task requirements in the large language model set and the segmented computing task;

[0030] An execution module is used to instruct the intelligent computing center to call the target large language model to execute the to-be-executed computing task according to the matching execution relationship between the target large language model and the segmented computing task.

[0031] In a third aspect, an embodiment of the present invention provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps in the computing power scheduling method of an intelligent computing center as described in any one of the first aspects.

[0032] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps in the computing power scheduling method of the intelligent computing center as described in any one of the first aspects are implemented.

[0033] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the computing power scheduling method for an intelligent computing center as described in any one of the first aspects.

[0034] In an embodiment of the present invention, through step S1: obtaining historical prediction data of each language model in a large language model set, and obtaining historical real calculation data of historical calculation tasks corresponding to the historical prediction data in actual scenarios; determining the quality scores of each language model according to the error between the historical prediction data and the historical real calculation data; wherein each language model in the large language model set is deployed in an intelligent computing center; step S2: determining at least one model as a target large language model from the large language model set based on an objective function according to the quality score and the task requirements of the to-be-executed calculation task to be executed by the intelligent computing center; each target large language model matches and executes at least one segmented calculation task in the to-be-executed calculation task, and the segmented calculation tasks matched by all target large language models are added to obtain the to-be-executed calculation task; the objective function is determined according to the quality score of the model matching the task requirements in the large language model set and the segmented calculation task; step S3: according to the matching execution relationship between the target large language model and the segmented calculation task, instructing the intelligent computing center to call the target large language model to execute the to-be-executed calculation task, the embodiment of the present invention can select and determine the target large language model that executes the to-be-executed calculation task with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0036] Figure 1 A schematic diagram of a flow chart of a computing power scheduling method for an intelligent computing center according to an embodiment of the present invention;

[0037] Figure 2 This is a principle block diagram of a computing power scheduling device for an intelligent computing center according to an embodiment of the present invention;

[0038] Figure 3 The figure is a functional block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0041] In the technical solution of the present disclosure, words such as “connect”, “couple” or “connected” are not limited to physical or mechanical connections, but may include electrical connections.

[0042] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0043] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0044] First, the technical terms involved in the present invention are briefly explained below.

[0045] The "computing power" mentioned in the present invention is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to execute certain computing requirements. It is the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to the society through computing power infrastructure.

[0046] The "computing power" (Computational Power, CP) described in the present invention is the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating point operations performed per second (FLOPS: Floating Point Operations Per Second, 1EFLOPS = 10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs or 2 million mainstream notebooks. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0047] The "Network Power" (NP) described in the present invention is a manifestation of the data transmission capability of computing facilities, including comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. The carrying capacity involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities. In the embodiment of the present invention, the carrying capacity uses the memory bandwidth.

[0048] The "Storage Power" (SP) described in the present invention is the comprehensive ability of a data center in terms of data storage capacity, performance, safety and reliability, and green and low-carbon. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and server built-in storage devices. The commonly used unit of measurement for storage capacity is exabyte (EB, 1EB = 2^60bytes), and the commonly used unit of measurement for performance is the number of reads and writes per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB). The disaster recovery ratio is an important manifestation of safety and reliability.

[0049] The "computing power infrastructure" described in the present invention is a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity. It can realize centralized computing, storage, transmission and application of information, and presents characteristics such as diversity and ubiquity, intelligence and agility, security and reliability, and green and low-carbon.

[0050] The “computing power” mentioned in the present invention includes general computing power, intelligent computing power and super computing power.

[0051] The “general computing power” mentioned in the present invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0052] The "intelligent computing power" described in the present invention is a computing platform for large-scale deployment of special chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) for various innovative applications of artificial intelligence, such as natural language processing and machine vision.

[0053] The "super computing power" mentioned in the present invention is mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0054] The "intelligent computing center" described in the present invention refers to a facility that provides the required computing power, data and algorithms for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning scenarios) by using large-scale heterogeneous computing power resources, including general computing power (CPU: Central Processing Unit) and intelligent computing power (GPU: Graphics Processing Unit, FPGA: Field Programmable Gate Array, ASIC: Application Specific Integrated Circuit, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0055] The "computing resources" mentioned in the present invention refer to the technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of the digital society, including but not limited to computing resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, as well as supporting and guarantee resources such as wind, fire, water and electricity.

[0056] The "Large Language Model (LLM)" described in the present invention is an artificial intelligence model based on deep learning, which is designed to understand and generate natural language. These models are usually trained with large amounts of text data to learn the structure, grammar, semantics and context of the language, and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.

[0057] The embodiment of the present invention provides a computing power scheduling method for an intelligent computing center, see Figure 1 As shown, Figure 1 The flowchart of the computing power scheduling method of the intelligent computing center according to the embodiment of the present invention is as follows:

[0058] Step S1: Obtain historical prediction data of each language model in the large language model set, and obtain historical real calculation data of historical calculation tasks corresponding to the historical prediction data in actual scenarios; determine the quality scores of each language model according to the error between the historical prediction data and the historical real calculation data; wherein each language model in the large language model set is deployed in the intelligent computing center;

[0059] Step S2: According to the quality score and the task requirements of the computing task to be executed by the intelligent computing center, at least one model is determined as the target large language model from the large language model set based on the objective function; each target large language model matches and executes at least one segmented computing task in the computing task to be executed, and the segmented computing tasks matched by all the target large language models are added to obtain the computing task to be executed; the objective function is determined according to the quality score and segmented computing task of the model matching the task requirements in the large language model set;

[0060] Step S3: According to the matching execution relationship between the target large language model and the segmented computing tasks, instruct the intelligent computing center to call the target large language model to execute the computing tasks to be executed.

[0061] It should be noted that the historical prediction data in the embodiment of the present invention may refer to the output data of each language model, and the type of the output data matches the historical calculation task, and it is not a prediction of a certain calculation result according to the literal meaning. In other words, the quality scores of each language model are determined according to the error between the historical prediction data and the historical real calculation data in step S1, which can actually be the error comparison between the output data of the large language model for the historical calculation task and the real data to obtain the quality scores of each language model.

[0062] In addition, the actual scenario can be the actual computing resources, actual time, and actual input conditions used when executing the historical computing task. It should be noted that in some optional embodiments, in order to improve the accuracy of the quality score, the historical real calculation data can be the average or median of multiple historical real calculation values ​​of the historical computing task corresponding to the historical prediction data in the actual scenario. Correspondingly, the historical prediction data can also be the average or median of multiple historical prediction values, specifically the average or median, which is the same as the historical real calculation data.

[0063] In some optional embodiments, the mean square error (MSE) loss (denoted as L MSE ) as the quality score, then determine the quality score L of each language model MSE The formula is as follows:

[0064]

[0065] in, is the predicted value (i.e., the historical predicted data in the embodiment of the present invention); i is the true value (i.e., the historical real calculation data in the embodiment of the present invention); n′ is the total number of data points representing the calculation error in a batch or set (in the embodiment of the present invention, n′ is the total amount of historical prediction data).

[0066] In some specific embodiments of the present invention, optionally, based on the quality score and the task requirements of the computing tasks to be executed by the intelligent computing center, at least one model is determined as the target large language model from the large language model set, including: based on the task requirements of the computing tasks to be executed, a model that matches the task requirements is determined from the large language model set from the objective function, for example: if the task requirements of one or more segmented computing tasks in the computing tasks to be executed are to recognize audio information, then a model that can be used to recognize audio information is determined from the large language model set, and so on, which will not be repeated here. In combination with all the task requirements of the segmented computing tasks, a subset consisting of models that can meet the task requirements can be determined from the large language model set. Furthermore, based on the quality score, a target large language model that matches the execution of each segmented computing task is determined from the above subset.

[0067] In some embodiments of the present invention, optionally, according to the quality score, the target large language model is determined from the subset of models that can meet the task requirements, so as to ensure that the quality score reaches the highest optimization goal, and the target large language model is determined from the subset. For example: ensure that the sum of the quality scores of the target large language models used in all segmented computing tasks is the highest, or ensure that the product of the quality scores of the target large language models used in all segmented computing tasks is the highest.

[0068] It should be noted that, based on determining the target large language model according to the quality score, there is a situation where one target large language model matches and executes multiple segmented computing tasks.

[0069] In step S3 of the embodiment of the present invention, according to the matching execution relationship between the target large language model and the segmented computing task, the intelligent computing center is instructed to call the target large language model to execute the computing task to be executed. For example, the target large language model a1 that matches and executes segmented computing task A determined in step S2 is assigned to perform the calculation of segmented computing task A, and the target large language model a2 that matches and executes segmented computing task B determined in step S2 is assigned to perform the calculation of segmented computing task B, and so on. No further details are given here.

[0070] In some embodiments of the present invention, the computing task may be optionally described in text form, for example, computing task w is "identify all incoming call recordings within one hour and generate a text summary", or "identify employee k in the surveillance video and capture all video clips that include employee k". It should be noted that the collection, collection, update, analysis, processing, use, transmission, storage and other aspects of personal information in incoming call recordings and surveillance videos are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals.

[0071] In an embodiment of the present invention, through step S1: obtaining historical prediction data of each language model in a large language model set, and obtaining historical real calculation data of historical calculation tasks corresponding to the historical prediction data in actual scenarios; determining the quality scores of each language model according to the error between the historical prediction data and the historical real calculation data; wherein each language model in the large language model set is deployed in an intelligent computing center; step S2: determining at least one model as a target large language model from the large language model set based on an objective function according to the quality score and the task requirements of the to-be-executed calculation task to be executed by the intelligent computing center; each target large language model matches and executes at least one segmented calculation task in the to-be-executed calculation task, and the segmented calculation tasks matched by all target large language models are added to obtain the to-be-executed calculation task; the objective function is determined according to the quality score of the model matching the task requirements in the large language model set and the segmented calculation task; step S3: according to the matching execution relationship between the target large language model and the segmented calculation task, instructing the intelligent computing center to call the target large language model to execute the to-be-executed calculation task, the embodiment of the present invention can select and determine the target large language model that executes the to-be-executed calculation task with high accuracy.

[0072] In some embodiments of the present invention, optionally, step S3 includes:

[0073] Step S301: During the execution of the computing task to be executed by the target large language model, the intelligent computing center calls the query large language model deployed in the intelligent computing center to identify the query requirements of the computing task to be executed, obtains the SQL query statement that meets the query requirements, and queries the SQL database with the SQL query statement.

[0074] In actual applications, during the execution of computing tasks, it is inevitable that the intelligent computing center needs to query data from the SQL database. For example, the computing task to be executed proposes descriptions of query requirements such as "combining the multiple parameters described above", "using xxxx data", "based on xxx data", "excluding x data from xx data", etc., then when executing the computing task, the intelligent computing center needs to query the above data (multiple parameters, xxxx data, xxx data, x data excluding xx data) pre-stored in the SQL database from the SQL database. For example, if the computing task w is "identify all incoming call recordings within one hour and generate a text summary", then when executing the computing task, the intelligent computing center needs to query the above data (all incoming call recordings within one hour) pre-stored in the SQL database from the SQL database. For example, if the computing task w1 is "identify employee k in the surveillance video and capture all video clips including employee k", then when executing the computing task, the intelligent computing center needs to query the above data (video data) pre-stored in the SQL database from the SQL database. Therefore, it can be understood that the intelligent computing center calls the query language model deployed in the intelligent computing center to identify the query requirements of the computing tasks to be executed. Specifically, in this example, it identifies the query requirements contained in the context semantics, which can improve the query efficiency and thus improve the computing power scheduling efficiency.

[0075] In some embodiments of the present invention, optionally, the query large language model can also generate an SQL query statement that meets the query requirements based on the identified query requirements. The above functions of the query large language model can be implemented based on the existing technology, and are not the key points of improvement of the present application, so they will not be described in detail here. Implementation method, exemplarily, the training process of the query large language model includes: using historical SQL query statements to annotate the text description (text) containing the query requirements to obtain a training data set; using the training data set to train the large language model to obtain a query large language model.

[0076] The following is a description with specific examples. Through natural language questions (such as "how to make a cake"), the query large language model identifies the user's intention (that is, obtains the query requirements) and generates SQL query statements to extract relevant information from the SQL database. For example, if you enter "how to make a cake", the query large language model will generate an SQL query statement "SELECT Recipe_Name, Ingredients, Steps FROM Recipes WHERE Recipe_Name LIKE'% Cake%';". After the intelligent computing center queries the SQL database with this SQL query statement, it will return the cake name, required ingredients and production steps, allowing users to intuitively obtain answers.

[0077] In some embodiments of the present invention, optionally, step S301 includes:

[0078] Step S3011: Optimizing the model reasoning process of querying the large language model to identify the query requirements to improve the reasoning efficiency;

[0079] The optimization process includes at least one of the following: prefix sharing optimization, deduplication optimization, query optimization, and batch processing optimization.

[0080] 1) Prefix sharing optimization: By dynamically adjusting the column and row order of data (based on column sorting algorithm and row sorting algorithm), the prefix sharing degree of input data is maximized, the hit rate of LLM's key-value cache (KV, also known as Key-Value Cache, the key-value cache is used to store and reuse the calculation results in large language model calls) is improved, repeated calculations are reduced, and system overhead and running time are reduced.

[0081] Column sorting algorithm: Calculate the average string length and number of unique values ​​of each data column to get a score. According to the score, determine the priority of the shared prefix of the column and sort the columns with the highest degree of shared prefix in front.

[0082] The scoring formula is: Among them, ASL is the average string length of the column; Score(col) is the priority score of the current column. Columns with higher scores are considered to have greater prefix sharing potential and are prioritized. ASL_col (Average String Length) is the average length of the strings in the current column. Longer strings usually mean a larger potential prefix sharing space. Total Number of Rows is the total number of rows in the data table, which reflects the size of the data set and is used to measure the overall impact of the data in the column. C_col (Cardinality) is the number of unique values ​​in the current column (cardinality). The smaller the cardinality, the more repeated values ​​there are, and the greater the possibility of prefix sharing.

[0083] The priority here is not just the one with the highest score, but is determined through a comprehensive optimization strategy. The sorting of columns and rows is based on the principle of maximizing prefix sharing, and the column sorting score calculation formula (i.e., the scoring formula) reflects the key indicators of the column priority in optimization (such as the combination of string length and the number of unique values). In short, the score is the basis for determining the priority, but the final sorting must also be combined with the actual needs of the specific cache hit rate and performance improvement to ensure that the overall efficiency is maximized during the optimization process.

[0084] Row sorting algorithm: concatenate all column values ​​into strings and sort them by string dictionary so that rows with the same prefix are concentrated to improve the utilization efficiency of the KV cache.

[0085] In the process of column sorting and row sorting, column sorting aims to improve the utilization of KV cache by maximizing shared prefixes, while row sorting adjusts the order of rows to bring rows with similar prefixes closer together. The two do not conflict, because column sorting determines the order of columns, while row sorting is performed after the column arrangement is determined, and it adjusts the order of rows in the same column set.

[0086] Result of prefix sharing optimization: The maximum prefix sharing is determined by the KV cache hit rate. The combined result of column sorting and row sorting is determined by the optimization effect of improving the cache hit rate. To illustrate with a specific example, assume that there is a data table with the following columns:

[0087] Customer_Name

[0088] Product_Category

[0089] Purchase_Date

[0090] Column sorting:

[0091] Calculate the score for each column according to the scoring formula:

[0092] Customer_Name: ASL is longer and cardinality is higher.

[0093] Product_Category: ASL is shorter, but has lower cardinality (more repeated values).

[0094] Purchase_Date: Shorter ASL, higher cardinality.

[0095] Sorting result: Product_Category>Customer_Name>Purchase_Date.

[0096] Row sorting:

[0097] Concatenate the rows into strings and sort them lexicographically based on the order of the columns:

[0098] Stitching result: ElectronicsAlice2023-01-01, ElectronicsBob2023-01-02.

[0099] 2) Deduplication optimization: Before LLM is called, duplicate input requests are identified and removed to avoid multiple inferences on the same input and reduce computational overhead.

[0100] Deduplication step: Use data structures such as hash tables to identify and remove duplicate input requests, and perform LLM reasoning only on unique input text.

[0101] Result mapping: After LLM reasoning is completed, the results are mapped back to all corresponding positions in the original data set to ensure the integrity of the query results. Specifically, according to the deduplication optimization and result mapping mechanism, the reasoning results will be accurately mapped back to all corresponding positions in the original data set to ensure the integrity and consistency of the data. For example: A1, A2, and B are inputs before the call, A1 and A2 are repeated, and after deduplication, they are A1 and B. The reasoning result is that A1 gets C1 and B gets C2. After result mapping, A1 gets C1, A2 gets C1, and B gets C2. In this scenario, A1 and A2 have the same content, so the reasoning results are both C1, while B's reasoning result is C2. This mapping design not only ensures the accuracy of the query results, but also effectively avoids the waste of resources caused by repeated calls, fully reflecting the efficiency and reliability of system optimization.

[0102] Extension to semantic deduplication: Semantic-based deduplication technology can be introduced to identify semantically equivalent texts to further reduce duplicate calls.

[0103] 3) Query optimization: introduce the cost estimation mechanism of LLM operators, optimize the query execution order based on cost estimation, and postpone high-cost LLM calls to low-cost filtering and other operations. Specific packages:

[0104] Cost estimation mechanism: Cost estimation is performed on LLM operators, and the execution cost is evaluated based on the expected execution time and resource consumption of LLM calls. Execution order optimization: Based on cost estimation, non-LLM filtering operations are prioritized to reduce the amount of data that needs to be inferred by LLM, so as to achieve significant performance optimization. Cost C LLM The estimation formula is as follows:

[0105] C LLM =L input ×C input +L decode ×C decode

[0106] Among them, L input L is the token length of the input prompt; decode is the decoded (or output) length; C input The computation cost for each input token; C decode Compute cost for each decoded token.

[0107] Input cost item L input ×C input : Proportional to the input token length, reflecting the computational overhead of processing the input prompt.

[0108] Decoding cost term L decode ×C decode : Proportional to the decoded length, representing the computational overhead of generating the output.

[0109] For example, suppose there is a data set containing timestamps and comment content. The query optimization process is divided into two steps:

[0110] Ignore the cost of LLM operators: Call LLM UDF directly for each row, and then apply the timestamp filter. The core function of the timestamp filter is to prioritize reducing the amount of data, thereby reducing the computational overhead of subsequent LLM operator calls. Comments that do not meet the query time range are directly filtered to avoid invalid data from participating in LLM reasoning, significantly improving the processing efficiency of the system. For comments outside the time range, not calling LLM is a clear optimization strategy because this data is semantically irrelevant and cannot contribute to the query results. By prioritizing the application of timestamp filters in SQL queries, high-cost LLM reasoning operations can be limited to smaller data sets, fundamentally reducing resource consumption and improving query execution efficiency. This strategy is not only a precise control of resource utilization, but also a necessary means to optimize large-scale query systems.

[0111] Consider the LLM operator cost: apply the timestamp filter first, reduce the dataset size, and then call the LLM UDF.

[0112] Here, the reason for executing non-LLM operators first is to significantly reduce the amount of data that needs to make LLM calls, thereby reducing the overall computational cost.

[0113] Here’s how it works:

[0114] The computational cost of non-LLM operations is lower: The computational cost of operators such as filtering and projection is usually much lower than the inference cost of LLM. Therefore, prioritizing these operations can reduce the size of the data at a lower cost.

[0115] Data reduction effect: Non-LLM operations (e.g., filtering operations) can exclude irrelevant data at an early stage, reducing the number of input rows that need to be passed to LLM. In this way, LLM only needs to process the reduced data set, which greatly reduces the number of inference calls and computing resource consumption.

[0116] High cost of LLM calls: The inference cost of LLM is directly related to the input length and output length. By prioritizing non-LLM operations, the amount of input passed to LLM can be reduced, thereby reducing the inference cost.

[0117] Optimize query execution plan: SQL query optimizer will sort by the cost of each operation. The execution order of non-LLM operations is cost-estimated, which can reduce the computational burden for subsequent high-cost LLM operations, thereby optimizing overall performance.

[0118] Therefore, the priority execution of non-LLM operators is based on the principle of decreasing cost to minimize the data size and frequency of LLM calls, thereby achieving the goal of reducing costs and increasing efficiency.

[0119] 4) Batch optimization: For queries involving multiple LLM calls, multiple requests are executed in batch order to fully utilize the sharing of KV cache and reduce repeated calculations. Batch optimization strategy: Sort requests based on the prefix information in the request to ensure that requests with the same prefix are processed in one batch, reduce the number of cache loads, and improve inference efficiency.

[0120] In some embodiments of the present invention, optionally, the objective function is expressed as follows:

[0121]

[0122] Among them, S i,j Perform segment calculation task d for matching j The large language model M i Quality rating of x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j , for example: a value of 1 indicates selection, and a value of 0 indicates non-selection; D is the computing task to be executed; and M is the model in the large language model set that matches the task requirements.

[0123] In the embodiment of the present invention, the above objective function is used to ensure that the target large language model is determined by achieving the highest optimization goal with a quality score, and the objective function with a high quality score executes the task to be executed, ensuring that a high-accuracy execution result is obtained after executing the computing task to be executed.

[0124] In some embodiments of the present invention, optionally, the constraint condition of the target large language model includes at least one of the following constraint conditions: a first constraint condition, a second constraint condition;

[0125] The expression of the first constraint is as follows:

[0126]

[0127] Among them, C i,j For model M i Processing segmented computing tasks d j The cost required; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j , for example: the value is 1 for selection, and the value is 0 for non-selection; D is the computing task to be executed; M is the model that matches the task requirements in the large language model set; B is the preset total cost;

[0128] The expression of the second constraint is as follows:

[0129]

[0130] Among them, l i,j For model M i Processing segmented computing tasks d j delay; L is the preset delay threshold; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j , for example: a value of 1 indicates selection, and a value of 0 indicates non-selection; D is the computing task to be executed; and M is the model in the large language model set that matches the task requirements.

[0131] In the embodiment of the present invention, the first constraint condition and the second constraint condition are used to ensure optimal output quality while controlling costs.

[0132] In some embodiments of the present invention, optionally, step S3 includes:

[0133] Step S302: Optimize the number of tokens required to call the target large language model.

[0134] Reducing the number of input tokens is an effective way to reduce the cost of LLM calls, especially in scenarios where charges are based on the number of tokens. Optimizing the number of tokens required to call the target large language model, using techniques such as sentence simplification and semantic analysis, while compressing the number of tokens to ensure information integrity, reduces computing costs and avoids additional overhead caused by reduced output quality.

[0135] Specifically, step S302 can reduce the length of the input text without significantly affecting the output quality through sentence simplification and heuristic-based methods (e.g., deleting stop words, merging synonyms), thereby effectively reducing the computational cost of LLM. Token optimization can be modeled by a knapsack problem, where the number of tokens is minimized while maintaining output quality. The token optimization formula is as follows:

[0136]

[0137] This formulation is used to model the knapsack problem in the token optimization module to maximize the output quality under resource constraints.

[0138] The purpose of this objective function is to maximize the total benefit (i.e., quality score) of the selected items, where: p i,j To perform segmented computing tasks d j The large language model M i Quality rating of x i,j Is a binary variable indicating whether model M is selected i Processing paragraph d j , a value of 1 indicates selection, and a value of 0 indicates deselection.

[0139] This constraint ensures that the cost of the choice does not exceed a given budget C, where: i,j For model M i Processing segmented computing tasks d j The cost required; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j , for example: the value is 1 for selection, the value is 0 for non-selection; D is the computational task to be performed; C is the preset total cost. This constraint ensures the optimal selection under the control of the budget.

[0140] In this way, the optimization model strikes a balance between quality and cost in order to maximize the quality score within the budget constraints.

[0141] The following is a description with reference to specific embodiments.

[0142] A computing power-driven large language model call optimization system based on quality prediction and budget-aware optimization in an intelligent computing center (hereinafter referred to as the system, or the optimization system). The system realizes the intelligentization and dynamic optimization of LLM calls from multiple dimensions by introducing a quality predictor module, a budget-aware optimization algorithm module, and a token optimization module.

[0143] D1. Quality Predictor Module

[0144] The quality predictor module uses the BERTScore metric to estimate the output quality of different LLMs. By modeling the semantic features of the input text, it estimates the output quality that different LLMs may generate. In specific implementation, the pre-trained BERT model is used to perform semantic analysis on the input text, and then the quality score is calculated in combination with the regression head. This process does not require the actual call of the LLM, but estimates the output performance of each model under a specific input in advance based on the semantic distribution of the known training data.

[0145] For the same input text, the potential output quality of different LLMs is evaluated based on the pre-trained predictor model. By comparing multiple dimensions such as semantic consistency and structural integrity of each model, the predictor can generate a quality ranking. This method not only avoids the high cost of frequent LLM calls, but also significantly improves the response speed, providing reliable support for real-time tasks.

[0146] Through this predictor, the system can estimate the output quality of LLM without actually calling it, thereby avoiding frequent calls to LLM and reducing latency and costs. The predictor is trained on a specific dataset and uses a multi-layer network and GELU activation function to improve prediction results.

[0147] The error calculation formula for BERTScore quality prediction is as follows:

[0148]

[0149] in, is the predicted value (i.e., the historical predicted data in the embodiment of the present invention); i is the true value (i.e., the historical real calculation data in the embodiment of the present invention); n′ is the total number of data points representing the calculation error in a batch or set (in the embodiment of the present invention, n′ is the total amount of historical prediction data).

[0150] L MSE Denotes the mean squared error (MSE) loss. : This term is used to normalize the sum of the errors by dividing it by the number of samples n′, where n′ represents the total number of data points for which the errors are calculated in this batch or set. The summation symbol indicates the accumulation of errors of all data points i from 1 to n′. The double vertical bars ||·|| in the expression represent the Euclidean norm (or L2 norm), and the inner Represents the true value y i With the predicted value The square operation will amplify the larger error value, making MSE more sensitive to outliers.

[0151] The mean squared error (MSE) loss calculates the average of the squared differences between the predicted and true values. Since the square of the difference will amplify large error values, the model will pay special attention to and try to minimize large deviations during training. By averaging over n′, MSE provides a standardized error metric that avoids the impact of data scale or batch size on the results.

[0152] In regression tasks, the mean square error loss function is often used to make the predicted value as close to the actual value as possible. The goal of the optimization process is to minimize

[0153] D2. Budget-aware optimization algorithm module

[0154] The budget-aware optimization algorithm selects the best LLM within the budget and latency constraints through linear programming, ensuring the best output quality while controlling costs. The optimization problem selects the most suitable LLM to handle each task based on the input context and task requirements.

[0155] Assuming the system budget is B and the delay limit is L, the optimization problem can be expressed as:

[0156]

[0157] Among them, S i,j Perform segment calculation task d for matching j The large language model M i Quality rating of x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j , for example: a value of 1 indicates selection, and a value of 0 indicates non-selection; D is the computing task to be executed; and M is the model in the large language model set that matches the task requirements.

[0158] And the following constraints (i.e., the first constraint and the second constraint in the embodiment of the present invention) are satisfied:

[0159]

[0160] Among them, C i,j For model M i Processing segmented computing tasks d j The cost required; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j , for example: the value is 1 for selection, and the value is 0 for non-selection; D is the computing task to be executed; M is the model that matches the task requirements in the large language model set; B is the preset total cost;

[0161]

[0162] Among them, l i,j For model M i Processing segmented computing tasks d j delay; L is the preset delay threshold; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j , for example: a value of 1 indicates selection, and a value of 0 indicates non-selection; D is the computing task to be executed; and M is the model in the large language model set that matches the task requirements.

[0163] D3, Token Optimization Module

[0164] The token optimization module uses a variety of methods to reduce the number of input tokens to further reduce the call cost. Through sentence simplification and heuristic-based methods (such as removing stop words and merging synonyms), this module reduces the length of the input text without significantly affecting the output quality, thereby effectively reducing the computational cost of LLM.

[0165] Token optimization can be modeled by the knapsack problem, where the number of tokens is minimized while maintaining the output quality. The token optimization formula is as follows:

[0166]

[0167] This formulation is used to model the knapsack problem in the token optimization module to maximize the output quality under resource constraints.

[0168] The purpose of this objective function is to maximize the total benefit (i.e., quality score) of the selected items, where: p i,j To perform segmented computing tasks d j The large language model M i Quality rating of x i,j Is a binary variable indicating whether model M is selected i Processing paragraph d j , a value of 1 indicates selection, and a value of 0 indicates deselection.

[0169] This constraint ensures that the cost of the choice does not exceed a given budget C, where: i,j For model M i Processing segmented computing tasks d j The cost required; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j, for example: the value is 1 for selection, the value is 0 for non-selection; D is the computational task to be performed; C is the preset total cost. This constraint ensures the optimal selection under the control of the budget.

[0170] In this way, the optimization model strikes a balance between quality and cost in order to maximize the quality score within the budget constraints.

[0171] The embodiment of the present invention provides a computing power scheduling device for an intelligent computing center, see Figure 2 As shown, Figure 2 This is a principle block diagram of a computing power scheduling device of an intelligent computing center according to an embodiment of the present invention. The computing power scheduling device 20 of the intelligent computing center includes:

[0172] The acquisition module 21 is used to obtain the historical prediction data of each of the large language models in the large language model set, and to obtain the historical real calculation data of the historical calculation task corresponding to the historical prediction data in the actual scenario; according to the error between the historical prediction data and the historical real calculation data, the quality score of each of the large language models is determined; wherein each of the large language models in the large language model set is deployed in an intelligent computing center;

[0173] The analysis module 22 is used to determine at least one model from the large language model set as a target large language model based on an objective function according to the quality score and the task requirements of the computing task to be executed by the intelligent computing center; each of the target large language models matches and executes at least one segmented computing task in the computing task to be executed, and the segmented computing tasks matched by all the target large language models are added together to obtain the computing task to be executed; the objective function is determined according to the quality score of the model matching the task requirements in the large language model set and the segmented computing task;

[0174] The execution module 23 is used to instruct the intelligent computing center to call the target large language model to execute the computing task to be executed according to the matching execution relationship between the target large language model and the segmented computing task.

[0175] In some embodiments of the present invention, optionally, the execution module 23 is also used for, during the execution process of the target large language model executing the computing task to be executed, the intelligent computing center calls the query large language model deployed in the intelligent computing center to identify the query requirements of the computing task to be executed, obtains an SQL query statement that meets the query requirements, and queries the SQL database with the SQL query statement.

[0176] In some embodiments of the present invention, optionally, the execution module 23 is further used to optimize the model reasoning process of the query large language model identifying the query requirement to improve reasoning efficiency;

[0177] The optimization process includes at least one of the following: prefix sharing optimization, deduplication optimization, query optimization, and batch processing optimization.

[0178] In some embodiments of the present invention, optionally, the objective function is expressed as follows:

[0179]

[0180] Among them, S i,j Perform segment calculation task d for matching j The large language model M i The quality score of i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirements.

[0181] In some embodiments of the present invention, optionally, the constraint condition of the target large language model includes at least one of the following constraint conditions: a first constraint condition, a second constraint condition;

[0182] The expression of the first constraint is as follows:

[0183]

[0184] Among them, C i,j For model M i Processing segmented computing tasks d j The cost required; x i,j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirement; B is the preset total cost;

[0185] The expression of the second constraint is as follows:

[0186]

[0187] Among them, l i,j For model M i Processing segmented computing tasks d j delay; L is the preset delay threshold; x i,jis a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirements.

[0188] In some embodiments of the present invention, optionally, the execution module 23 is further configured to optimize the number of tokens required to call the target large language model.

[0189] The computing power scheduling device of the intelligent computing center provided in the embodiment of the present application can achieve Figure 1 The various processes implemented in the method embodiment achieve the same technical effect and will not be described again here to avoid repetition.

[0190] The embodiment of the present invention provides an electronic device 30, see Figure 3 As shown, Figure 3 This is a principle block diagram of an electronic device 30 according to an embodiment of the present invention, including a processor 31, a memory 32, and a program or instruction stored in the memory 32 and executable on the processor 31. When the program or instruction is executed by the processor, the steps in the computing power scheduling method of any intelligent computing center of the present invention are implemented.

[0191] An embodiment of the present invention provides a readable storage medium, on which programs or instructions are stored. When the programs or instructions are executed by a processor, the various processes of an embodiment of a computing power scheduling method for an intelligent computing center such as any of the above-mentioned items are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0192] The readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0193] An embodiment of the present invention also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the computing power scheduling method embodiment of any of the above-mentioned intelligent computing centers are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0194] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A computing power scheduling method for an intelligent computing center, characterized in that: include: Step S1: obtaining historical prediction data of each major language model in a large language model set, and obtaining historical real calculation data of historical calculation tasks corresponding to the historical prediction data in actual scenarios; Determining a quality score of each of the large language models according to an error between the historical prediction data and the historical real calculation data; wherein each of the large language models in the large language model set is deployed in an intelligent computing center; Step S2: According to the quality score and the task requirements of the computing task to be executed by the intelligent computing center, at least one model is determined as a target large language model from the large language model set based on the objective function; each of the target large language models matches and executes at least one segmented computing task in the computing task to be executed, and the segmented computing tasks matched by all the target large language models are added together to obtain the computing task to be executed; the objective function is determined according to the quality score of the model matching the task requirements in the large language model set and the segmented computing task; Step S3: According to the matching execution relationship between the target large language model and the segmented computing task, instruct the intelligent computing center to call the target large language model to execute the computing task to be executed.

2. The computing power scheduling method of the intelligent computing center according to claim 1 is characterized in that: The step S3 comprises: Step S301: During the execution of the computing task to be executed by the target large language model, the intelligent computing center calls the query large language model deployed in the intelligent computing center to identify the query requirements of the computing task to be executed, obtains an SQL query statement that meets the query requirements, and queries the SQL database with the SQL query statement.

3. The computing power scheduling method of the intelligent computing center according to claim 2 is characterized in that: The step S301 includes: Step S3011: optimizing the model reasoning process of the query large language model identifying the query requirement to improve reasoning efficiency; The optimization process includes at least one of the following: prefix sharing optimization, deduplication optimization, query optimization, and batch processing optimization.

4. The computing power scheduling method of the intelligent computing center according to claim 1, characterized in that: The expression of the objective function is as follows: Among them, S i，j Perform segment calculation task d for matching j The large language model M i The quality score of i，j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirements.

5. The computing power scheduling method of the intelligent computing center according to claim 4 is characterized in that: The constraint condition of the target large language model includes at least one of the following constraint conditions: a first constraint condition and a second constraint condition; The expression of the first constraint is as follows: Among them, C i，j For model M i Processing segmented computing tasks d j The cost required; x i，j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirement; B is the preset total cost; The expression of the second constraint is as follows: Among them, l i，j For model M i Processing segmented computing tasks d j delay; L is the preset delay threshold; x i，j is a binary variable indicating whether to select the large language model M i Processing segmented computing tasks d j ; D is the computing task to be executed; M is the model in the large language model set that matches the task requirements.

6. The computing power scheduling method of the intelligent computing center according to claim 1, characterized in that: The step S3 comprises: Step S302: Optimizing the number of tokens required to call the target large language model.

7. A computing power scheduling device for an intelligent computing center, characterized in that: include: An acquisition module is used to acquire historical prediction data of each major language model in a large language model set, and to acquire historical real calculation data of historical calculation tasks corresponding to the historical prediction data in actual scenarios; Determining a quality score of each of the large language models according to an error between the historical prediction data and the historical real calculation data; wherein each of the large language models in the large language model set is deployed in an intelligent computing center; An analysis module is used to determine at least one model from the large language model set as a target large language model based on an objective function according to the quality score and the task requirements of the computing task to be executed by the intelligent computing center; each of the target large language models matches and executes at least one segmented computing task in the computing task to be executed, and the segmented computing tasks matched by all the target large language models are added together to obtain the computing task to be executed; the objective function is determined according to the quality score of the model matching the task requirements in the large language model set and the segmented computing task; An execution module is used to instruct the intelligent computing center to call the target large language model to execute the to-be-executed computing task according to the matching execution relationship between the target large language model and the segmented computing task.

8. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps in the computing power scheduling method of the intelligent computing center as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps in the computing power scheduling method of the intelligent computing center as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the computing power scheduling method of the intelligent computing center as described in any one of claims 1 to 6.

Images