A method and system for evaluating computing power and allocating resources for large language model reasoning

Abstract

This invention discloses a method and system for computational power assessment and resource allocation in large language model inference, relating to the field of large model computational power scheduling and optimization technology. It converts inference task features, hardware computational power indicators, and historical load data into a time-series characterization, which is then mapped to the semantic space of a pre-trained large language model through cross-modal semantic alignment. Natural language prompts are constructed and fused with the semantic characterization as input, and deep features are output through semantic understanding and relational inference. Preliminary computational power requirements and load prediction results are generated synchronously through parallel collaborative prediction, and joint optimization is performed using a multi-objective resource allocation objective function, outputting a computational power assessment value and the optimal resource allocation scheme. Simultaneously, a teacher-student knowledge distillation model is introduced, using the large model with frozen parameters as the teacher and the lightweight model as the student, achieving accuracy preservation and inference acceleration. This invention achieves accurate computational power requirement assessment, dynamic load adaptation, multi-objective balanced scheduling, and lightweight, efficient deployment.

Description

Technical Field

[0001] This invention relates to the field of large model computing power scheduling and optimization technology, and more specifically to a method and system for computing power evaluation and resource allocation for large language model inference. Background Technology

[0002] With the rapid growth in the scale of large language models, the demands on computing power, GPU memory, bandwidth, and node scheduling during the inference stage have increased dramatically, highlighting the significant shortcomings of traditional computing power assessment and resource allocation methods. Computing power assessment relies on fixed formulas and empirical thresholds, which cannot adapt to dynamic inference tasks and heterogeneous hardware, and has insufficient heterogeneous data fusion capabilities; there is a systematic deviation between hardware load and nominal computing power, and traditional methods directly use nominal indicators without localized correction, resulting in distorted assessments; resource configuration is static and cannot be dynamically adjusted according to real-time task characteristics, concurrency, and segment length, resulting in poor cross-scenario generalization ability; computing power scheduling is disconnected from inference tasks, which can easily lead to idle resources or computing power bottlenecks, affecting inference throughput and latency.

[0003] Therefore, how to propose a method and system for evaluating computing power and allocating resources for large language model reasoning, and overcome the shortcomings of existing technologies, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a method and system for computing power evaluation and resource allocation for large language model inference, achieving dynamic, accurate, and adaptive computing power management. To achieve the above objectives, the present invention adopts the following technical solution: A method for evaluating computational power and allocating resources for reasoning in large language models includes: Convert the feature data of large language model inference tasks, hardware computing power index data and historical load data into time-series characterization symbols; By cross-modal semantic alignment, the temporal characterization symbols in numerical form are mapped to the semantic space of a pre-trained large language model, generating semantic characterization symbols. Construct natural language prompts that include contextual information on computing power assessment and resource allocation, input the natural language prompts into a pre-trained large language model to obtain prompt characterization, and combine the prompt characterization with semantic characterization symbolism to form a fused input paragraph; The fused input paragraphs are fed into a pre-trained large language model for semantic understanding and relational reasoning to output deep features. Based on the deep features, a preliminary computing power demand prediction result and a load prediction result that corrects the hardware computing power index data are generated simultaneously through a parallel collaborative prediction mechanism. Construct a multi-objective optimization resource allocation objective function with the optimization objectives of maximizing computing power utilization, minimizing inference latency, maximizing throughput, and cost constraints. Using the load forecast results as a correction signal, the preliminary computing power demand forecast results are jointly optimized by combining them with a multi-objective optimization objective function, and the computing power evaluation value and the optimal resource allocation scheme are output.

[0005] Optionally, the objective function for resource allocation in the multi-objective optimization is: ; ; in, For inference delay, For resource costs, For computing power utilization, For throughput; , , , The weighting coefficients are and satisfy the following conditions: + + + =1; Constraints on the inequalities of video memory, bandwidth, and power consumption; The node topology and service level agreement (SLA) are constraints.

[0006] Optionally, it also includes: introducing a teacher-student knowledge distillation model, using a large language model with frozen parameters as the teacher model, and training a lightweight student model to achieve rapid inference and deployment: Using a pre-trained large language model with frozen parameters as the teacher model, high-dimensional soft labels and fused feature distributions are output. Build a lightweight Transformer student model that learns from the same fused input paragraph; The distillation joint loss is constructed by combining soft label loss, feature matching loss, and hard label true loss. ; in, For hard labeling, the actual loss For soft label loss, For feature matching loss, For feature comparison loss in the same stage; By updating the student model parameters through backpropagation, the model can be lightweighted and inference can be accelerated.

[0007] Optionally, the step of converting the large language model inference task feature data, hardware computing power index data, and historical load data into temporal characterization symbols includes: standardizing the inference task feature data, hardware computing power index data, and historical load data; dividing the standardized temporal data segments into multiple data blocks of fixed length; and encoding each data block into a characterization vector through a characterization layer to form a temporal characterization symbol.

[0008] Optionally, the cross-modal semantic alignment includes: introducing learnable text prototypes, and calculating attention outputs based on a multi-head cross-attention mechanism, using temporal characterization symbols as queries and text prototypes as keys and values; aggregating the attention outputs of all attention heads and generating semantic characterization symbols through linear projection; the text prototypes are initialized by selecting sub-vocabularies from the word characterization matrix of a pre-trained large language model.

[0009] Optionally, the natural language prompt includes at least one of the following: a computing resource description, used to describe the hardware computing power specifications, video memory, bandwidth and node topology; a task feature description, used to define the batch, paragraph length, inference accuracy and concurrent request volume of the large language model inference; and a load statistics description, used to convert the mean, peak and fluctuation characteristics of historical load data into text descriptions.

[0010] Optionally, the collaborative prediction mechanism includes: The deep features are flattened and converted into two-dimensional feature vectors. The two-dimensional feature vectors are then mapped to the computing power prediction space through an initial linear transformation layer to generate preliminary computing power demand prediction results. The two-dimensional feature vectors are then mapped to the load prediction space through a deep linear transformation layer to generate load prediction results. The initial linear transformation layer and the deep linear transformation layer use independent parameter sets.

[0011] Optionally, the joint optimization includes: concatenating the preliminary computing power demand prediction results and load prediction results to form a comprehensive feature symbol; flattening the comprehensive feature symbol and mapping it to the final computing power evaluation space through a linear transformation layer; and combining the multi-objective optimization objective function to output the computing power evaluation value and resource allocation scheme.

[0012] Optionally, model training optimization may also be included: constructing a joint loss function, which is composed of a weighted sum of computational power prediction loss and load correction loss; using local measured hardware load data as a supervision signal for load correction loss; minimizing the joint loss function through the gradient descent algorithm and updating all learnable parameters of the model.

[0013] Optionally, a computational power evaluation and resource allocation system for large language model inference includes: The data preprocessing module is used to preprocess and convert inference task characteristics, hardware computing power indicators, and historical load data into time-series characterization symbols. The cross-modal semantic alignment module is used to map temporal characterization symbols to the semantic space of a pre-trained large language model, generating semantic characterization symbols. The prompt processing module is used to construct natural language prompts for computing power assessment and resource allocation and encode them into prompt characteristics; The fusion input module is used to combine prompt descriptions and semantic descriptions to form fusion input paragraphs; The large language model inference module is used to perform semantic understanding and relational inference on the fused input paragraphs and output deep features; The collaborative prediction module is used to synchronously generate preliminary computing power demand prediction results and load prediction results based on deep features; The multi-objective optimization module is used to construct resource allocation objective functions with latency, cost, utilization, and throughput as the goals; The teacher-student distillation module is used to train lightweight student models for efficient inference deployment. The joint optimization module is used to correct the initial computing power demand forecast by combining the load forecast results with the multi-objective optimization objective function, and output the computing power assessment value and the optimal resource allocation scheme.

[0014] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method and system for computing power evaluation and resource allocation for large language model reasoning, which has the following beneficial effects: By bridging the modal gap between numerical data and semantic information through temporal characterization and cross-modal semantic alignment, collaborative prediction and load correction reduce computational power estimation bias, resulting in assessments that more closely reflect real-world loads. A multi-objective optimization function based on latency, cost, utilization, and throughput achieves balanced allocation of computational resources, avoiding waste or bottlenecks. Lightweight teacher-student distillation significantly compresses model size and reduces inference latency while preserving prediction accuracy, meeting the needs of edge and high-concurrency scenarios. Dynamically adjustable hardware metrics adapt to different model specifications, inference accuracy, and request fluctuations, ensuring long-term stable and reliable operation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a method for evaluating computing power and allocating resources for large language model reasoning, provided by the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This invention discloses a method for evaluating computing power and allocating resources for large language model reasoning, such as... Figure 1 As shown, it includes: Convert the feature data of large language model inference tasks, hardware computing power index data and historical load data into time-series characterization symbols; By cross-modal semantic alignment, the temporal characterization symbols in numerical form are mapped to the semantic space of a pre-trained large language model, generating semantic characterization symbols. Construct natural language prompts that include contextual information on computing power assessment and resource allocation, input the natural language prompts into a pre-trained large language model to obtain prompt characterization, and combine the prompt characterization with semantic characterization symbolism to form a fused input paragraph; The fused input paragraphs are fed into a pre-trained large language model for semantic understanding and relational reasoning to output deep features. Based on the deep features, a preliminary computing power demand prediction result and a load prediction result that corrects the hardware computing power index data are generated simultaneously through a parallel collaborative prediction mechanism. Construct a multi-objective optimization resource allocation objective function with the optimization objectives of maximizing computing power utilization, minimizing inference latency, maximizing throughput, and cost constraints. Using the load forecast results as a correction signal, the preliminary computing power demand forecast results are jointly optimized by combining them with a multi-objective optimization objective function, and the computing power evaluation value and the optimal resource allocation scheme are output.

[0019] Furthermore, the objective function for resource allocation in the multi-objective optimization is: ; ; in, For inference delay, For resource costs, For computing power utilization, For throughput; , , , The weighting coefficients are and satisfy the following conditions: + + + =1; Constraints on the inequalities of video memory, bandwidth, and power consumption; The node topology and service level agreement (SLA) are constraints.

[0020] Furthermore, it also includes: introducing a teacher-student knowledge distillation model, using a large language model with frozen parameters as the teacher model, and training a lightweight student model to achieve rapid inference and deployment: Using a pre-trained large language model with frozen parameters as the teacher model, high-dimensional soft labels and fused feature distributions are output. Build a lightweight Transformer student model that learns from the same fused input paragraph; The distillation joint loss is constructed by combining soft label loss, feature matching loss, and hard label true loss. ; in, For hard labeling, the actual loss For soft label loss, For feature matching loss, For feature comparison loss in the same stage; By updating the student model parameters through backpropagation, the model can be lightweighted and inference can be accelerated.

[0021] In a specific implementation, to address the inference efficiency bottleneck of a large language model computing power assessment and resource allocation system in high-concurrency, edge-deployment scenarios, and to overcome problems such as large differences in feature distribution between teacher and student models, insufficient knowledge transfer in intermediate layers, and attenuation of computing power prediction accuracy in traditional knowledge distillation, this invention introduces a teacher-student knowledge distillation model that integrates multi-stage feature fusion. This model uses a parameter-frozen large language model as the teacher and a lightweight Transformer structure as the student. While fully preserving the accuracy of computing power assessment, load prediction, and resource allocation decisions, it achieves model lightweighting, accelerated inference, and adaptive deployment on heterogeneous hardware. The specific implementation process and technical details are as follows: I. Teacher Model Construction and Knowledge Consolidation (1) Teacher model selection and initialization A pre-trained large language model is used as the teacher model. This model has completed the entire training process of computing power assessment, load prediction, and multi-objective resource allocation optimization, and has accurate numerical feature understanding, temporal relationship reasoning, and multi-objective decision-making capabilities. The input of the teacher model is a system-generated fusion input paragraph (cue characterization + semantic characterization), and the output includes computing power demand prediction, load prediction, multi-objective optimization decision results, and multi-level intermediate feature distribution.

[0022] (2) Weight freezing and knowledge stability

[0023] During the student model training phase, all pre-trained weights of the teacher model are frozen throughout the process and do not participate in gradient updates and backpropagation. This avoids knowledge drift in the teacher model due to student learning and ensures that the distribution of the output high-dimensional soft labels, intermediate layer features, and fused features remains stable and consistent, providing reliable and unified knowledge supervision signals for the student model.

[0024] (3) Teacher model knowledge output

[0025] The teacher model, based on the same fused input paragraph, outputs three types of core knowledge after multi-layer Transformer encoding and relational reasoning: High-dimensional soft tags: probability distributions for computing power assessment values, load prediction values, and resource allocation schemes, containing rich knowledge of uncertainty and related decision-making; Global fusion features: The overall features output by the last layer of the model, symbolizing the global correlation between computing power, load, and resources; Multi-stage intermediate features: The stage features output by each layer of the model correspond to the implicit knowledge of different stages such as temporal feature understanding, cross-modal semantic alignment, collaborative prediction, and multi-objective optimization.

[0026] II. Lightweight Student Model Construction

[0027] (1) Student model structure design

[0028] Construct a lightweight Transformer student model and perform multi-level compression while maintaining consistency with the input-output space of the teacher model: Reduce the number of Transformer layers, retaining only 1 / 3–1 / 2 of the depth of the teacher model; Reduce the number of attention heads and the dimension of hidden layers, thereby compressing the number of parameters and computational cost; Remove redundant fully connected layers and regularization branches to simplify the forward propagation path; It retains the core capabilities of cross-modal semantic understanding, temporal feature encoding, and collaborative prediction to ensure adaptability to computing power assessment and resource allocation tasks.

[0029] (2) Input alignment and task adaptation

[0030] The student model input is completely identical to the teacher model input, both being system-generated fused input segments (with prompts and semantic representations), ensuring that students learn teacher knowledge within the same information space and avoiding distillation failure due to input differences. Simultaneously, a linear output header adapted for computing power prediction, load correction, and resource allocation is added at the end of the student model, directly interfacing with downstream modules of the system.

[0031] III. Construction of a Multi-Stage Feature Fusion Framework

[0032] To address the issues of asymmetrical teacher-student model structures and difficulty in aligning intermediate-layer features, this invention constructs a symmetrical multi-stage feature fusion framework for the synchronous development of teacher-student models, enabling cross-stage knowledge transfer from shallow to deep features: (1) Definition of frame structure Both the teacher model T and the student model S have n feature output stages, corresponding to n fusion modules. ; The 0th fusion module: Single input entry; The first to n-1 fusion modules: dual input entry (inheriting the fusion features of the previous stage + the original features of the current stage); The nth fusion module has a single output port, while the remaining modules retain dual output ports.

[0033] (2) Feature fusion calculation rules

[0034] Student network feature fusion: ; Teacher network feature integration: ; Through this framework, the teacher-student model achieves a layer-by-layer fusion of shallow detailed features and deep semantic features, allowing students to absorb knowledge from teachers at different stages and significantly improving generalization and feature learning ability.

[0035] IV. Cross-stage Feature Fusion Attention Module

[0036] In a multi-stage feature fusion framework, a cross-stage feature fusion attention module is characterized to address the fusion failure caused by inconsistencies in feature size and channel number across different stages, thereby enhancing effective knowledge transfer. (1) Uniform feature dimensions Input features at different stages The initial fusion feature I is obtained by adjusting the convolution and normalization to the same size and number of channels and then adding them together.

[0037] (2) Enhanced dual-channel attention

[0038] A parallel structure of channel attention Ac + spatial attention As is used to process the fused features, respectively strengthening the key computing power indicators in the channel dimension and the temporal correlation information in the spatial dimension. The two outputs are added together to obtain the enhanced fused feature F, calculated as follows: ; (3) Feature output adaptation Two output features are generated through convolution and normalization, which are adapted to the input and output rules of the multi-stage fusion framework to achieve organic fusion of cross-stage features and effective knowledge enhancement.

[0039] V. Construction of the Joint Loss Function for Distillation

[0040] This invention, based on traditional soft labeling, hard labeling, and feature matching loss, incorporates a fusion feature comparison loss at the same stage to construct a four-layer composite distillation joint loss, achieving accurate knowledge transfer: Total loss formula ; Among them, the true loss of hard labels Cross-entropy loss, using real computing power consumption, measured load data, and optimal resource allocation schemes as supervisory signals, constrains student outputs to align with real business objectives, ensuring unbiased computing power assessment. Soft label loss. The KL divergence loss between the student model output and the high-dimensional soft labels of the teacher model allows the student to learn the teacher's probability distribution and decision preferences, while preserving fuzzy boundaries and relational knowledge. Feature matching loss... The MSE loss between globally fused features in the teacher-student model is used to align the global feature distribution and ensure consistency in the overall relationship between computing power, load, and resources. The loss is compared between fused features at the same stage. Perform triple MSE matching on the fusion features of teachers and students in stage i, aligning the distribution from three dimensions: original, channel compression, and spatial compression. The formula is: ,in: / Features of fusion in the original stage; / This is a channel compression feature; / λ represents the spatial compression feature, and λ is a weight hyperparameter used to balance the loss contribution of channel and spatial dimensions.

[0041] VI. Model Training, Parameter Update and Joint Optimization

[0042] (1) Joint monitoring signal

[0043] Using locally measured hardware load, actual computing power consumption, and business-labeled resource configuration schemes as monitoring, combined with the system's own computing power prediction loss. With load correction loss Construct the overall training objective: , To balance the coefficients and ensure that the distillation effect is consistent with the system's business objectives. Among these, Used to measure the difference between the estimated computing power value and the actual computing power consumption value; Used to measure the difference between the predicted load and the actual measured hardware load.

[0044] (2) Parameter update rules

[0045] by To optimize the objective, the AdamW optimizer is used to perform backpropagation, updating only the student model parameters while keeping the teacher weights frozen; training is performed iteratively in batches until the loss converges and the computational power assessment accuracy meets the target.

[0046] (3) Dynamic adaptation optimization

[0047] A heterogeneous hardware adaptive mechanism is incorporated into the training process, dynamically adjusting the loss weights based on different GPU / TPU, video memory, and bandwidth conditions to ensure that the student model can run stably on edge devices and cloud servers.

[0048] Furthermore, the step of converting the large language model inference task feature data, hardware computing power index data, and historical load data into temporal characterization symbols includes: standardizing the inference task feature data, hardware computing power index data, and historical load data; dividing the standardized temporal data segments into multiple fixed-length data blocks; and encoding each data block into a characterization vector through a characterization layer to form a temporal characterization symbol.

[0049] Furthermore, the cross-modal semantic alignment includes: introducing learnable text prototypes, and calculating attention outputs based on a multi-head cross-attention mechanism, using temporal characterization symbols as queries and text prototypes as keys and values; aggregating the attention outputs of all attention heads and generating semantic characterization symbols through linear projection; the text prototypes are initialized by selecting sub-vocabularies from the word characterization matrix of a pre-trained large language model.

[0050] Furthermore, the natural language prompts include at least one of the following: a computing resource description, used to describe the hardware computing power specifications, video memory, bandwidth and node topology; a task feature description, used to define the batch, paragraph length, inference accuracy and concurrent request volume of the large language model inference; and a load statistics description, used to convert the mean, peak and fluctuation characteristics of historical load data into text descriptions.

[0051] Furthermore, the collaborative prediction mechanism includes: The deep features are flattened and converted into two-dimensional feature vectors. The two-dimensional feature vectors are then mapped to the computing power prediction space through an initial linear transformation layer to generate preliminary computing power demand prediction results. The two-dimensional feature vectors are then mapped to the load prediction space through a deep linear transformation layer to generate load prediction results. The initial linear transformation layer and the deep linear transformation layer use independent parameter sets.

[0052] Furthermore, the joint optimization includes: concatenating the preliminary computing power demand prediction results and load prediction results to form a comprehensive feature symbol; flattening the comprehensive feature symbol and mapping it to the final computing power evaluation space through a linear transformation layer; and combining the multi-objective optimization objective function to output the computing power evaluation value and resource allocation scheme.

[0053] Furthermore, it also includes model training optimization: constructing a joint loss function, which is composed of a weighted sum of computational power prediction loss and load correction loss; using local measured hardware load data as a supervision signal for load correction loss; minimizing the joint loss function through the gradient descent algorithm and updating all learnable parameters of the model.

[0054] In a specific embodiment, a method for evaluating computing power and allocating resources for large language model inference includes the following steps: Step 1: Data Preprocessing → Temporal Characterization Symbolism (1) Data collection and standardization Task characteristics: batch size (1 / 2 / 4 / 8), paragraph length (512 / 1024 / 2048), concurrency (10~200); Hardware metrics: GPU utilization, video memory usage, bandwidth utilization, power consumption; Historical load: computing load and request fluctuation every 10 seconds; Standardization: Map all values ​​to the [0,1] interval to eliminate dimensional differences; (2) Blocking and Encoding The time-series data is divided into blocks of 64 with a fixed length. Each data block is encoded into a 512-dimensional characterization vector by the characterization layer, forming a temporal characterization symbol.

[0055] Step 2: Cross-modal semantic alignment → Semantic characterization

[0056] (1) Initialization of text prototype

[0057] The initial text prototypes are selected from the pre-trained large model word characterization matrix, which uses sub-vocabularies related to computing power, load, resources, and inference. (2) Multi-head cross-attention calculation The query is characterized by time sequence (Q), and the text prototype is characterized by key (K) and value (V). The output is aggregated after 8-head attention calculations and then linearly projected to generate a 768-dimensional semantic characterization, completing the numerical-to-semantic space mapping.

[0058] Step 3: Construct natural language prompts → Integrate input paragraphs

[0059] (1) Prompt text (enter directly)

[0060] Task: Dialogue inference using a 7B model, batch size 1-8, maximum paragraph length 2048, concurrency 10-200, FP16 precision; Load: average 65% over the past 7 days, peak 92%, fluctuation range ±25%. Please assess the computing power requirements and provide the optimal resource allocation.

[0061] (2) Coding and Fusion

[0062] Encode the prompt text into a 768-dimensional prompt representation; Combined with semantic characterization, it forms a 1536-dimensional fused input paragraph.

[0063] Step 4: Large Model Inference → Deep Feature Output

[0064] The fused input paragraphs are input into the pre-trained large model, and after 12 layers of Transformer encoding and relational reasoning, 1024-dimensional deep features are output (including global correlation information of computing power-load-resources).

[0065] Step 5: Parallel Collaborative Prediction → Preliminary Computing Power + Load Prediction

[0066] Deep features are flattened into two-dimensional vectors; Independent bilinear layer parallel prediction: Initial linear layer: Initial computing power requirement for output: 1.2 A100 sheets are needed for a single round of inference; Deep linear layer: Output load prediction: The average load for the next 5 minutes is 72%, and the peak load is 88%. It needs to be corrected by 0.93 times the nominal computing power of the hardware.

[0067] Step 6: Multi-objective optimization → Joint correction output

[0068] (1) Objective function: minF(x) = 0.3 T(x)+0.2 C(x)-0.3 U(x)-0.2 Q(x); Constraints: Memory usage ≤ 75GB, bandwidth utilization ≤ 85%, SLA latency ≤ 300ms; (2) Joint optimization The load prediction (×0.93) is used as the correction signal; Iterative optimization of initial computing power requirements; (3) Final output Computing power assessment: 3.2 A100 chips are needed for 200 concurrent connections; Optimal resource configuration: 2 servers each provide 2 A100s, for a total of 4; batch size is set to 4, with dynamic scaling up and down.

[0069] Step 7: Teacher-Student Knowledge Distillation → Lightweight Deployment

[0070] Teacher model: The original pre-trained large model with frozen parameters outputs soft labels, global features, and multi-stage features; Student model: Lightweight Transformer (6 layers, 4-head attention, 512-dimensional hidden layers, parameter count compressed to 1 / 6); Distillation loss ; Training results Accuracy is retained at 96.2%; inference speed is increased by 4.1 times; memory usage is reduced by 72%, and it can be deployed on edge GPUs.

[0071] In a specific implementation, a computational power evaluation and resource allocation system for large language model reasoning includes: The data preprocessing module is used to preprocess and convert inference task characteristics, hardware computing power indicators, and historical load data into time-series characterization symbols. The cross-modal semantic alignment module is used to map temporal characterization symbols to the semantic space of a pre-trained large language model, generating semantic characterization symbols. The prompt processing module is used to construct natural language prompts for computing power assessment and resource allocation and encode them into prompt characteristics; The fusion input module is used to combine prompt descriptions and semantic descriptions to form fusion input paragraphs; The large language model inference module is used to perform semantic understanding and relational inference on the fused input paragraphs and output deep features; The collaborative prediction module is used to synchronously generate preliminary computing power demand prediction results and load prediction results based on deep features; The multi-objective optimization module is used to construct resource allocation objective functions with latency, cost, utilization, and throughput as the goals; The teacher-student distillation module is used to train lightweight student models for efficient inference deployment. The joint optimization module is used to correct the initial computing power demand forecast by combining the load forecast results with the multi-objective optimization objective function, and output the computing power assessment value and the optimal resource allocation scheme.

[0072] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0073] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating computing power and allocating resources for reasoning in large language models, characterized in that, include: Convert the feature data of large language model inference tasks, hardware computing power index data and historical load data into time-series characterization symbols; By cross-modal semantic alignment, the temporal characterization symbols in numerical form are mapped to the semantic space of a pre-trained large language model, generating semantic characterization symbols. Construct natural language prompts that include contextual information on computing power assessment and resource allocation, input the natural language prompts into a pre-trained large language model to obtain prompt characterization, and combine the prompt characterization with semantic characterization symbolism to form a fused input paragraph; The fused input paragraphs are fed into a pre-trained large language model for semantic understanding and relational reasoning to output deep features; based on Deep features are used to simultaneously generate preliminary computing power demand prediction results and load prediction results that correct hardware computing power index data through a parallel collaborative prediction mechanism. Construct a multi-objective optimization resource allocation objective function with the optimization objectives of maximizing computing power utilization, minimizing inference latency, maximizing throughput, and cost constraints. Using the load forecast results as a correction signal, the preliminary computing power demand forecast results are jointly optimized by combining them with a multi-objective optimization objective function, and the computing power evaluation value and the optimal resource allocation scheme are output.

2. The method for computational power evaluation and resource allocation for large language model reasoning according to claim 1, characterized in that, The objective function for resource allocation in the multi-objective optimization is: ； ； in, For inference delay, For resource costs, For computing power utilization, For throughput; , , , The weighting coefficients are satisfied. + + + =1; Constraints on the inequalities of video memory, bandwidth, and power consumption; The node topology and service level agreement (SLA) are constraints.

3. The method for computational power evaluation and resource allocation for large language model reasoning according to claim 1, characterized in that, Also includes: A teacher-student knowledge distillation model is introduced, using a large language model with frozen parameters as the teacher model to train a lightweight student model for rapid inference and deployment. Using a pre-trained large language model with frozen parameters as the teacher model, high-dimensional soft labels and fused feature distributions are output. Build a lightweight Transformer student model that learns from the same fused input paragraph; The distillation joint loss is constructed by combining soft label loss, feature matching loss, and hard label true loss: ； in, For hard labeling, the actual loss For soft label loss, For feature matching loss, For feature comparison loss in the same stage; By updating the student model parameters through backpropagation, the model can be lightweighted and inference can be accelerated.

4. The method for computational power evaluation and resource allocation for large language model reasoning according to claim 1, characterized in that, The process of converting large language model inference task feature data, hardware computing power index data, and historical load data into temporal characterization symbols includes: standardizing the inference task feature data, hardware computing power index data, and historical load data; dividing the standardized temporal data segments into multiple fixed-length data blocks; and encoding each data block into a characterization vector through a characterization layer to form a temporal characterization symbol.

5. The method for computational power evaluation and resource allocation for large language model reasoning according to claim 1, characterized in that, The cross-modal semantic alignment includes: introducing learnable text prototypes, and using a multi-head cross-attention mechanism, using temporal characterization symbols as queries and text prototypes as keys and values ​​to calculate attention outputs; aggregating the attention outputs of all attention heads and generating semantic characterization symbols through linear projection; the text prototypes are initialized by selecting sub-vocabularies from the word characterization matrix of a pre-trained large language model.

6. The method for computational power evaluation and resource allocation for large language model reasoning according to claim 1, characterized in that, The natural language prompts include at least one of the following: a computing resource description, used to describe the hardware computing power specifications, video memory, bandwidth and node topology; a task feature description, used to define the batch, paragraph length, inference accuracy and concurrent request volume of the large language model inference; and a load statistics description, used to convert the mean, peak and fluctuation characteristics of historical load data into text descriptions.

7. The method for computational power evaluation and resource allocation for large language model reasoning according to claim 1, characterized in that, The collaborative prediction mechanism includes: Flatten the depth features to convert them into two-dimensional feature vectors; The initial linear transformation layer maps the two-dimensional feature vectors to the computing power prediction space, generating preliminary computing power demand prediction results. The two-dimensional feature vectors are mapped to the load prediction space through a deep linear transformation layer to generate load prediction results.

8. The method for computational power evaluation and resource allocation for large language model reasoning according to claim 1, characterized in that, The joint optimization includes: concatenating the preliminary computing power demand prediction results and load prediction results to form a comprehensive feature symbol; flattening the comprehensive feature symbol and mapping it to the final computing power evaluation space through a linear transformation layer; and combining the multi-objective optimization objective function to output the computing power evaluation value and resource allocation scheme.

9. The method for computational power evaluation and resource allocation for large language model reasoning according to claim 1, characterized in that, It also includes model training optimization: constructing a joint loss function, which is composed of a weighted sum of computational power prediction loss and load correction loss; using local measured hardware load data as a supervision signal for load correction loss; minimizing the joint loss function through the gradient descent algorithm and updating all learnable parameters of the model.

10. A computational power evaluation and resource allocation system for large language model reasoning, characterized in that, include: The data preprocessing module is used to preprocess and convert inference task characteristics, hardware computing power indicators, and historical load data into time-series characterization symbols. The cross-modal semantic alignment module is used to map temporal characterization symbols to the semantic space of a pre-trained large language model, generating semantic characterization symbols. The prompt processing module is used to construct natural language prompts for computing power assessment and resource allocation and encode them into prompt characteristics; The fusion input module is used to combine prompt descriptions and semantic descriptions to form fusion input paragraphs; The large language model inference module is used to perform semantic understanding and relational inference on the fused input paragraphs and output deep features; The collaborative prediction module is used to synchronously generate preliminary computing power demand prediction results and load prediction results based on deep features; The multi-objective optimization module is used to construct resource allocation objective functions with latency, cost, utilization, and throughput as the objectives. The teacher-student distillation module is used to train lightweight student models for inference deployment. The joint optimization module is used to correct the initial computing power demand forecast by combining the load forecast results with the multi-objective optimization objective function, and output the computing power assessment value and the optimal resource allocation scheme.

Images