Multi-scene large model lightweight deployment and dynamic adaptation method

Abstract

This invention belongs to the field of large model deployment technology. It discloses a lightweight deployment and dynamic adaptation method for large models in multiple scenarios, including: constructing a policy network with hardware parameters as input and policy combinations as output; collecting sampling points in the hardware parameter space and performing performance tests to obtain the optimal policy label; detecting mutations in the optimal policy between adjacent sampling points and repeatedly supplementing sampling at the boundary to form a training sample set; using the local sample density of each sampling point as weight, interpolating between the accurate fitting target and the security policy regularization target to construct a loss function to train the policy network, obtaining the deployment policy network; inputting the hardware parameters of the hardware node to be deployed into the deployment policy network, filtering out policy combinations that violate constraints, and outputting the highest probability as the deployment policy. By adaptively sampling mutation boundaries, the switching rules of the optimal policy under different hardware scenarios are accurately characterized, solving the problem of insufficient training data near the policy boundary.

Description

Technical Field

[0001] This invention relates to the field of large model deployment technology, and more specifically, to a method for lightweight deployment and dynamic adaptation of large models in multiple scenarios. Background Technology

[0002] With the rapid development of large language model technology, the demand for practical deployment of large models continues to grow. However, the hardware conditions in real-world deployment environments are highly diverse, ranging from cloud servers equipped with high-end GPUs to edge inference devices with limited computing power, with a wide range of resource specifications across different hardware nodes. To ensure that large models run stably on specific hardware and achieve acceptable inference performance, a series of lightweight processing steps, such as quantization, pruning, and batch configuration, are typically required to tailor the model to the resource conditions of that hardware.

[0003] However, the combinations of lightweight strategies adapted to different hardware environments vary significantly. An optimal strategy for one type of hardware may lead to resource overruns or severe performance degradation on another. Existing methods typically rely on engineers' experience to formulate deployment configurations for specific hardware, making it difficult to systematically cover the diverse combinations of hardware environments in real-world scenarios. These methods suffer from low automation and poor reusability. Another approach is to build strategy selection models using a data-driven method. This approach also has problems. For example, the optimal deployment strategy does not change smoothly with hardware conditions but rather abruptly occurs within specific hardware parameter ranges. Conventional sampling methods struggle to accumulate sufficient training data in critical areas of strategy switching, resulting in weak model judgment capabilities in these regions. Furthermore, for hardware scenarios not fully covered during training, the model lacks effective safety mechanisms, posing a risk of outputting deployment strategies exceeding the target hardware's resource capacity, leading to inference service unavailability. Summary of the Invention

[0004] To overcome the aforementioned problems of existing technologies, this invention proposes a method for lightweight deployment and dynamic adaptation of large models across multiple scenarios to address these issues.

[0005] This invention provides the following technical solution: Methods for lightweight deployment and dynamic adaptation of large models across multiple scenarios include: A policy network is constructed using hardware parameters as input and policy combinations as output. Multiple sampling points are collected in the hardware parameter space. Performance tests are performed on each sampling point to obtain the optimal policy label for each sampling point. Mutations in the optimal policy combination are detected between adjacent sampling points. Supplementary sampling is performed between adjacent sampling points where mutations occur, and detection is repeated to form a training sample set. Calculate the local sample density of each sample point in the training sample set, and use the local sample density as weight to construct a loss function for each sample point by interpolation between the exact fitting target and the security policy regularization target. Based on the training sample set, the policy network is trained using the loss function to obtain the deployment policy network; The hardware parameters of the hardware node to be deployed are input into the deployment policy network. Policy combinations that violate hardware resource constraints are filtered out. The policy combination with the highest output probability is selected from the policy combinations that meet the constraints and used as the deployment policy for the hardware node to be deployed.

[0006] Preferably, the hardware parameters include available video memory, system memory, computing power, memory bandwidth, and inference backend type; the strategy combination is obtained by enumerating and combining the values ​​of the strategy variables, and the strategy variables include quantization precision, retention layer ratio, batch size, and caching strategy.

[0007] Preferably, the construction of the policy network includes: constructing a neural network that takes hardware parameters as input and the selection probability of each policy combination as output, as the policy network.

[0008] Preferably, the step of collecting multiple sampling points in the hardware parameter space and performing performance tests on each sampling point to obtain the optimal strategy label for each sampling point includes: The initial set of sampling points is acquired in the hardware parameter space using the Latin hypercube method; For each sampling point in the initial sampling point set, the model configuration corresponding to each strategy combination is loaded sequentially, a preset number of inference requests are executed, and the measured memory usage, inference latency, and inference throughput are recorded. The performance score of each strategy combination is calculated based on the measured memory usage, inference latency, and inference throughput. The strategy combination with the highest performance score is marked as the optimal strategy label for that sampling point.

[0009] Preferably, the step of detecting mutations in the optimal strategy combination between adjacent sampling points, supplementing sampling and iteratively detecting between adjacent sampling points where mutations occur, and forming a training sample set includes: The current set of sampling points is grouped according to the inference backend type. A nearest neighbor graph is constructed within each group. For each edge in the nearest neighbor graph, it is determined whether the optimal strategy labels of the sampling points at both ends of the edge are different. For edges with different optimal policy labels, insert supplementary sampling points at the mean of the numerical hardware parameters at both ends of the edge, perform performance testing on the supplementary sampling points, and label them with the optimal policy. Add the supplementary sampling point to the current sampling point set, and repeatedly perform judgment and insertion on the newly added edge containing the supplementary sampling point until the parameter distance between all adjacent sampling points is lower than the preset resolution threshold or the total number of samples reaches the preset upper limit. The training sample set is obtained by summing up the sample pairs formed by each sample point in the final sample point set and the corresponding optimal policy label.

[0010] Preferably, calculating the local sample density of each sample point in the training sample set includes: For each sample point in the training sample set, find a preset number of nearest neighbors among the remaining sample points of the same inference backend type, and calculate the average distance of the nearest neighbors; The average nearest neighbor distance of each sample point is normalized by the global average nearest neighbor distance of all sample points in the training sample set. The normalization result is then subjected to a monotonically decreasing transformation to obtain the local sample density of each sample point.

[0011] Preferably, the step of constructing a loss function by interpolating between the accurate fitting target and the security policy regularization target for each sample point, using the local sample density as weight, includes: For each sample point in the training sample set, the negative logarithm of the prediction probability of the optimal policy label of the policy network for that sample point is taken as the exact fitting target. For each sample point, a security policy distribution is determined, and the divergence between the output distribution of the policy network for that sample point and the security policy distribution is used as the security policy regularization target. The precise fitting objective is weighted by the local sample density of the sample point, and the security policy regularization objective is weighted by subtracting the local sample density of the sample point. The two weighted results are added together to obtain the loss term corresponding to the sample point. The loss function is obtained by averaging the loss terms of all sample points in the training sample set.

[0012] Preferably, determining the security policy distribution for each sample point includes: For each sample point, select strategy combinations from the set of strategy combinations whose measured video memory usage does not exceed the upper limit of available video memory indicated by the hardware parameters of that sample point, and obtain the legal strategy subset of that sample point. Within the subset of legitimate strategies, the strategy combination with the lowest measured memory usage is selected, and a probability distribution with this strategy combination as the only non-zero term is constructed as the security strategy distribution for this sample point.

[0013] Preferably, the filtering strategy combination that violates hardware resource constraints, selecting the strategy combination with the highest output probability from those that satisfy the constraints, includes: From the set of strategy combinations, select strategy combinations whose measured video memory usage does not exceed the upper limit of available video memory of the hardware node to be deployed, to obtain a subset of legal strategy combinations; From the subset of legal strategy combinations, the strategy combination with the highest selection probability output by the deployment strategy network is selected as the deployment strategy.

[0014] This invention provides a lightweight deployment and dynamic adaptation method for large models in multiple scenarios, which has the following beneficial effects: This invention performs adaptive mutation boundary sampling in the hardware parameter space, detects mutations in the optimal strategy combination between adjacent sampling points, and repeatedly supplements sampling near the boundary, so that the training sample set forms dense coverage in the strategy switching boundary region, accurately characterizing the switching pattern of the optimal deployment strategy under different hardware scenarios. It solves the problem that uniform sampling method has insufficient training data near the strategy boundary, resulting in low accuracy of the strategy selection model in the key decision region.

[0015] This invention constructs a loss function by interpolating between the accurate fitting target and the security policy regularization target, using local sample density as weights. This enables the policy network to accurately learn the optimal policy in hardware scenarios with sufficient training data coverage, and output a legal policy that tends to be conservative in resource consumption in hardware scenarios with sparse training data. Combined with the memory constraint hard filtering mechanism in the inference stage, a dual security guarantee is formed by soft constraints in the training stage and hard constraints in the inference stage. This solves the problem that data-driven methods may output deployment policies that exceed hardware resource limits in hardware scenarios where training is not fully covered, leading to the unavailability of inference services. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the lightweight deployment and dynamic adaptation method for large models in multiple scenarios according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0018] Please see Figure 1 In this embodiment, the method for lightweight deployment and dynamic adaptation of large models across multiple scenarios includes: Step S1: Construct a policy network using hardware parameters as input and policy combinations as output; The hardware parameters include available video memory, system memory, computing power, memory bandwidth, and inference backend type; the strategy combination is obtained by enumerating and combining the values ​​of the strategy variables, which include quantization precision, retention layer ratio, batch size, and caching strategy.

[0019] In this embodiment, hardware parameters constitute a context vector describing the capabilities of the target hardware node. For example, the hardware parameters of a hardware node can be represented as: 8GB of available video memory, 16GB of system memory, 15 TFLOPS of computing power, 300GB / s of memory bandwidth, and CUDA (Compute Unified Device Architecture) inference backend type. Each policy variable has several discrete values. For example, quantization precision can be INT4, INT8, or FP16; the retention layer ratio can be 0.5, 0.75, or 1.0; the batch size can be 1, 2, or 4; and the caching policy can be enabled or disabled. By fully enumerating and combining all the above values, a set of policy combinations is obtained. Each element in the set represents a complete deployment scheme. The construction of the policy network includes: constructing a neural network that takes hardware parameters as input and the selection probability of each policy combination as output, as the policy network.

[0020] In this embodiment, the inference backend type is treated as a categorical variable and is processed using one-hot encoding before being input into the network. For example, the three inference backend types are encoded as [1,0,0], [0,1,0], and [0,0,1], respectively. These are concatenated with the other four numerical hardware parameters to form the input vector. The dimension of the input layer is determined according to the number of inference backend types. The policy network adopts a multi-layer fully connected network structure. The intermediate layers use non-linear activation functions to extract and map features from the input hardware parameter combinations, enabling the network to learn the non-linear correspondence between hardware parameters and the optimal policy. The number of nodes in the output layer is consistent with the number of elements in the policy combination set. Each output node corresponds to one policy combination. The output layer is processed by the Softmax function to ensure that the probability values ​​of all output nodes are non-negative and their sum is 1, thus forming a complete probability distribution output for the policy combination set. The number of intermediate layers and the number of nodes in each layer can be set according to the size of the policy combination set. For example, when the total number of policy combinations is dozens, a structure with two hidden layers and 64 to 128 nodes per layer can be used.

[0021] Step S2: Collect multiple sampling points in the hardware parameter space, perform performance tests on each sampling point to obtain the optimal policy label for each sampling point, detect mutations in the optimal policy combination between adjacent sampling points, supplement sampling between adjacent sampling points where mutations occur and cyclically detect to form a training sample set. The step of collecting multiple sampling points in the hardware parameter space and performing performance tests on each sampling point to obtain the optimal strategy label for each sampling point includes: The initial set of sampling points is acquired in the hardware parameter space using the Latin hypercube method; For each sampling point in the initial sampling point set, the model configuration corresponding to each strategy combination is loaded sequentially, a preset number of inference requests are executed, and the measured memory usage, inference latency, and inference throughput are recorded. The performance score of each strategy combination is calculated based on the measured memory usage, inference latency, and inference throughput. The strategy combination with the highest performance score is marked as the optimal strategy label for that sampling point.

[0022] In this embodiment, Latin hypercube sampling divides the value range of each numerical hardware parameter into several intervals, ensuring that each interval is covered exactly once in the sampling. Compared with pure random sampling, it can cover the hardware parameter space more evenly with fewer sampling points. For example, the available video memory parameter is set to have a value range of 4GB to 80GB. After dividing it into several intervals, a specific value is randomly selected from each interval to participate in the formation of sampling points. The other numerical hardware parameters are processed in the same way. For categorical parameter inference backend types, the number of sampling points is evenly distributed in each category, and combined with the numerical parameters to form the initial sampling point set.

[0023] For each sampling point in the initial sampling point set, each strategy combination in the strategy combination set is deployed to the corresponding hardware configuration test environment in sequence. A preset number of inference requests are executed, and the measured memory usage, inference latency, and inference throughput are recorded. The hardware parameters of the supplementary sampling points are the average of the numerical parameters of the two sampling points. The corresponding test environment is implemented by limiting the available resources of the real hardware through software. For example, the upper limit of available memory is set to the value specified by the supplementary sampling point through the memory limit interface, thereby simulating the actual operating conditions under the hardware parameter configuration and completing the performance test.

[0024] These three metrics reflect different performance dimensions of the deployment strategy. A comprehensive performance score needs to be calculated. For example, each metric can be normalized individually, and then a weighted sum can be taken, with the normalized values ​​for memory usage, inference latency, and inference throughput inverted and then weighted. The weights can be set according to the actual application's emphasis on resource efficiency and inference speed. The strategy combination with the highest performance score is the optimal deployment scheme for that sampling point under the current hardware parameters, and it is recorded as the optimal strategy label for that sampling point.

[0025] The step of detecting mutations in the optimal strategy combination between adjacent sampling points, supplementing sampling and iteratively detecting between adjacent sampling points where mutations occur, and forming a training sample set includes: The current set of sampling points is grouped according to the inference backend type. A nearest neighbor graph is constructed within each group. For each edge in the nearest neighbor graph, it is determined whether the optimal strategy labels of the sampling points at both ends of the edge are different. For edges with different optimal policy labels, insert supplementary sampling points at the mean of the numerical hardware parameters at both ends of the edge, perform performance testing on the supplementary sampling points, and label them with the optimal policy. Add the supplementary sampling point to the current sampling point set, and repeatedly perform judgment and insertion on the newly added edge containing the supplementary sampling point until the parameter distance between all adjacent sampling points is lower than the preset resolution threshold or the total number of samples reaches the preset upper limit. The training sample set is obtained by summing up the sample pairs formed by each sample point in the final sample point set and the corresponding optimal policy label.

[0026] In this embodiment, for sampling points of the same inference backend type in the current sampling point set, a nearest neighbor graph is constructed based on the numerical hardware parameters of each sampling point. Before calculating the distance between sampling points, each numerical hardware parameter is normalized by dividing it by its maximum value in the sampling point set, ensuring that the parameter values ​​of each dimension are of the same order of magnitude before calculating the Euclidean distance. For each sampling point, the K nearest neighbors by Euclidean distance are found and connected. The value of K can be set according to the total number of sampling points; when there are fewer sampling points, K can be taken as a smaller value to avoid connecting edges that cross excessively large distances. For each edge in the nearest neighbor graph, the optimal policy labels of the sampling points at both ends of the edge are compared. If the labels at both ends are different, it indicates that there is a switch in the optimal policy between these two hardware parameters, i.e., a policy mutation. For the edge where a mutation occurs, a supplementary sampling point is inserted at the mean of the numerical hardware parameters of the sampling points at both ends of the edge. For example, if the available video memory of the sampling points at both ends is 8GB and 16GB respectively, then the available video memory of the supplementary sampling point is 12GB. The remaining numerical parameters are similarly taken as the mean, and the inference backend type remains consistent with the group it belongs to.

[0027] The same performance testing process as the initial sampling points is performed on the supplementary sampling points. After obtaining their optimal policy labels, they are added to the current sampling point set. Mutation detection and insertion operations are repeatedly performed on newly added edges containing the supplementary sampling points. This process iterates until the numerical hardware parameter Euclidean distance between all adjacent sampling points is lower than a preset resolution threshold or the total number of samples reaches a preset upper limit. The preset resolution threshold is the Euclidean distance scalar in the normalized hardware parameter space. The smaller the threshold, the more accurate the boundary localization. It can be set according to the accuracy requirements of the policy switching boundary in the actual deployment scenario. After the above process, a dense distribution of sampling points will form near the policy switching boundary, while the sampling points in the policy-stable area will be relatively sparse. Finally, all sampling points and their corresponding optimal policy labels are combined to form sample pairs, resulting in the training sample set.

[0028] Step S3: Calculate the local sample density of each sample point in the training sample set, and construct a loss function by interpolating between the accurate fitting target and the security policy regularization target for each sample point using the local sample density as the weight. The calculation of the local sample density of each sample point in the training sample set includes: For each sample point in the training sample set, find a preset number of nearest neighbors among the remaining sample points of the same inference backend type, and calculate the average distance of the nearest neighbors; The average nearest neighbor distance of each sample point is normalized by the global average nearest neighbor distance of all sample points in the training sample set. The normalization result is then subjected to a monotonically decreasing transformation to obtain the local sample density of each sample point.

[0029] In this embodiment, for each sample point in the training sample set, a preset number of nearest neighbors are found among the remaining sample points of the same type in the inference backend, for example, K=5. The average Euclidean distance from the sample point to its 5 nearest neighbors is calculated. Intuitively, if a sample point is located near the policy switching boundary, the surrounding sample points are densely distributed after repeated replenishment, resulting in a smaller average distance to the nearest neighbor; if a sample point is located in a policy stable region, the surrounding sample points are sparse, resulting in a larger average distance to the nearest neighbor. Using the global average nearest neighbor distance of all sample points in the training sample set as a benchmark, the average nearest neighbor distance of each sample point is normalized by dividing it by the global mean, and then a monotonically decreasing transformation is performed to obtain the local sample density. The monotonically decreasing transformation selects a function with a value range of 0 to 1, such as a negative exponential function with the natural constant as the base. The normalized average nearest neighbor distance is given a negative exponent, so that the local sample density value is always within the range of 0 to 1, ensuring that the interpolation weights of the accurate fitting target and the security policy regularization target in the subsequent loss function are non-negative and their sum is 1. After this transformation, the local sample density of sample points in dense areas near the boundary is high, while the local sample density of sample points in sparse areas with stable strategies is low.

[0030] The step of constructing a loss function by interpolating between the accurate fitting target and the security policy regularization target for each sample point, using the local sample density as weight, includes: For each sample point in the training sample set, the negative logarithm of the prediction probability of the optimal policy label of the policy network for that sample point is taken as the exact fitting target. For each sample point, a security policy distribution is determined, and the divergence between the output distribution of the policy network for that sample point and the security policy distribution is used as the security policy regularization target. The precise fitting objective is weighted by the local sample density of the sample point, and the security policy regularization objective is weighted by subtracting the local sample density of the sample point. The two weighted results are added together to obtain the loss term corresponding to the sample point. The loss function is obtained by averaging the loss terms of all sample points in the training sample set.

[0031] In this embodiment, the accurate fitting target is achieved using cross-entropy loss, which is calculated by taking the negative of the logarithm of the probability predicted by the policy network for the optimal policy label of the sample point. A smaller cross-entropy value indicates a higher probability of the network predicting the correct label, resulting in a more accurate fit. The cross-entropy between the probability distribution currently output by the policy network and the security policy distribution is used as the security policy regularization target. Specifically, the cross-entropy loss is calculated using the security policy distribution as the target label and the network output probability distribution as the prediction distribution. A smaller cross-entropy value indicates that the network output is closer to the security policy distribution. Since the security policy distribution is a single-point distribution, the cross-entropy in this case is equivalent to taking the negative of the logarithm of the probability corresponding to the security policy combination in the network output, making the calculation simple and numerically stable.

[0032] Determining the security policy distribution for each sample point includes: For each sample point, select strategy combinations from the set of strategy combinations whose measured video memory usage does not exceed the upper limit of available video memory indicated by the hardware parameters of that sample point, and obtain the legal strategy subset of that sample point. Within the subset of legitimate strategies, the strategy combination with the lowest measured memory usage is selected, and a probability distribution with this strategy combination as the only non-zero term is constructed as the security strategy distribution for this sample point.

[0033] In this embodiment, the legitimate policy subset excludes policy combinations whose measured memory usage exceeds the available memory limit of the hardware node, ensuring that candidate solutions in the security policy distribution can run normally on this hardware without memory overflow. The policy combination with the lowest measured memory usage is selected from the legitimate policy subset as the security policy, and the probability of this policy combination is set to 1, while the probabilities of the remaining policy combinations are set to 0, thus forming the security policy distribution. The underlying logic of this design is that, for hardware scenarios with sparse samples, the training data is insufficient to support the network in making accurate policy judgments. In this case, guiding the network to tend to output the most conservative legitimate policy in terms of resource usage ensures that even in hardware environments that have not been sufficiently trained, the policies output by the network will not lead to memory overflow.

[0034] The local sample density of each sample point is used to interpolate and weight the exact fitting target and the safety policy regularization target. Near the policy switching boundary, after repeated supplementary sampling, the sample points are dense, resulting in high local sample density and a large weight for the exact fitting target. This drives the network to finely learn the switching patterns of the optimal policy under different hardware parameters in these regions. In policy stable regions, no supplementary sampling is triggered, resulting in sparse sample points and low local sample density. This leads to a large weight for the safety policy regularization target, driving the network to output a more conservative policy in hardware scenarios with insufficient data coverage, reducing the risk of resource overruns due to insufficient generalization. The two weighted results are added together to obtain the loss term for that sample point. The average of the loss terms for all sample points in the training sample set is taken to obtain the final loss function. After this training, the network develops accurate policy discrimination capabilities near the policy boundary and maintains a conservative, safe output tendency in hardware scenarios with insufficient data coverage. These two behaviors are automatically balanced by local sample density without manual intervention.

[0035] Step S4: Based on the training sample set, train the policy network with the loss function to obtain the deployment policy network; Using the hardware parameter input vectors of each sample point in the training sample set as the input to the neural network, and taking the aforementioned loss function as the optimization objective, the backpropagation algorithm is used to iteratively update the parameters of the fully connected layers of the policy network until the loss function converges, thus obtaining the deployment policy network. Since the loss function drives accurate fitting in dense regions and drives the network output to lean towards a conservative policy in sparse regions, the trained deployment policy network can provide an accurate policy probability distribution for different hardware parameter inputs in hardware scenarios with sufficient training data coverage, and a probability distribution biased towards a conservative policy in hardware scenarios with sparse training data.

[0036] Step S5: Input the hardware parameters of the hardware node to be deployed into the deployment policy network, filter policy combinations that violate hardware resource constraints, and select the policy combination with the highest output probability from the policy combinations that meet the constraints as the deployment policy for the hardware node to be deployed.

[0037] The filtering strategy combination that violates hardware resource constraints selects the strategy combination with the highest output probability from those that satisfy the constraints, including: From the set of strategy combinations, select strategy combinations whose measured video memory usage does not exceed the upper limit of available video memory of the hardware node to be deployed, to obtain a subset of legal strategy combinations; From the subset of legal strategy combinations, the strategy combination with the highest selection probability output by the deployment strategy network is selected as the deployment strategy.

[0038] The filtering of policy combinations that violate hardware resource constraints, and the selection of the policy combination with the highest output probability from the policy combination that satisfies the constraints, includes: filtering policy combinations from the policy combination set whose measured video memory usage does not exceed the upper limit of the available video memory of the hardware node to be deployed, to obtain a subset of legal policy combinations; and selecting the policy combination with the highest selection probability output by the deployment policy network from the subset of legal policy combinations as the deployment policy.

[0039] In this embodiment, during the actual deployment phase, hardware parameters of the hardware node to be deployed are first collected from the system interface. The inference backend type is processed using the same encoding method as in the training phase, and then concatenated with the remaining numerical hardware parameters to form an input vector matching the dimension of the policy network's input layer. This input vector is then fed into the deployment policy network. After forward inference, the network outputs the corresponding selection probability for all policy combinations in the policy combination set. Subsequently, the policy combination set is filtered for validity, removing policy combinations whose measured memory usage exceeds the available memory limit of the hardware node to be deployed, resulting in a subset of valid policy combinations. The measured memory usage used for this filtering is derived from the measured data of each policy combination recorded during the performance testing phase, maintaining consistency with the training phase. Within the subset of valid policy combinations, the policy combination with the highest output probability from the deployment policy network is selected as the final deployment policy, completing the configuration of the automated, lightweight deployment scheme for the hardware node to be deployed.

[0040] As shown above, this embodiment constructs a policy neural network with hardware parameters as input and policy combination selection probability as output, uses adaptive mutation boundary sampling to form a training sample set, and drives network training with a density-weighted interpolation loss function, thus achieving accurate characterization of policy switching boundaries. Specifically, the safety policy regularization in the loss function guides the network towards conservative policies in sparse hardware scenarios through soft constraints during the training phase, while the legality filtering in the inference phase effectively ensures that the final output policy does not exceed the available GPU memory limit through hard constraints. These two mechanisms, acting on the training and inference phases respectively, together form a dual guarantee mechanism for deployment security, thereby achieving efficient, secure, automated, and lightweight deployment of large models in diverse hardware environments.

[0041] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0042] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0043] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight deployment and dynamic adaptation method for large models in multiple scenarios, characterized in that, include: A policy network is constructed using hardware parameters as input and policy combinations as output. Multiple sampling points are collected in the hardware parameter space. Performance tests are performed on each sampling point to obtain the optimal policy label for each sampling point. Mutations in the optimal policy combination are detected between adjacent sampling points. Supplementary sampling is performed between adjacent sampling points where mutations occur, and detection is repeated to form a training sample set. Calculate the local sample density of each sample point in the training sample set, and use the local sample density as weight to construct a loss function for each sample point by interpolation between the exact fitting target and the security policy regularization target. Based on the training sample set, the policy network is trained using the loss function to obtain the deployment policy network; The hardware parameters of the hardware node to be deployed are input into the deployment policy network. Policy combinations that violate hardware resource constraints are filtered out. The policy combination with the highest output probability is selected from the policy combinations that meet the constraints and used as the deployment policy for the hardware node to be deployed.

2. The method for lightweight deployment and dynamic adaptation of large models in multiple scenarios according to claim 1, characterized in that, The hardware parameters include available video memory, system memory, computing power, memory bandwidth, and inference backend type; the strategy combination is obtained by enumerating and combining the values ​​of the strategy variables, which include quantization precision, retention layer ratio, batch size, and caching strategy.

3. The method for lightweight deployment and dynamic adaptation of large models in multiple scenarios according to claim 2, characterized in that, The construction of the policy network includes: constructing a neural network that takes hardware parameters as input and the selection probability of each policy combination as output, as the policy network.

4. The method for lightweight deployment and dynamic adaptation of large models in multiple scenarios according to claim 3, characterized in that, The step of collecting multiple sampling points in the hardware parameter space and performing performance tests on each sampling point to obtain the optimal strategy label for each sampling point includes: The initial set of sampling points is acquired in the hardware parameter space using the Latin hypercube method; For each sampling point in the initial sampling point set, the model configuration corresponding to each strategy combination is loaded sequentially, a preset number of inference requests are executed, and the measured memory usage, inference latency, and inference throughput are recorded. The performance score of each strategy combination is calculated based on the measured memory usage, inference latency, and inference throughput. The strategy combination with the highest performance score is marked as the optimal strategy label for that sampling point.

5. The method for lightweight deployment and dynamic adaptation of large models in multiple scenarios according to claim 4, characterized in that, The step of detecting mutations in the optimal strategy combination between adjacent sampling points, supplementing sampling and iteratively detecting between adjacent sampling points where mutations occur, and forming a training sample set includes: The current set of sampling points is grouped according to the inference backend type. A nearest neighbor graph is constructed within each group. For each edge in the nearest neighbor graph, it is determined whether the optimal strategy labels of the sampling points at both ends of the edge are different. For edges with different optimal policy labels, insert supplementary sampling points at the mean of the numerical hardware parameters at both ends of the edge, perform performance testing on the supplementary sampling points, and label them with the optimal policy. Add the supplementary sampling point to the current sampling point set, and repeatedly perform judgment and insertion on the newly added edge containing the supplementary sampling point until the parameter distance between all adjacent sampling points is lower than the preset resolution threshold or the total number of samples reaches the preset upper limit. The training sample set is obtained by summing up the sample pairs formed by each sample point in the final sample point set and the corresponding optimal policy label.

6. The method for lightweight deployment and dynamic adaptation of large models in multiple scenarios according to claim 5, characterized in that, The calculation of the local sample density of each sample point in the training sample set includes: For each sample point in the training sample set, find a preset number of nearest neighbors among the remaining sample points of the same inference backend type, and calculate the average distance of the nearest neighbors; The average nearest neighbor distance of each sample point is normalized by the global average nearest neighbor distance of all sample points in the training sample set. The normalization result is then subjected to a monotonically decreasing transformation to obtain the local sample density of each sample point.

7. The method for lightweight deployment and dynamic adaptation of large models in multiple scenarios according to claim 6, characterized in that, The step of constructing a loss function by interpolating between the accurate fitting target and the security policy regularization target for each sample point, using the local sample density as weight, includes: For each sample point in the training sample set, the negative logarithm of the prediction probability of the optimal policy label of the policy network for that sample point is taken as the exact fitting target. For each sample point, a security policy distribution is determined, and the divergence between the output distribution of the policy network for that sample point and the security policy distribution is used as the security policy regularization target. The precise fitting objective is weighted by the local sample density of the sample point, and the security policy regularization objective is weighted by subtracting the local sample density of the sample point. The two weighted results are added together to obtain the loss term corresponding to the sample point. The loss function is obtained by averaging the loss terms of all sample points in the training sample set.

8. The method for lightweight deployment and dynamic adaptation of large models in multiple scenarios according to claim 7, characterized in that, Determining the security policy distribution for each sample point includes: For each sample point, select strategy combinations from the set of strategy combinations whose measured video memory usage does not exceed the upper limit of available video memory indicated by the hardware parameters of that sample point, and obtain the legal strategy subset of that sample point. Within the subset of legitimate strategies, the strategy combination with the lowest measured memory usage is selected, and a probability distribution with this strategy combination as the only non-zero term is constructed as the security strategy distribution for this sample point.

9. The method for lightweight deployment and dynamic adaptation of large models in multiple scenarios according to claim 8, characterized in that, The filtering strategy combination that violates hardware resource constraints selects the strategy combination with the highest output probability from those that satisfy the constraints, including: From the set of strategy combinations, select strategy combinations whose measured video memory usage does not exceed the upper limit of available video memory of the hardware node to be deployed, to obtain a subset of legal strategy combinations; From the subset of legal strategy combinations, the strategy combination with the highest selection probability output by the deployment strategy network is selected as the deployment strategy.

Images