Large model parameter optimization and adaptive adjustment method and device based on reinforcement learning

Abstract

The invention relates to the technical field of large model optimization, and discloses a large model parameter optimization and adaptive adjustment method and device based on reinforcement learning, and the method comprises the steps: obtaining a current operation state of a target large model, and carrying out the comprehensive evaluation of the operation state of the target large model through combining with data features, and obtaining an operation state vector; constructing an intelligent agent, and inputting the operation state vector into the intelligent agent to obtain a parameter optimization strategy; optimizing parameters of the target large model based on a parameter optimization strategy to obtain an optimized large model; calculating a plurality of preset indexes for optimizing the large model by utilizing a reward function to serve as reward results; and adaptively adjusting parameters of the intelligent agent based on the reward result. In training and practical application of the large model, large model parameters are dynamically adjusted by means of an intelligent agent, it is ensured that rapid convergence can be achieved in the initial stage of training, the global optimal solution can be accurately approached in the later stage of training, the training efficiency and quality are effectively improved, and by means of a reward mechanism, the intelligent agent is made to adjust the parameters in a self-adaptive mode, and the generalization ability of the large model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large model optimization, and in particular to a method and device for large model parameter optimization and adaptive adjustment based on reinforcement learning. Background Art

[0002] In today's era of rapid development of artificial intelligence, large models have become the core technical support in many fields. Currently, large model parameter optimization mainly relies on the traditional gradient descent method and its improved algorithms, such as Stochastic Gradient Descent (SGD), Adaptive Gradient Algorithm (AdaGrad), and Adaptive Learning Rate Optimization Algorithm (AdaDelta). During the model training process, these methods calculate the gradient of the loss function with respect to the parameters and iteratively update the parameters to reduce the loss value.

[0003] Traditional methods have achieved some promising results, boasting relatively simple principles and ease of implementation. They are able to efficiently complete model training and parameter optimization for relatively simple tasks and datasets. However, with the increasing scale of large models and the increasing complexity of application scenarios, traditional methods have exposed numerous problems. First, they typically employ a fixed learning rate strategy, making it difficult to adapt to the demands of different training stages. While a higher learning rate can accelerate convergence in the early stages of training, it can easily lead to model oscillation and prevent convergence to the global optimal solution. Using a lower learning rate from the outset can make training extremely slow, consuming significant time and computational resources. Second, traditional methods lack the ability to adapt to dynamic changes in model runtime. Once deployed in real-world scenarios, data distribution may change over time, or the task requirements may change. Traditional methods struggle to adapt parameters in real time to these changes, leading to reduced model performance. For example, in machine translation tasks involving natural language processing, the emergence of new linguistic expressions and domain vocabulary can significantly reduce the accuracy and fluency of translation if the model cannot adaptively adjust parameters. Summary of the Invention

[0004] In view of this, the present invention provides a large model parameter optimization and adaptive adjustment method and device based on reinforcement learning to solve the problem of being unable to meet the training requirements of different stages and complex and changeable application scenarios.

[0005] In a first aspect, the present invention provides a method for large model parameter optimization and adaptive adjustment based on reinforcement learning, the method comprising:

[0006] Extract features from the input data to obtain data features, obtain the current operating status of the target large model, and conduct a comprehensive evaluation of the operating status of the target large model based on the data features to obtain the operating status vector;

[0007] Construct a parameter optimization agent for the target large model and input the running state vector into the parameter optimization agent to obtain the parameter optimization strategy;

[0008] Optimize the parameters of the target large model based on the parameter optimization strategy to obtain the optimized large model;

[0009] Use reward functions to calculate and optimize multiple preset indicators of large models as reward results for parameter optimization strategies;

[0010] Adaptively adjust the parameters based on the reward results to optimize the parameters of the agent.

[0011] The large-model parameter optimization and adaptive adjustment method based on reinforcement learning provided by the present invention dynamically adjusts the large-model parameters with the help of a parameter optimization agent during the training and actual application of the large model, ensuring rapid convergence in the early stage of training and accurate approximation to the global optimal solution in the later stage, effectively improving training efficiency and quality. With the help of a reward mechanism, the parameter optimization agent adaptively adjusts the parameters, so that the large model always maintains good performance in actual applications, improving its generalization ability and adaptability.

[0012] In an optional embodiment, the operating state of the target large model is comprehensively evaluated in combination with data features to obtain an operating state vector, including:

[0013] Perform data cleaning, standardization and feature coding preprocessing on the current operating status to obtain standard operating status data;

[0014] Construct an operating status indicator system consisting of a basic indicator layer, a performance indicator layer, and a comprehensive indicator layer, and map the characteristics of standard operating status data to corresponding indicators;

[0015] The weight of each indicator is determined by using the hierarchical analysis method, entropy weight method or principal component analysis method. A weighted calculation is performed based on the weight of each indicator and the indicator corresponding to the standard operating status data to obtain a comprehensive evaluation value of the operating status of the target large model.

[0016] The comprehensive evaluation value is converted into an operating status vector, and the dimension of the operating status vector is consistent with the number of indicators in the operating status indicator system.

[0017] The large-model parameter optimization and adaptive adjustment method based on reinforcement learning provided by the present invention timely adjusts the large-model parameters in combination with the operating status. When the environmental conditions of the large-model change, the large-model can quickly and adaptively adjust the relevant parameters, which significantly improves the generalization ability and adaptability of the model in complex and changeable real-world environments, enabling the large-model to operate stably and efficiently in more different fields and scenarios.

[0018] In an optional embodiment, the parameter optimization agent includes: a policy network and a value network. The running state vector is input into the parameter optimization agent to obtain a parameter optimization strategy, including:

[0019] Input the running state vector into the policy network to generate the adjustment strategy of the training parameters of the target large model;

[0020] Use the value network to evaluate the expected value of the adjustment strategy on the training effect of the target large model and output the value evaluation results;

[0021] Perform reinforcement learning training on the policy network based on the value assessment results and optimize the parameters of the policy network until the parameters of the policy network converge;

[0022] Generate parameter optimization strategy of the target large model based on the policy network after parameter optimization.

[0023] The large-model parameter optimization and adaptive adjustment method based on reinforcement learning provided by the present invention directly maps the operating state vector to a parameter optimization strategy through a policy network, eliminating the need for manual trial and error, significantly shortening the parameter tuning cycle. The value network guides the policy network to make decisions with a more global perspective by estimating the long-term returns of state-action pairs, avoiding local optimality. Through the collaborative work of the policy network and the value network, the operating state vector is converted into an efficient parameter optimization strategy, realizing the intelligent, automated, and efficient training of large models. This method not only improves model performance and resource utilization, but also reduces the technical threshold through interpretability and generalization capabilities, promoting the practical application of large models in more scenarios.

[0024] In an optional embodiment, the policy network corresponds to a preset action space, which includes a preset adjustment strategy for the training parameters of the target large model. The running state vector is input into the policy network to generate the adjustment strategy for the training parameters of the target large model, including:

[0025] Update the parameters of the policy network based on the preset adjustment strategy in the preset action space;

[0026] Based on the updated policy network parameters, the current learning rate is multiplied and scaled or additively offset within a preset range. The preset range is dynamically adjusted according to the large model training stage.

[0027] Select a new batch size from a predefined set of discrete values, or dynamically calculate the optimal batch size based on memory utilization;

[0028] The characteristics of the running state vector are analyzed to determine the update step scaling factors and dynamic priorities of different layers or parameter groups.

[0029] The large-model parameter optimization and adaptive adjustment method based on reinforcement learning provided by the present invention achieves high efficiency, intelligence and controllability of large-model parameter optimization by combining the preset action space with the policy network. The policy network dynamically selects the adjustment strategy based on the running state vector, which not only accelerates the model training and tuning process, but also improves resource utilization, model stability and interpretability, providing key technical support for the engineering implementation of large models.

[0030] In an optional embodiment, the preset indicators include: the reduction in loss value, the improvement in accuracy, and the model convergence speed. The reward function is used to calculate and optimize multiple preset indicators of the large model as the reward result of the parameter optimization strategy, including:

[0031] If the decrease in loss exceeds the preset threshold, the improvement in accuracy exceeds the preset threshold, and the model convergence speed exceeds the preset threshold, then the reward result is a positive reward.

[0032] If the decrease in loss fluctuates, or the increase in accuracy is less than the preset threshold, or the model convergence speed is less than the preset convergence speed threshold, the reward result will be a negative reward.

[0033] The reinforcement learning-based large-model parameter optimization and adaptive adjustment method provided by the present invention constructs a reward function based on preset indicators such as loss value, accuracy, and convergence speed, providing a comprehensive, flexible, and quantitative evaluation system for large-model parameter optimization. It can not only accelerate strategy iteration and improve model performance, but also reduce resource consumption, providing strong support for the efficient training and deployment of large models.

[0034] In an optional embodiment, the parameters of the target large model are optimized based on the parameter optimization strategy to obtain the optimized large model, including:

[0035] During the training of the target large model, the parameter optimization agent adjusts the parameter optimization strategy in real time according to the running state vector and reward results, and adjusts the parameters of the target large model based on the parameter optimization strategy;

[0036] After the target large model is deployed, the operating status data in the deployment environment is obtained, and the parameter optimization agent adaptively adjusts the parameters of the target large model according to the operating status data.

[0037] The large-model parameter optimization and adaptive adjustment method based on reinforcement learning provided by the present invention achieves a triple improvement in model performance (accuracy + efficiency), resource cost, and robustness through intelligent parameter adjustment in two stages, while reducing engineering thresholds and manpower dependence.

[0038] In a second aspect, the present invention provides a large model parameter optimization and adaptive adjustment device based on reinforcement learning, the device comprising:

[0039] The operation state vector determination module is used to extract the features of the input data to obtain data features, obtain the current operation state of the target large model, and comprehensively evaluate the operation state of the target large model based on the data features to obtain the operation state vector;

[0040] The optimization strategy determination module is used to build a parameter optimization agent for the target large model and input the running state vector into the parameter optimization agent to obtain the parameter optimization strategy;

[0041] The parameter optimization module is used to optimize the parameters of the target large model based on the parameter optimization strategy to obtain the optimized large model;

[0042] The reward calculation module is used to calculate and optimize multiple preset indicators of the large model using the reward function as the reward result for the parameter optimization strategy;

[0043] The adaptive adjustment module is used to adaptively adjust the parameters of the agent based on the reward results to optimize the parameters.

[0044] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0046] In a fifth aspect, the present invention provides a computer program product comprising computer instructions for causing a computer to execute the method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 1 is a flow chart of a method for large model parameter optimization and adaptive adjustment based on reinforcement learning according to an embodiment of the present invention;

[0049] Figure 2 2 is a flow chart of another method for large model parameter optimization and adaptive adjustment based on reinforcement learning according to an embodiment of the present invention;

[0050] Figure 3 2. It is a schematic diagram of the internal structure of an intelligent agent in a large model parameter optimization and adaptive adjustment method based on reinforcement learning according to an embodiment of the present invention;

[0051] Figure 4 2 is a structural block diagram of a large model parameter optimization and adaptive adjustment device based on reinforcement learning according to an embodiment of the present invention;

[0052] Figure 5 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0054] An embodiment of the present invention provides a large model parameter optimization and adaptive adjustment method based on reinforcement learning, which performs reinforcement learning and adaptive adjustment on the large model parameters through an intelligent agent to achieve the effect of meeting the training requirements of different stages and complex and changeable application scenarios.

[0055] According to an embodiment of the present invention, an embodiment of a large model parameter optimization and adaptive adjustment method based on reinforcement learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] In this embodiment, a large model parameter optimization and adaptive adjustment method based on reinforcement learning is provided, which can be used in the above-mentioned computer system. Figure 1 is a flow chart of a large model parameter optimization and adaptive adjustment method based on reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0057] Step S101 , extracting features from input data to obtain data features, obtaining the current operating state of the target large model, and comprehensively evaluating the operating state of the target large model in combination with the data features to obtain an operating state vector.

[0058] Specifically, during the large-scale model training and application phases, feature information of input data is collected in real time. For example, in natural language processing tasks, the input text's vocabulary distribution, grammatical structure, and semantic themes are analyzed and extracted; in image recognition tasks, image features such as color, texture, and shape are extracted. These are examples, but not limited to these. Feature extraction allows us to accurately grasp the characteristics of the data and provide a basis for subsequent parameter adjustments.

[0059] Combining the extracted data features and the current operating status of the large model, such as the model's loss value, accuracy, parameter update status, etc., the operating status of the large model is evaluated, and the evaluation results of the operating status are identified in the form of an operating status vector.

[0060] Step S102: construct a parameter optimization agent for the target large model, and input the operating state vector into the parameter optimization agent to obtain a parameter optimization strategy.

[0061] Specifically, the parameter optimization agent is used to provide parameter optimization strategies for the target large model, and continuously learns and optimizes parameter adjustment decisions through data interaction with the operating environment of the large model.

[0062] After building the basic architecture of the intelligent agent, the running state vector is input into the parameter optimization intelligent agent. After the parameter optimization intelligent agent performs specific operations on the large model parameters, a specific parameter optimization strategy is output, such as the adjustment range of the learning rate. This is only for example, but not limited to this.

[0063] Step S103: Optimize the parameters of the target large model based on the parameter optimization strategy to obtain an optimized large model.

[0064] Specifically, the parameters of the target large model are dynamically adjusted based on the parameter optimization strategy until the parameters converge to obtain the optimized large model.

[0065] Step S104: Utilize the reward function to calculate and optimize multiple preset indicators of the large model as a reward result for the parameter optimization strategy.

[0066] Specifically, the reward function is the key to guiding the parameter optimization agent to learn the correct parameter adjustment strategy. The design of the reward function is based on multiple indicators, including but not limited to: the reduction in model loss value, the improvement in accuracy, the speed of model convergence, etc.

[0067] After the parameter optimization agent generates a parameter adjustment strategy each time, it uses the parameter adjustment strategy to optimize the target large model. Then, it evaluates the optimized large model based on the reward function to obtain the evaluation values ​​of multiple preset indicators of the large model as the reward result of the parameter optimization strategy.

[0068] Step S105: adaptively adjust the parameters of the agent based on the reward result to optimize the parameters.

[0069] Specifically, after the parameter optimization agent adjusts the parameters of the large model, it will obtain corresponding reward results based on the feedback from the large model's operating environment. The reward results are not only used to evaluate the quality of the current parameter optimization strategy, but also backpropagate through the reinforcement learning algorithm to update the parameters of the parameter optimization agent. Through continuous trial and error and learning, the agent gradually masters the parameter adjustment strategy that can optimize the model performance.

[0070] The large model parameter optimization and adaptive adjustment method based on reinforcement learning provided in this embodiment dynamically adjusts the large model parameters with the help of a parameter optimization agent during the training and actual application of the large model, ensuring rapid convergence in the early stage of training and accurate approximation to the global optimal solution in the later stage, effectively improving training efficiency and quality. With the help of a reward mechanism, the parameter optimization agent adaptively adjusts the parameters, so that the large model always maintains good performance in actual applications, improving its generalization ability and adaptability.

[0071] In this embodiment, a large model parameter optimization and adaptive adjustment method based on reinforcement learning is provided, which can be used in the above-mentioned computer system. Figure 2 is a flow chart of a large model parameter optimization and adaptive adjustment method based on reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0072] Step S201 , extracting features from input data to obtain data features, obtaining the current operating state of the target large model, and comprehensively evaluating the operating state of the target large model in combination with the data features to obtain an operating state vector.

[0073] Specifically, the above step S201 includes:

[0074] Step S2011 , performing data cleaning, standardization and feature coding preprocessing on the current operating status to obtain standard operating status data.

[0075] Specifically, for the current operating status of the target large model, the preprocessing process includes: removing outliers and duplicate data, filling missing values, converting data of different magnitudes and units into a unified scale, and converting non-numeric data into numerical data. After preprocessing, standard operating status data is obtained.

[0076] Step S2012: construct an operating status indicator system including a basic indicator layer, a performance indicator layer, and a comprehensive indicator layer, and map the characteristics of the standard operating status data to corresponding indicators.

[0077] Specifically, the operational status indicator system consists of three layers: a basic indicator layer containing fundamental data such as hardware resources; a performance indicator layer used to measure performance parameters such as inference speed and accuracy; and a comprehensive indicator layer used to demonstrate comprehensive performance, such as stability and scalability. Based on the operational status indicator system, standard operational status data is precisely mapped to corresponding indicators at each level, enabling multi-dimensional evaluation.

[0078] Step S2013, using the hierarchical analysis method, entropy weight method or principal component analysis method to determine the weight of each indicator, and performing weighted calculation based on the weight of each indicator and the indicator corresponding to the standard operating status data to obtain a comprehensive evaluation value of the operating status of the target large model.

[0079] Specifically, define standard indicator values ​​under ideal conditions, such as: hardware resource utilization: CPU / GPU utilization ≤ 80% (avoid overload); loss value: close to the minimum value when the training set converges; accuracy: reach the expected target value (such as 90%), just as an example, but not limited to this. Based on the actual indicator data and standard indicator values ​​of the target large model's operating status, determine the actual score of each indicator. The weight of each indicator can be determined by hierarchical analysis, entropy weight method or principal component analysis. The specific process is a mature existing technology and will not be repeated here. A linear weighted calculation is performed using the actual score and weight of each indicator to obtain a comprehensive evaluation value of the target large model's operating status.

[0080] Step S2014: convert the comprehensive evaluation value into an operation state vector, where the dimension of the operation state vector is consistent with the number of indicators in the operation state indicator system.

[0081] Specifically, the process of converting the comprehensive evaluation value into the running state vector needs to be combined with the indicator system design, data mapping rules and vector semantic definition to ensure that the vector can not only fully reflect the status of each dimension, but also facilitate subsequent analysis. For example, the intelligent agent adjusts the learning rate, batch size and other parameters according to the running state vector. Example vector (simplified to three dimensions): v = [0.8, -0.5, 0.9] The meaning of each dimension in the vector: excellent CPU utilization, large loss fluctuations, and excellent label quality, indicating sufficient hardware resources, but loss fluctuations may be caused by an excessively high learning rate, and the intelligent agent should reduce the learning rate.

[0082] The large model parameter optimization and adaptive adjustment method based on reinforcement learning provided in this embodiment adjusts the large model parameters in a timely manner in combination with the operating status. When the environmental conditions of the large model change, the large model can quickly and adaptively adjust the relevant parameters, which significantly improves the generalization ability and adaptability of the model in complex and changeable real environments, enabling the large model to operate stably and efficiently in more different fields and scenarios.

[0083] Step S202: construct a parameter optimization agent for the target large model, and input the operating state vector into the parameter optimization agent to obtain a parameter optimization strategy.

[0084] Specifically, the parameter optimization agent includes: strategy network and value network, such as Figure 3 The figure shows the internal structure of the parameter optimization agent. The above step S202 includes:

[0085] Step S2021: Input the running state vector into the strategy network to generate an adjustment strategy for the training parameters of the target large model.

[0086] In some optional implementations, the policy network corresponds to a preset action space, which includes a preset adjustment strategy for the training parameters of the target large model. Step S2021 includes:

[0087] Step a1: Update the parameters of the policy network based on the preset adjustment strategy in the preset action space.

[0088] Specifically, the preset action space is a predefined set of possible policy adjustment operations, such as gradient descent, stochastic gradient descent, and adaptive gradient algorithms (such as Adagrad, Adadelta, RMSProp, Adam, etc.). The preset adjustment strategy is to select an appropriate method among these operations and determine its hyperparameters (such as learning rate, momentum, etc.).

[0089] In its implementation, the algorithm first selects an adjustment strategy from the preset action space based on the current policy network training state (such as loss function value, gradient size, etc.). It then uses this strategy to update the policy network parameters. The gradient of the loss function with respect to the network parameters is calculated using the backpropagation algorithm, and the parameters are updated according to the rules of the selected strategy, allowing the policy network to evolve in the direction of producing a more optimal action strategy.

[0090] In step a2, based on the policy network after parameter update, the current learning rate is multiplied and scaled or additively offset within a preset range. The preset range is dynamically adjusted according to the large model training stage.

[0091] Specifically, in the early stages of training, the model parameters are far from the optimal solution. To speed up convergence, a larger preset range can be set, allowing the learning rate to be adjusted within a larger range. Multiplication scaling (such as multiplication by a coefficient greater than 1) or additive offset (such as adding a larger value) is used to increase the learning rate. As training progresses, the model gradually approaches the optimal solution. To avoid model divergence caused by excessive parameter updates, the preset range will gradually narrow. At this time, the adjustment range of the learning rate will also become smaller. A smaller multiplication coefficient or additive offset value may be used, or even the learning rate may be gradually attenuated.

[0092] In step a3, a new batch size is selected from a predefined set of discrete values, or the optimal batch size is dynamically calculated based on memory utilization.

[0093] Specifically, the predefined discrete value set is a set of possible batch size values ​​predetermined based on experience or experimentation, such as [16, 32, 64, 128, 256]. When selecting a new batch size, an appropriate value can be selected from this discrete value set based on the current state and requirements of model training. For example, when the amount of training data is large and computing resources are sufficient, a larger batch size can be selected to speed up training; when the training data is complex and the model is prone to overfitting, a smaller batch size can be selected to increase training stability.

[0094] Dynamically calculate the optimal batch size based on memory utilization. During training, the system memory usage is monitored in real time. Based on the model parameter size and the memory limitations of the computing device, a specific algorithm is used to dynamically calculate the maximum batch size that can be supported under the current memory conditions. Alternatively, based on the relationship between memory utilization and training efficiency, an optimal batch size is found that fully utilizes memory resources while ensuring training efficiency.

[0095] Step a4: Analyze the characteristics of the running state vector to determine the update step scaling factors and dynamic priorities of different layers or parameter groups.

[0096] Specifically, the running state vector records various state information of the policy network during operation, such as the activation values ​​and gradient information of each layer. By analyzing the characteristics of the running state vector, we can understand the importance and sensitivity differences of different layers or parameter groups during the training process. For different layers or parameter groups, the corresponding update step scaling factors are determined according to their functions and training status in the model. For example, for the bottom layer of the network, it is mainly responsible for extracting basic features and is usually relatively stable. The update step scaling factor can be set to a smaller value to avoid excessive updates that destroy the learning of basic features. For the high layer of the network, it is responsible for abstracting and making decisions on features. In the later stages of training, a larger update step may be required to adapt to changes in data. Therefore, a larger update step scaling factor can be set.

[0097] The reinforcement learning-based large-model parameter optimization and adaptive adjustment method provided in this embodiment achieves high efficiency, intelligence and controllability of large-model parameter optimization by combining the preset action space with the policy network. The policy network dynamically selects the adjustment strategy based on the running state vector, which not only accelerates the model training and tuning process, but also improves resource utilization, model stability and interpretability, providing key technical support for the engineering implementation of large models.

[0098] Step S2022: Use the value network to evaluate the expected value of the adjustment strategy on the training effect of the target large model and output the value evaluation result.

[0099] Specifically, the state information of the current policy network and the adjustment strategy to be evaluated are encoded and converted into a vector form that can be processed by the value network. For example, numerical parameters such as learning rate and batch size are directly used as elements of the vector, and the type of adjustment strategy (such as which optimization algorithm is used) is one-hot encoded and added to the vector.

[0100] The encoded input vector is passed into the trained value network. The network calculates and outputs the expected value score of the adjustment strategy through forward propagation. The score is a quantitative value that reflects the value network's prediction of the strategy in improving the training effect of the large model.

[0101] Step S2023: Perform reinforcement learning training on the policy network according to the value evaluation result to optimize the parameters of the policy network until the parameters of the policy network converge.

[0102] Specifically, the decision to adopt the adjustment strategy is made based on the expected value score output by the value network, combined with actual business needs and resource constraints. For example, if the score is above a certain threshold and the uncertainty is within an acceptable range, the strategy can be considered for large-scale model training. If the score is low or the uncertainty is too high, the strategy needs to be reselected or adjusted and re-evaluated.

[0103] Step S2024: Generate a parameter optimization strategy for the target large model based on the policy network after parameter optimization.

[0104] The large-model parameter optimization and adaptive adjustment method based on reinforcement learning provided in this embodiment directly maps the running state vector to a parameter optimization strategy through the policy network, eliminating the need for manual trial and error, significantly shortening the parameter tuning cycle. The value network guides the policy network to make decisions with a more global perspective by estimating the long-term returns of the state-action pair, avoiding local optimality. Through the collaborative work of the policy network and the value network, the running state vector is converted into an efficient parameter optimization strategy, realizing the intelligent, automated and efficient training of large models. This method not only improves model performance and resource utilization, but also reduces the technical threshold through interpretability and generalization capabilities, promoting the application of large models in more scenarios.

[0105] Step S203: Optimize the parameters of the target large model based on the parameter optimization strategy to obtain an optimized large model.

[0106] Specifically, the above step S203 includes:

[0107] Step S2031: During the training of the target large model, the parameter optimization agent adjusts the parameter optimization strategy in real time according to the running state vector and reward results, and adjusts the parameters of the target large model based on the parameter optimization strategy.

[0108] Specifically, during large-scale model training, the agent adjusts model parameters in real time based on the model's operational status and reward feedback. For example, if the model converges slowly in the early stages of training, the agent, through the policy network, increases the learning rate. Later in training, as convergence approaches, the agent decreases the learning rate to prevent oscillations. Furthermore, the priority of parameter updates is dynamically adjusted based on the update status of parameters at different layers, ensuring more timely and effective updates of important parameters.

[0109] In step S2032, after the target large model is deployed, the operating status data in the deployment environment is obtained, and the parameters of the target large model are adaptively adjusted according to the operating status data based on the parameter optimization agent.

[0110] Specifically, after the large model is deployed in a real-world application scenario, the intelligent agent continuously monitors changes in data and task requirements. Once a change in the environment is detected, the agent quickly responds by adjusting the large model's parameters to adapt to the new environment. For example, in intelligent customer service scenarios, when encountering new question types or changes in user language style, the model can automatically adjust its parameters to improve the accuracy and relevance of responses.

[0111] The large-model parameter optimization and adaptive adjustment method based on reinforcement learning provided in this embodiment uses two-stage continuous optimization to achieve a triple improvement in model performance (accuracy + efficiency), resource cost, and robustness through intelligent parameter adjustment, while reducing engineering thresholds and manpower dependence.

[0112] Step S204: Utilize the reward function to calculate and optimize multiple preset indicators of the large model as a reward result for the parameter optimization strategy.

[0113] Specifically, the preset indicators include: the reduction of loss value, the improvement of accuracy, and the model convergence speed. The above step S204 includes:

[0114] In step S2041, if the decrease in the loss value exceeds the preset amplitude threshold, the increase in accuracy is greater than the preset amplitude threshold, and the model convergence speed is greater than the preset convergence speed threshold, then the reward result is a positive reward.

[0115] In step S2042, if the decrease in the loss value fluctuates, or the increase in accuracy is less than the preset threshold, or the model convergence speed is less than the preset convergence speed threshold, the reward result is a negative reward.

[0116] The reinforcement learning-based large-model parameter optimization and adaptive adjustment method provided in this embodiment constructs a reward function based on preset indicators such as loss value, accuracy, and convergence speed, providing a comprehensive, flexible, and quantitative evaluation system for large-model parameter optimization. It can not only accelerate strategy iteration and improve model performance, but also reduce resource consumption, providing strong support for the efficient training and deployment of large models.

[0117] Step S205: Adaptively adjust the parameters based on the reward results to optimize the parameters of the agent. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0118] In a specific embodiment, the process of using a large model for text recognition and optimizing parameters of the large model includes:

[0119] (1) Data processing: We collect a wide range of text data, including news, literary works, academic papers, and social media comments, to build a large and rich corpus. To ensure the diversity and representativeness of the data, we obtain data from data sources in different fields, different language styles, and different time spans. For example, we collect not only news reports from mainstream media, but also professional literature in niche fields and daily speeches of netizens on social platforms.

[0120] Use professional word segmentation tools, such as deep learning-based word segmentation models or classic word segmentation algorithms (such as Jieba word segmentation), to split continuous text into individual word units. For English text, simple word segmentation can be achieved by relying on spaces and punctuation marks; for languages like Chinese that lack natural delimiters, more complex algorithms are required to accurately identify word boundaries. Use part-of-speech tagging tools to label the part of speech of each segmented word, such as nouns, verbs, adjectives, etc. This helps to understand the grammatical functions of words in sentences and provides a basis for subsequent semantic analysis. Establish a stop word list to remove those words that frequently appear in the text but contribute little to semantic expression, such as "de", "shi", "zai", etc. This can reduce the amount of data, improve the processing efficiency of the model, and avoid noise interference.

[0121] Convert the processed text into a vector form that the computer can understand. Vectors not only contain the semantic information of words but can also measure the similarity between words through operations between vectors.

[0122] (2) Initialize the running environment: Set the window size and frequency threshold for counting word frequencies. By counting the number of times each word appears within a certain window, the lexical distribution characteristics of the text can be obtained, which helps to understand the theme and common words of the text. Determine the method for analyzing the dimensions of word vectors, such as principal component analysis (PCA) or singular value decomposition (SVD), to extract the main characteristic components of word vectors, reduce dimensional redundancy, and retain key semantic information at the same time.

[0123] Set evaluation metrics for the large model state: Define the calculation method and update frequency of perplexity. Perplexity is an important indicator for measuring the prediction ability of a language model and reflects the fitting degree of the model to the text. A lower perplexity indicates that the model has a stronger understanding and prediction ability for the text.

[0124] Accuracy evaluation: Determine the specific method for calculating accuracy on the validation set. For example, for text classification tasks, accuracy refers to the proportion of the number of samples correctly classified by the model to the total number of samples. At the same time, set the improvement threshold for accuracy. When the model accuracy reaches or exceeds this threshold, it is considered that the model performance has been significantly improved.

[0125] (3) Build an agent: For the policy network: Adopt a multi-layer perceptron (MLP) structure (including multiple hidden layers, and each hidden layer consists of multiple neurons). The input layer receives the state vector output by the environmental perception module. After non-linear transformation by the hidden layer, the output layer outputs parameter adjustment actions. To improve the generalization ability and training efficiency of the model, add a Dropout layer between the hidden layers to randomly discard some neurons to prevent overfitting.

[0126] The value network is also built on an MLP, and its input is the same as the policy network: the environment state vector. The value network uses feedback from the environment to estimate the cumulative reward the agent will receive over a period of time after taking a specific action. To speed up training, pre-training can be used to initialize the value network's parameters, bringing them closer to the optimal solution.

[0127] Initialize the policy and value network parameters using a random initialization method (such as Xavier or Kaiming) to ensure that the initial parameters are distributed within a reasonable range, which is conducive to model convergence. At the same time, set optimizer parameters such as learning rate and momentum, which will affect the training speed and convergence of the model.

[0128] (4) Training the large model: For each batch of input text data, first extract its data features, such as word frequency and word vector principal components, and combine the current model's state information such as perplexity and accuracy on the training set and validation set to generate a complete state vector. This state vector serves as the input of the reinforcement learning module agent, allowing the agent to understand the current environment of the model.

[0129] The agent adjusts the parameters of the large model based on the actions output by the policy network. For example, when the policy network outputs an action to increase the learning rate, the agent increases the current learning rate by a certain percentage (e.g., 10%). When the policy network outputs an action to adjust the parameter update step size, the agent adjusts the parameter update step size according to the step size specified in the action. Furthermore, the agent can adjust the update priority of different layer parameters based on the actions, increasing the frequency and magnitude of parameter updates for layers that are more relevant to the current task.

[0130] The agent is rewarded based on the model's performance during training. If the model's perplexity decreases and its accuracy improves on the validation set, the agent receives a positive reward, the value of which varies depending on the magnitude of the decrease or increase. If the model overfits or its performance degrades, a negative reward is given. Based on reward feedback, the agent uses a reinforcement learning algorithm (such as the Proximal Policy Optimization (PPO) algorithm or the Deep Q-Network (DQN) algorithm) to update the parameters of the policy and value networks, allowing the agent to gradually learn a more optimal parameter adjustment strategy.

[0131] (5) Applying large models: When the model is deployed in real-world applications, such as intelligent writing assistance tools, it continuously monitors changes in the characteristics of user input text. By analyzing the vocabulary usage habits, sentence structure characteristics, and writing topics of the user input text, it can be determined whether the user's writing style and needs have changed. For example, if a user frequently uses professional terms over a period of time, it means that they may be writing in a professional field. The model needs to adjust parameters accordingly to better understand and assist this writing style.

[0132] Once a change in a user's writing style or needs is detected, the relevant information is immediately passed to the agent. Based on this information, the agent uses the policy network to adjust the parameters of the large model. For example, the language model's word vector representations can be adjusted to better adapt to the new vocabulary distribution, or the model's attention mechanism parameters can be adjusted to better capture the semantic relationships in the user's text. Through this real-time adaptive parameter adjustment, the model can provide users with more accurate and tailored writing suggestions, such as grammatical correction, vocabulary recommendations, and sentence polishing.

[0133] In another specific embodiment, a large model is used for image recognition, and a process of parameter optimization of the large model includes:

[0134] (1) Data processing: Collect large-scale image datasets covering a variety of categories and scenarios. Take the common image recognition datasets CIFAR-10 and ImageNet as examples. CIFAR-10 contains 60,000 color images from 10 different categories, while ImageNet is a large-scale dataset with over 14 million images covering more than 20,000 categories. In addition to these public datasets, image data from specific fields can also be collected based on specific application scenarios, such as medical imaging and industrial inspection images.

[0135] Normalize the image's pixel values, mapping the pixel value range from [0, 255] to [0, 1] or [-1, 1]. This ensures consistent data distribution across different images, facilitating model training and convergence. Cropping and scaling the images are performed based on the model input requirements. For example, images can be uniformly cropped to a fixed size (e.g., 224x224 pixels) to accommodate the input dimensions of a convolutional neural network. To increase data diversity, data augmentation techniques such as random cropping, flipping, and rotation can be used to expand the dataset and improve the model's generalization capabilities. Image enhancement algorithms, such as histogram equalization, contrast enhancement, and Gaussian blurring, can be used to improve image quality and visual quality. These operations can highlight key features in the image, reduce noise interference, and enhance the model's ability to recognize images.

[0136] (2) Determine the running feature data: Image feature extraction parameter settings: Color histogram: Set the number of bins and calculation method for the color histogram. By statistically analyzing the distribution of different colors in the image, the color features of the image are obtained. The color histogram can reflect the overall hue and color distribution characteristics of the image, which is helpful for distinguishing different types of images.

[0137] HOG features: Determine the parameters of the Histogram of Oriented Gradients (HOG), such as cell size, block size, and number of gradient directions. HOG features describe the edge and shape information of an image by calculating the gradient direction and magnitude of a local area of ​​the image, and are widely used in object detection and image classification tasks.

[0138] Accuracy calculation: Define the method for calculating accuracy on the training and validation sets. For image recognition tasks, accuracy refers to the ratio of images correctly classified by the model to the total number of images. Also, set the frequency for monitoring accuracy and the target for improvement to ensure timely evaluation of model performance changes.

[0139] Recall evaluation: In addition to precision, recall is also a metric to consider, especially in applications where a high miss detection rate is critical. Recall refers to the ratio of correctly classified samples to the actual number of samples. By comprehensively evaluating precision and recall, we can gain a more comprehensive understanding of model performance.

[0140] (3) Building an intelligent agent: A convolutional neural network (CNN) is used as the underlying structure for the policy network and value network. CNN can automatically extract local features of an image and efficiently process image data through a combination of convolutional layers, pooling layers, and fully connected layers. In the policy network, the input layer receives the state vector (containing image features and model state information) output by the environment perception module. After multiple convolutional layers and fully connected layers, the output parameter adjustment action is processed. The value network has a similar structure, but outputs an action value estimate.

[0141] Initialize the CNN parameters using a suitable initialization method (such as Kaiming) to ensure a good distribution of the initial parameters. During training, use an optimization algorithm (such as the Adam optimizer) to update the parameters of the policy network and value network. At the same time, set a learning rate decay strategy to gradually reduce the learning rate as training progresses to ensure the convergence stability of the model.

[0142] (4) Training a large model: When training an image recognition model, the features of each batch of input images are extracted in real time. Combined with the current model’s accuracy, recall, and other state information, a state vector is generated and fed into the agent. For example, if the model’s recognition accuracy for certain categories of images is low during training, this information is passed to the agent as part of the state vector, allowing the agent to understand the model’s current problems.

[0143] The agent adjusts the parameters of the image recognition model based on the actions output by the policy network. These actions can include adjusting the size, number, and step size of the convolution kernels, changing the number and connection structure of neurons in the fully connected layer, and adjusting the learning rate and regularization parameters. For example, if the policy network determines that the model is insufficiently capturing certain detailed features, it may increase the number of convolution kernels in the convolution layer to enhance the model's ability to perceive details. If the model shows signs of overfitting, the agent adjusts the regularization parameters to improve the model's generalization ability.

[0144] The agent is rewarded based on the model's performance on the training and validation sets. If both the model's precision and recall improve, a positive reward is given; if performance degrades or overfits, a negative reward is given. The agent uses a reinforcement learning algorithm (such as A2C or DDPG) to update the parameters of the policy and value networks based on reward feedback, continuously optimizing the parameter adjustment strategy to achieve gradually better performance during training.

[0145] (5) Applying large models: In actual image recognition applications, such as face recognition systems in security monitoring, the environmental perception module continuously monitors factors such as the quality of the input image, lighting conditions, and posture changes. When a change in lighting conditions is detected, this information is passed to the reinforcement learning module. Based on the decision of the policy network, the agent adjusts the parameters of the face recognition model, such as adjusting the lighting compensation parameters of image preprocessing or adjusting the parameters of the lighting-related feature extraction layer in the convolutional neural network to adapt to the new lighting conditions and ensure accurate recognition of the target person in different lighting environments.

[0146] Models require varying adaptability for different application scenarios, such as access control systems, video surveillance, and mobile device recognition. By monitoring scenario-related information, such as the frequency of entry and exit in access control systems and the scene complexity in video surveillance, the intelligent agent provides a basis for decision-making. Based on this information, the intelligent agent dynamically adjusts model parameters to ensure optimal performance across diverse scenarios. For example, in access control systems with frequent entry and exit, the model can simplify the complex feature extraction process to improve recognition speed while maintaining a certain level of recognition accuracy. In complex video surveillance scenarios, the model needs to be more robust to various interference factors, adjusting parameters to enhance face recognition capabilities in various poses and under occlusion.

[0147] In this embodiment, a large model parameter optimization and adaptive adjustment device based on reinforcement learning is also provided. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the details that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0148] This embodiment provides a large model parameter optimization and adaptive adjustment device based on reinforcement learning, such as Figure 4 Shown, including:

[0149] The operation state vector determination module 401 is used to extract features from input data to obtain data features, obtain the current operation state of the target large model, and comprehensively evaluate the operation state of the target large model based on the data features to obtain the operation state vector.

[0150] The optimization strategy determination module 402 is used to construct a parameter optimization agent for the target large model, and input the operating state vector into the parameter optimization agent to obtain a parameter optimization strategy.

[0151] The parameter optimization module 403 is used to optimize the parameters of the target large model based on the parameter optimization strategy to obtain an optimized large model.

[0152] The reward calculation module 404 is used to calculate and optimize multiple preset indicators of the large model using the reward function as the reward result for the parameter optimization strategy.

[0153] The adaptive adjustment module 405 is used to adaptively adjust the parameters of the optimization agent based on the reward results.

[0154] In some optional implementations, the operation state vector determination module 401 includes:

[0155] The data preprocessing unit is used to perform data cleaning, standardization and feature coding on the current operating status to obtain standard operating status data.

[0156] The indicator system construction unit is used to construct an operating status indicator system including a basic indicator layer, a performance indicator layer and a comprehensive indicator layer, and map the characteristics of the standard operating status data to corresponding indicators.

[0157] The comprehensive evaluation unit is used to determine the weight of each indicator using the hierarchical analysis method, entropy weight method or principal component analysis method, and perform weighted calculation based on the weight of each indicator and the indicator corresponding to the standard operating status data to obtain a comprehensive evaluation value of the operating status of the target large model.

[0158] The vector conversion unit is used to convert the comprehensive evaluation value into an operation status vector, and the dimension of the operation status vector is consistent with the number of indicators in the operation status indicator system.

[0159] In some optional implementations, the optimization strategy determination module 402 includes:

[0160] The running state input unit is used to input the running state vector into the strategy network to generate the adjustment strategy of the training parameters of the target large model.

[0161] The strategy evaluation unit is used to use the value network to evaluate the expected value of the adjustment strategy on the training effect of the target large model and output the value evaluation result.

[0162] The policy network parameter optimization unit is used to perform reinforcement learning training on the policy network according to the value evaluation results and optimize the parameters of the policy network until the parameters of the policy network converge.

[0163] The parameter optimization strategy generation unit is used to generate the parameter optimization strategy of the target large model based on the policy network after parameter optimization.

[0164] In some optional implementations, the parameter optimization module 403 includes:

[0165] The training optimization unit is used during the training of the target large model. The parameter optimization agent adjusts the parameter optimization strategy in real time according to the running state vector and reward results, and adjusts the parameters of the target large model based on the parameter optimization strategy.

[0166] The optimization unit is used to obtain the operating status data of the deployment environment after the target large model is deployed, and the parameter optimization agent adaptively adjusts the parameters of the target large model according to the operating status data.

[0167] In some optional implementations, the reward calculation module 404 includes:

[0168] The positive reward unit is used to reward a positive reward if the decrease in the loss value exceeds a preset threshold, the increase in accuracy is greater than a preset threshold, and the model convergence speed is greater than a preset convergence speed threshold.

[0169] The negative reward unit is used to reward a negative reward if the decrease in the loss value fluctuates, or the increase in accuracy is less than the preset threshold, or the model convergence speed is less than the preset convergence speed threshold.

[0170] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0171] The large model parameter optimization and adaptive adjustment device based on reinforcement learning in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0172] The embodiment of the present invention also provides a computer device having the above Figure 4 The large model parameter optimization and adaptive adjustment device based on reinforcement learning is shown.

[0173] See also Figure 5 , Figure 5 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0174] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0175] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0176] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0178] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0179] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0180] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0181] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A large model parameter optimization and adaptive adjustment method based on reinforcement learning, characterized in that: The method comprises: Extracting features from the input data to obtain data features, obtaining the current operating state of the target large model, and comprehensively evaluating the operating state of the target large model based on the data features to obtain an operating state vector; Constructing a parameter optimization agent for the target large model, and inputting the operating state vector into the parameter optimization agent to obtain a parameter optimization strategy; Optimizing the parameters of the target large model based on the parameter optimization strategy to obtain an optimized large model; Calculating multiple preset indicators of the optimized large model using a reward function as a reward result for the parameter optimization strategy; Adaptively adjust the parameters of the parameter optimization agent based on the reward result.

2. The method according to claim 1, characterized in that Combined with the data features, the operating status of the target large model is comprehensively evaluated to obtain an operating status vector, including: Perform data cleaning, standardization and feature coding preprocessing on the current operating status to obtain standard operating status data; Construct an operating status indicator system consisting of a basic indicator layer, a performance indicator layer, and a comprehensive indicator layer, and map the characteristics of standard operating status data to corresponding indicators; The weight of each indicator is determined by using the hierarchical analysis method, entropy weight method or principal component analysis method. A weighted calculation is performed based on the weight of each indicator and the indicator corresponding to the standard operating status data to obtain a comprehensive evaluation value of the operating status of the target large model. The comprehensive evaluation value is converted into an operation state vector, and the dimension of the operation state vector is consistent with the number of indicators in the operation state indicator system.

3. The method according to claim 1, characterized in that The parameter optimization agent includes: a strategy network and a value network. The inputting the running state vector into the parameter optimization agent to obtain the parameter optimization strategy includes: Inputting the running state vector into the strategy network to generate an adjustment strategy for the training parameters of the target large model; Using the value network, evaluate the expected value of the adjustment strategy on the training effect of the target large model, and output the value evaluation result; Performing reinforcement learning training on the policy network according to the value assessment result, and optimizing the parameters of the policy network until the parameters of the policy network converge; Generate parameter optimization strategy of the target large model based on the policy network after parameter optimization.

4. The method according to claim 3, characterized in that The strategy network corresponds to a preset action space, and the preset action space includes a preset adjustment strategy for the training parameters of the target large model. The operation state vector is input into the strategy network to generate an adjustment strategy for the training parameters of the target large model, including: Updating parameters of the policy network based on a preset adjustment strategy in the preset action space; Based on the updated policy network parameters, multiply scale or additively offset the current learning rate within a preset range, where the preset range is dynamically adjusted based on the large model training stage; Select a new batch size from a predefined set of discrete values, or dynamically calculate the optimal batch size based on memory utilization; The characteristics of the running state vector are analyzed to determine the update step scaling factors and dynamic priorities of different layers or parameter groups.

5. The method according to claim 1, wherein The preset indicators include: the reduction of loss value, the improvement of accuracy, and the speed of model convergence. The reward function is used to calculate multiple preset indicators of the optimized large model as the reward result of the parameter optimization strategy, including: If the decrease in the loss value exceeds the preset threshold, the increase in accuracy is greater than the preset threshold, and the model convergence speed is greater than the preset convergence speed threshold, then the reward result is a positive reward; If the decrease in the loss value fluctuates, or the increase in accuracy is less than the preset threshold, or the model convergence speed is less than the preset convergence speed threshold, the reward result is a negative reward.

6. The method according to claim 1, characterized in that Optimizing the parameters of the target large model based on the parameter optimization strategy to obtain an optimized large model includes: During the training of the target large model, the parameter optimization agent adjusts the parameter optimization strategy in real time according to the running state vector and reward results, and adjusts the parameters of the target large model based on the parameter optimization strategy; After the target large model is deployed, the operating status data in the deployment environment is obtained, and the parameter optimization agent adaptively adjusts the parameters of the target large model according to the operating status data.

7. A large model parameter optimization and adaptive adjustment device based on reinforcement learning, characterized in that: The device comprises: An operation state vector determination module is used to extract features from input data to obtain data features, obtain the current operation state of the target large model, and perform a comprehensive evaluation of the operation state of the target large model based on the data features to obtain an operation state vector; An optimization strategy determination module is used to construct a parameter optimization agent for the target large model, and input the operating state vector into the parameter optimization agent to obtain a parameter optimization strategy; A parameter optimization module, configured to optimize the parameters of the target large model based on the parameter optimization strategy to obtain an optimized large model; A reward calculation module, configured to calculate a plurality of preset indicators of the optimization model using a reward function as a reward result for the parameter optimization strategy; An adaptive adjustment module is used to adaptively adjust the parameters of the parameter optimization agent based on the reward result.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method according to any one of claims 1 to 6.

Images