Reinforcement learning-based cascaded fusion model inference optimization method and inference method

Abstract

The application discloses a reinforcement learning-based cascading fusion model inference optimization method and inference method. The optimization method comprises the following steps: first, input information in a user model service API call request is received and analyzed, and the input information is input into a policy network model to obtain a scheduling strategy, on the basis of which a target model is determined and a call request is generated, and first and second inference results are obtained by using the target model and a large-parameter model respectively; a difference of tokens used for inference of the two models is calculated, and a comprehensive reward function is determined in combination with the inference results; and parameters of the policy network model are updated according to the function, so that the model reaches the best performance in corresponding tasks and scenes. According to the method, the inference result can be directly obtained by using the updated policy network model, and the large-parameter model does not need to be called every time, thereby significantly reducing irrelevant calculation overhead, improving inference speed and throughput, effectively reducing GPU resource consumption on the premise that the output semantic quality is basically stable, and realizing efficient and energy-saving model inference.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and large parameter model inference technology, and in particular to a cascaded fusion model inference optimization method and inference method based on reinforcement learning. Background Technology

[0002] In today's field of artificial intelligence, large parameter models have been widely applied in reasoning and generation tasks due to their powerful capabilities. Whether it's text generation, image creation, or complex logical reasoning, large parameter models have demonstrated remarkable high accuracy and strong generalization ability. This capability enables large parameter models to play a crucial role in many industries. For example, intelligent customer service can accurately understand user questions and generate reasonable responses, and content creation can quickly generate high-quality copy, highlighting their increasing value.

[0003] The accuracy of a model is usually closely related to the number of parameters. Large-scale parametric models, due to their abundant number of parameters, can learn more complex patterns and rules, thus possessing stronger inference performance and generative capabilities. However, this advantage is also accompanied by significant drawbacks. Large-scale parametric models require massive computing resources to run, which not only increases hardware costs but also leads to longer response times, limiting their application in scenarios with high real-time requirements.

[0004] In contrast, small-parameter models are characterized by low resource consumption and fast response, enabling them to run quickly and produce results on resource-constrained devices, making them suitable for scenarios with high real-time requirements and relatively scarce computing resources. However, when faced with complex inference tasks, small-parameter models often fall short in terms of generation capabilities, making it difficult to meet practical needs. Summary of the Invention

[0005] In view of this, the present disclosure provides a cascaded fusion model inference optimization method and inference method based on reinforcement learning, which can balance inference performance, accuracy and computational cost.

[0006] In a first aspect, embodiments of this disclosure provide a method for optimizing inference in a cascaded fusion model based on reinforcement learning, including: Receive user's model service API call request and parse out the user input information in the request; The user input information is input into the policy network model to obtain the model scheduling policy; Determine the target model corresponding to the model scheduling strategy, and generate an API request to call the target model according to the model scheduling strategy; The API request is used to invoke the target model to obtain the first inference result of the user input information; A second inference result is obtained based on the user input information using a large parameter model; Based on the token used in the inference process of the target model and the token used in the inference process of the large parameter model, obtain the token difference; Based on the first reasoning result, the second reasoning result, and the token difference, determine the comprehensive reward function; The parameters of the policy network model are updated according to the comprehensive reward function.

[0007] Secondly, this disclosure also provides a reasoning method, including: Obtain the rate of change of the current parameters in the latest policy network model compared to the parameters before the last update; If the rate of change does not meet the preset change conditions, the cascaded fusion model inference optimization method based on reinforcement learning disclosed in the first aspect of the present disclosure is invoked to optimize the latest policy network model; The user input information is analyzed based on the optimized policy network model to obtain the current model scheduling policy; Determine the current target model corresponding to the current model scheduling strategy; The reasoning results obtained based on the current target model are obtained from the user's input information and then fed back to the user.

[0008] Thirdly, this disclosure also provides a computer device, which adopts the following technical solution: The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform any of the reinforcement learning-based cascaded fusion model inference optimization methods described above.

[0009] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to execute any of the reinforcement learning-based cascaded fusion model inference optimization methods described above.

[0010] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0011] The reinforcement learning-based cascaded fusion model inference optimization method disclosed in this application first receives a user's model service API call request, parses the user input information in the request, inputs the user input information into a policy network model, obtains a model scheduling policy, determines the target model corresponding to the model scheduling policy, and generates an API request to call the target model according to the model scheduling policy. The first inference result based on the user input information is obtained by calling the target model according to the API request. The policy network model can dynamically generate a suitable model scheduling policy based on the user input, effectively avoiding the limitation of using a fixed model for inference. It can select a more suitable model and parameters according to different inputs, improving inference efficiency and accuracy. Then, a second inference result based on a large parameter model is obtained. Large parameter models usually have higher accuracy and richer knowledge reserves, and their inference results can serve as a more accurate reference. Finally, the tok used in the inference process is executed according to the target model. The tokens used in the inference process of the large parameter model are used to obtain the token difference, which reflects the difference in resource consumption of different models during the inference process. Finally, based on the first inference result, the second inference result, and the token difference, the comprehensive reward function is determined, and the parameters of the policy network model are updated according to the comprehensive reward function. That is, the model parameters of the policy network are continuously optimized and updated based on the comprehensive reward function. The comprehensive reward function can comprehensively evaluate the merits of the model scheduling strategy. It not only considers the accuracy of the inference results, but also the factors of resource consumption, providing a clear goal for the optimization of the policy network model. By updating the parameters of the policy network model according to the comprehensive reward function, it can achieve the best performance in different tasks and scenarios without calling the large parameter model every time. This significantly reduces the irrelevant computational overhead during model inference, improves inference speed and throughput, and effectively reduces GPU resource consumption while ensuring that the output semantic quality is not significantly degraded.

[0012] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1This is a flowchart illustrating the inference optimization method for a cascaded fusion model based on reinforcement learning provided in an embodiment of this disclosure.

[0015] Figure 2 This is a flowchart illustrating the method for obtaining the model scheduling strategy provided in an embodiment of this disclosure.

[0016] Figure 3 A flowchart illustrating the method for determining the comprehensive reward function provided in this embodiment of the disclosure.

[0017] Figure 4 This is a flowchart illustrating the reasoning method provided in an embodiment of the present disclosure.

[0018] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. Detailed Implementation

[0019] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0020] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0022] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0023] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0024] Reference Figure 1 This application discloses a cascaded fusion model inference optimization method based on reinforcement learning, including: S100 receives the user's model service API call request and parses out the user input information in the request (i.e., the user input information of the question asked by the user).

[0025] S200: Input the user input information into the policy network model to obtain the model scheduling policy.

[0026] In this embodiment, the policy network model is a lightweight neural network model based on reinforcement learning, and the model scheduling policy includes scheduling actions and scheduling parameters. The policy network model can dynamically generate appropriate model scheduling policies based on user input, effectively avoiding the limitations of using a single model for inference. It can select more suitable models and parameters based on different inputs, improving inference efficiency and accuracy.

[0027] S300: Determine the target model corresponding to the model scheduling strategy, and generate an API request to call the target model according to the model scheduling strategy.

[0028] This step clarifies the specific target model to be used and generates appropriate API requests, preparing for accurate invocation of the target model for inference. It ensures that the correct model is used for processing according to the scheduling requirements of the policy network model.

[0029] S400 calls the target model based on the API request to obtain the first inference result of the user input information.

[0030] S500 is a second inference result obtained based on a large parameter model to obtain user input information.

[0031] Large parameter models typically have higher accuracy and richer knowledge reserves. Their inference results can serve as a more accurate reference, and can be compared and integrated with the first inference result, which helps to improve the quality of the final result.

[0032] S600: Obtain the token difference based on the token used in the inference process of the target model and the token used in the inference process of the large parameter model.

[0033] The token difference reflects the difference in resource consumption among different models during the inference process. Smaller models typically use fewer tokens, while models with larger parameters use more tokens. By calculating the token difference, this difference in resource consumption can be quantified, providing an important metric for subsequent optimization.

[0034] S700 determines the comprehensive reward function based on the first inference result, the second inference result, and the token difference.

[0035] The comprehensive reward function can comprehensively evaluate the merits of the model scheduling strategy. It not only considers the accuracy of the inference results but also the resource consumption factors, providing a clear objective for the optimization of the policy network model.

[0036] S800 updates the parameters of the policy network model based on the comprehensive reward function, that is, it continuously optimizes and updates the model parameters of the policy network based on the comprehensive reward function.

[0037] By continuously updating the parameters of the policy network model based on the comprehensive reward function, the model scheduling strategy can be continuously optimized. Over time, the policy network model will become increasingly intelligent, able to select the optimal model and parameters based on different user inputs, thereby improving the inference efficiency and accuracy of the entire system.

[0038] The reinforcement learning-based cascaded fusion model inference optimization method disclosed in this application dynamically selects a suitable target model for initial inference through a policy network model. The rapid inference capability of the policy network model can provide a rough answer in a short time, meeting the user's need for rapid response. The results of the large parameter model are only used as a reference. At the same time, token difference and comprehensive reward function are used to encourage the use of lightweight models with lower resource consumption for inference. While ensuring a certain level of inference accuracy, the reliance on large parameter models is reduced, thereby reducing the overall computational resource consumption and cost. By continuously updating the parameters of the policy network model according to the comprehensive reward function, the entire system can automatically learn and adapt to different user inputs and scenarios, continuously optimize the model scheduling strategy, and improve the system's performance and user experience.

[0039] The method disclosed in this application combines lightweight prompt optimization of policy network models with high-precision inference of large models to form a collaborative inference system. By updating the parameters of the policy network model according to the comprehensive reward function, it can achieve optimal performance in different tasks and scenarios without calling the large parameter model every time. This significantly reduces irrelevant computational overhead during model inference, improves inference speed and throughput, and effectively reduces GPU resource consumption while ensuring that the semantic quality of the output is not significantly degraded. The method has good portability and can be applied to various inference optimization scenarios of large language models.

[0040] In this embodiment, the inference service of the policy network is deployed via API. Upon receiving a user request, the model scheduler first parses the prompt and constructs a state vector, then calls the policy network to obtain the action and parameters. The policy network is a neural network model; its input is the user's prompt, and its output is the action and its corresponding parameters. To balance speed and local semantic extraction capabilities, the policy network model preferably adopts a lightweight structure based on a 1D-CNN (one-dimensional convolutional neural network), outputting a hybrid discrete / continuous action space.

[0041] Reference Figure 2 The method for S200, "inputting user input information into the policy network model to obtain the model scheduling policy," specifically includes the following methods for obtaining the model scheduling policy: S210 receives user input information (i.e., the original Prompt text) through the input layer of the policy network model and encodes the user input information into a dense vector representation to form a time series matrix.

[0042] This step transforms the text into a matrix. Specifically, dense vector representation can preserve the semantic information of the text to a certain extent, with each vector element potentially representing certain semantic features of the text. Temporal matrix, on the other hand, can reflect the sequential relationship between words or phrases in the text, helping the model learn the contextual information and semantic structure of the text.

[0043] S220 extracts features from the temporal matrix through the feature extraction layer of the policy network model, obtaining local feature vectors containing local semantic information of different granularities.

[0044] Specifically, in the feature extraction stage, the model first uses a multi-kernel convolutional structure to process the temporal matrix output by the embedding layer. This structure uses convolutional kernels of different sizes (1, 3, 5, 7) in parallel to capture local semantic patterns at different granularities. Next, batch normalization is applied to the convolutional outputs to eliminate differences in feature distribution. Then, a non-linear activation function is used to introduce non-linear characteristics, followed by a pooling operation to integrate the multiple convolutional outputs. To enhance gradient flow and model stability, residual blocks are used to process the pooling results. The feature vectors obtained after these operations contain rich local semantic information but lack modeling of long-range dependencies. The self-attention layer further processes these feature vectors that have already extracted local features.

[0045] In this step, the feature extraction layer uses operations such as multi-kernel convolution structure, batch normalization, non-linear activation, convergence and residual blocks to capture local semantic patterns at different granularities. These local semantic information reflect the semantic features within a small range of the text, such as the meaning implied by some phrases and local sentence structures.

[0046] S230 uses the self-attention layer of the policy network model to perform self-attention calculation based on the correlation between different local feature vectors, thereby obtaining several global feature vectors containing global contextual information of the text.

[0047] The sequence length of all global feature vectors is the same as the sequence length of all local feature vectors. Self-attention calculation involves weighted summation of the vectors at each position, with the weights reflecting the degree of correlation between that position and other positions.

[0048] In this step, the self-attention layer fuses local semantic information and long-distance dependency information, resulting in a feature vector that contains richer and more comprehensive information, thereby capturing the global contextual information of the text. Specifically, the self-attention layer performs a linear transformation on the input feature vector, generating three matrices: query, key, and value. An attention score is obtained by calculating the similarity between the query matrix and the key matrix. Next, the attention score is normalized to obtain the attention weights. Finally, the attention weights are multiplied by the value matrix to obtain the feature vector processed by the self-attention mechanism. This new feature vector not only retains local semantic information but also incorporates long-distance dependency information; that is, each output vector integrates information from other relevant positions in the sequence, thus including long-distance dependency information. This makes the feature representation of each vector richer and more comprehensive, better reflecting the overall semantics and structure of the sequence.

[0049] S240, several global feature vectors are input into a shared hidden layer for fusion to obtain several target feature vectors containing local details and overall semantics of the text.

[0050] Specifically, a linear transformation is performed on the vector output by the self-attention layer, for the matrix O∈R of the self-attention layer output... n×d (where n is the sequence length and d is the vector dimension), the shared hidden layer uses a weight matrix W∈R d×h (h is the number of hidden layer neurons) and a bias vector b∈R h The calculation of linear combination and linear transformation can be expressed as Z = OW + b, where Z ∈ R. n×h This is the result of a linear transformation, which projects the input vector into a new feature space, enabling more effective interactions between different features. After the linear transformation, a non-linear activation function, such as ReLU (Rectified Linear Unit), is typically applied. This non-linear activation function introduces non-linear characteristics into the model, allowing it to learn more complex patterns and relationships. After the linear transformation and non-linear activation, the shared hidden layer fuses the resulting feature vectors. Since the vector output by the self-attention layer already contains local semantics and global contextual information, the shared hidden layer further integrates this information. In the fully connected layer structure, feature vectors at different locations interact through neuron connections, allowing for a full fusion of local and global information.

[0051] S250: Several target feature vectors are input into the policy head and the value head respectively. The policy head generates a model scheduling policy for the hybrid action space based on the several target feature vectors.

[0052] The value head outputs a value estimate of the current state based on several target feature vectors, which is used for advantage function calculation and policy optimization in reinforcement learning. In this embodiment, the policy head and value head are typically also fully connected layers.

[0053] In this embodiment, the scheduling action includes a first type of scheduling decision, a second type of scheduling decision, or a third type of scheduling decision; the first type of scheduling decision includes calling a small parameter model, and the corresponding scheduling parameters include the relevant parameters for calling the small parameter model.

[0054] The second type of scheduling decision includes calling a large parameter model, and the corresponding scheduling parameters include the relevant parameters for calling the large parameter model.

[0055] The third type of scheduling decision includes calling small parameter models and large parameter models, and the corresponding scheduling parameters include the relevant parameters for calling small parameter models and large parameter models.

[0056] In this embodiment, taking the OpenAI API as an example, the messages.content field in the request body is parsed; the input of the policy network model is the user's Prompt, and the output is Action and Parameters. The enumeration value of Action is [1, 2, 3]. Parameters consists of two parts: one part is the model call parameters model_call_params, including [temperature, top_p, max_tokens, presence_penalty], and the other part is the Prompt optimization parameters prompt_optimize_params, including [context_simp_ratio, core_objective_simp_ratio].

[0057] If the value of Action is 1, it means that the request can directly call the small parameter model for inference. In the format of openaiAPI, an API request to call the small model is generated. The model parameter value of the request is the model name of the small parameter model, the message parameter value is the prompt input by the user, and the temperature, top_p, max_tokens, and presence_penalty parameters are the corresponding parameter values ​​in the model_call_params part of Parameters.

[0058] If the value of Action is 2, it means that the request can directly call the large parameter model for inference. In the format of openaiAPI, an API request to call the large model is generated. The model parameter value of the request is the model name of the large parameter model, the message parameter value is the prompt input by the user, and the temperature, top_p, max_tokens, and presence_penalty parameters are the corresponding parameter values ​​in the model_call_params part of Parameters.

[0059] If the Action value is 3, it means that the request is performed through inference using a cascaded fusion of small and large models. First, an API request to call the small model is generated in the format of the OpenAI API. The model parameter of the request is the model name of the small parameter model, and the messages parameter is the prompt generated using the prompt template.

[0060] The method for S300, which "determines the target model corresponding to the model scheduling strategy and generates an API request to call the target model based on the model scheduling strategy," includes: If the decision corresponding to the model scheduling strategy is the first type of scheduling decision, the target model is determined to be a small parameter model. A small parameter model is a model with a scale of less than 10B (i.e., 10 billion) and mainly handles some simple inference tasks.

[0061] If the decision corresponding to the model scheduling strategy is the second type of scheduling decision, the target model is determined to be a large parameter model. A large parameter model is a model with a scale of 100B (i.e., 100 billion) or more, which mainly handles some complex inference tasks.

[0062] If the decision corresponding to the model scheduling strategy is a third type of scheduling decision, the target model is determined to be a cascaded fusion model composed of a small parameter model and a large parameter model.

[0063] The S600 method of "obtaining the token difference based on the tokens used in the inference process of the target model and the tokens used in the inference process of the large parameter model" includes the following methods for obtaining the token difference: S610, obtain the difference between the token used by the target model in the inference process and the token used by the large parameter model in the inference process; S620, normalize the difference to obtain the token difference.

[0064] Among them, the token difference = difference ÷ second inference result. The value of the token difference is also between [0,1]. The larger the value, the larger the difference, and the more the large model inference resource overhead and latency are saved.

[0065] Reference Figure 3 The method for S700 to "determine the comprehensive reward function based on the first inference result, the second inference result, and the token difference" includes the following methods for determining the comprehensive reward function: S710, obtain the semantic similarity between the first reasoning result and the second reasoning result.

[0066] Specifically, the cosine similarity algorithm is used to obtain the semantic similarity (i.e., text similarity) between two inference results, with values ​​between [0,1]; the larger the value, the closer the semantics.

[0067] S720, obtains the task completion rate based on the first reasoning result.

[0068] Specifically, the methods for obtaining the task completion score Sim include: 1) obtaining a feedback score based on the user's feedback on the first inference result; 2) normalizing the feedback score to obtain the task completion score.

[0069] In this embodiment, the feedback value given by the user is generally between [0, 100]. By using Sim = user feedback value / 100, the value of Sim is also between [0, 1]. The larger the value, the more it meets the user's expectations.

[0070] S730 determines the comprehensive reward function based on semantic proximity, task completion, and token difference.

[0071] The overall reward function is R t :R t =α×Acc+β×Sim +γ×Perf;α+β+γ=1; Where Acc represents semantic proximity, Sim represents task completion, Perf represents token difference, and α, β, and γ represent the weights of the first, second, and third dimensions, respectively.

[0072] Generally, the most important factors are the completeness of model inference and the output effect. Therefore, it is preferable that α and β are the same and both are greater than γ. If you are more concerned about reducing model resource consumption and response latency, you can appropriately increase the value of γ and decrease the values ​​of α and β.

[0073] The S800 method for "updating the parameters of the policy network model based on the comprehensive reward function" specifically includes: S810, Determine the policy gradient loss function : .

[0074] in, For the first A state, It is in the Actions to be taken in each state It is with state-action pairs ( , The corresponding comprehensive reward function, These are the parameters to be updated in the policy network model. For parameters are Under the policy network in state Take action The probability distribution, It is an action In state The logarithmic probability below.

[0075] S820, Obtain the policy gradient loss function gradient with respect to model parameters θ .

[0076] gradient is : .

[0077] S830 updates the parameters of the policy network model according to the gradient.

[0078] The parameter update formulas for gradient update strategy network models include: ,in, The learning rate (controls the step size for each parameter update). These are the parameters to be updated in the policy network model. To and The corresponding updated parameter, b, is the reward baseline (preferably the historical average reward, i.e., the average of the historical comprehensive reward function).

[0079] By introducing a baseline b, gradient variance can be effectively reduced and training stability enhanced; actions with rewards higher than the baseline will be strengthened, while actions with rewards lower than the baseline will be weakened.

[0080] Reference Figure 4 Secondly, this application discloses a reasoning method, including: S10: Obtain the rate of change of the current parameters in the latest policy network model compared to the parameters before the last update.

[0081] By obtaining the rate of change of parameters, we can understand the updating status of the policy network model. If the rate of change is too large, it may indicate that the model update is unstable and the optimization strategy needs to be adjusted. This step provides an important basis for subsequent judgment on whether model optimization is necessary.

[0082] S20, if the rate of change does not meet the preset change conditions, the aforementioned reinforcement learning-based cascaded fusion model inference optimization method is invoked to optimize the latest policy network model.

[0083] Specifically, determine the parameters in the latest policy network model. Parameters compared to the last update Does the rate of change corresponding to the difference between them meet the preset change conditions? If not, it means that the parameter update of the policy network model needs to be triggered.

[0084] Where, rate of change = ( - ) ÷ The preset change condition is preferably less than 1%.

[0085] When the rate of change of parameters in the policy network model does not meet the preset conditions, it indicates that the model's performance may not have been effectively improved. By calling the cascaded fusion model inference optimization method, we can take advantage of its advantages, such as dynamically selecting a suitable model, optimizing the prompt, and comprehensively considering inference results and resource consumption, to further optimize the policy network model so that it can better adapt to different tasks and scenarios.

[0086] In this application, the reinforcement learning-based cascaded fusion model inference optimization method disclosed in the first aspect of this application does not trigger optimization every time inference is used, but only when a custom condition is met, to ensure the accuracy of the reward, that is, to ensure the optimal performance of the latest policy network model. This satisfies the requirements of high-quality inference, while avoiding the need to call a large parameter model for inference analysis every time, thus ensuring high inference efficiency while minimizing resource consumption.

[0087] S30: Analyze user input information based on the optimized policy network model to obtain the current model scheduling policy.

[0088] Specifically, the system receives user requests for model service API calls, parses out user input information from the requests, inputs the user input information into the optimized policy network model, and obtains the current model scheduling policy.

[0089] The optimized policy network model can analyze user input more accurately and generate more suitable model scheduling strategies based on different tasks and scenarios. This helps to select the most suitable model for inference, improve the efficiency and accuracy of inference, and avoid resource waste and inaccurate inference results caused by using inappropriate models.

[0090] S40, determine the current target model corresponding to the current model scheduling strategy.

[0091] The method for determining the current target model in this step is consistent with the method for determining the target model in the reinforcement learning-based cascade fusion model inference optimization method disclosed in the first aspect of this application, so it will not be described in detail here.

[0092] This step clarifies the specific target model to be used, making the reasoning process more targeted. Different models have different characteristics and applicable scenarios. Choosing the appropriate target model can give full play to its advantages and improve the quality and efficiency of reasoning.

[0093] S50: Obtain the reasoning results based on the current target model and provide feedback to the user.

[0094] Specifically, an API request to invoke the current target model is generated based on the current model scheduling strategy. The API request is then used to invoke the target model to obtain the inference result based on the user's input information, which is then fed back to the user. This allows users to obtain inference results about their input information in a timely manner, meeting their needs. Furthermore, by selecting an appropriate target model for inference, the speed and efficiency of model inference are effectively improved while ensuring high inference quality, thus enhancing the user experience.

[0095] In this application, by monitoring the rate of change of parameters of the policy network model, optimization is performed in a timely manner when the rate of change does not meet the preset conditions, ensuring that the policy network model can continuously learn and adapt to different tasks and scenarios, and continuously improve its performance. Based on the optimized policy network model, a suitable model scheduling strategy is generated to select the most suitable target model for inference, without having to call a large parameter model for processing every time. This effectively avoids the waste of resources and inaccurate inference results caused by blindly using models, and improves the efficiency and accuracy of inference.

[0096] The inference method disclosed in this application generates a scheduling strategy through a policy network model, which can flexibly select the target model according to the complexity of the task. For simple tasks, it prioritizes the use of small parameter models, reducing unnecessary calls to large parameter models and significantly reducing hardware costs and resource consumption. In resource-limited environments, it can dynamically allocate resources. Small parameter models have low resource consumption and fast response, and can run efficiently on resource-limited devices. Large parameter models have advantages when handling complex tasks. This method combines the characteristics of both, making resource allocation more reasonable and improving overall resource utilization.

[0097] In real-world applications, tasks are complex and diverse, with varying requirements for model accuracy, speed, and resource consumption. This method dynamically selects the appropriate target model based on task characteristics, ensuring good system performance across different tasks and scenarios, thus enhancing the system's adaptability and flexibility. By updating the parameters of the policy network model through a comprehensive reward function, the model can continuously learn and adapt to new tasks and scenarios. As training progresses, the policy network model can more accurately determine task requirements and select the optimal model scheduling strategy, improving the overall system performance. Traditional methods require manual model selection based on task characteristics, which is inefficient and difficult to guarantee accuracy. This method utilizes reinforcement learning to automatically adjust the model scheduling strategy, effectively reducing manual intervention and improving the system's automation and operational efficiency.

[0098] Furthermore, this application also includes continuous monitoring of the changing trend of average reward, the stability of policy distribution, and whether latency and resource consumption meet constraints. If policy parameter updates are detected to cause a significant decrease in inference accuracy or an abnormal increase in resource consumption, the system will trigger policy rollback or adjust the learning rate to ensure that the convergence process is smooth and controllable.

[0099] To analyze the trend of average reward, specifically, after each training cycle or a certain number of inference tasks are completed, the total reward obtained by all inference tasks in that phase is calculated and divided by the number of tasks to obtain the average reward; a time series graph is used to record the change of average reward with training cycles.

[0100] For stability analysis of policy distribution, the model scheduling policy output by the policy network model can be viewed as a probability distribution, such as the probability of selecting different target models. Stability is measured by calculating the entropy of the policy distribution. A larger entropy value indicates a more dispersed policy distribution and poorer stability; a smaller entropy value indicates a more concentrated policy distribution and better stability. After each training cycle or a certain number of inference tasks, the entropy of the current policy distribution is calculated and compared with historical entropy values. If the change in entropy exceeds a preset threshold, it indicates a significant change in the policy distribution, potentially indicating instability.

[0101] For latency and resource consumption analysis, record the time from initiating a request to obtaining the inference result for each inference task and calculate the average latency. Performance monitoring tools can be used to accurately measure latency; monitor resource consumption metrics such as GPU utilization and memory usage. These metrics can be obtained using system monitoring tools or resource monitoring interfaces provided by deep learning frameworks; set constraints on latency and resource consumption, such as an average latency not exceeding 100 milliseconds and GPU utilization not exceeding 80%. After each inference task is completed, check whether these constraints are met; if not, it indicates abnormal resource consumption.

[0102] For a significant drop in inference accuracy, specific detection methods include: after each training cycle or a certain number of inference tasks, using a set of test data to evaluate the model's inference accuracy. If the current inference accuracy drops by more than a preset threshold (e.g., 5%) compared to the previous cycle, it is considered a significant drop in inference accuracy. When a significant drop in inference accuracy is detected, the system triggers policy rollback, restoring the parameters of the policy network model to the state before the last update. Policy rollback can be achieved by saving historical versions of the model parameters. Besides policy rollback, the learning rate can also be adjusted. If the drop in inference accuracy is due to an excessively high learning rate causing model instability, the learning rate can be appropriately reduced. For example, the current learning rate can be multiplied by a coefficient less than 1, and then training can continue.

[0103] When latency or resource consumption fails to meet preset constraints, an abnormal increase in resource consumption is considered. Similar to a decrease in inference accuracy, the system triggers a policy rollback, restoring the policy network model's parameters to their state before the last update to attempt to resolve the abnormal resource consumption issue. If the abnormal increase in resource consumption is due to excessively rapid model updates, the learning rate can be appropriately reduced to make model updates more stable and reduce excessive resource consumption. Through these monitoring and processing mechanisms, the convergence process of the policy network model can be ensured to be smooth and controllable, avoiding performance degradation and resource waste caused by parameter updates.

[0104] A computer device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0105] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the reinforcement learning-based cascaded fusion model inference optimization method described in the foregoing embodiments of this disclosure.

[0106] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0107] like Figure 5 This is a schematic diagram of a computer device provided for an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the computer device in the embodiments of the present disclosure. Figure 5 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0108] like Figure 5 As shown, a computer device may include a processor (such as a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from storage devices into random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0109] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow the computer device to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 5A computer apparatus with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.

[0110] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the reinforcement learning-based cascaded fusion model inference optimization method of embodiments of this disclosure are performed.

[0111] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0112] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the reinforcement learning-based cascaded fusion model inference optimization method described in the foregoing embodiments of the present disclosure are performed.

[0113] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0114] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0115] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0116] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0117] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0118] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0119] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0120] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0121] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for optimizing inference in a cascaded fusion model based on reinforcement learning, characterized in that, include: Receive user's model service API call request and parse out the user input information in the request; The user input information is input into the policy network model to obtain the model scheduling policy; Determine the target model corresponding to the model scheduling strategy, and generate an API request to call the target model according to the model scheduling strategy; The API request is used to invoke the target model to obtain the first inference result of the user input information; A second inference result is obtained based on the user input information using a large parameter model; Based on the token used in the inference process of the target model and the token used in the inference process of the large parameter model, obtain the token difference; Based on the first reasoning result, the second reasoning result, and the token difference, determine the comprehensive reward function; Update the parameters of the policy network model according to the comprehensive reward function; The step of determining the comprehensive reward function based on the first inference result, the second inference result, and the token difference includes: obtaining the semantic similarity between the first inference result and the second inference result; obtaining the task completion degree based on the first inference result; and determining the comprehensive reward function based on the semantic similarity, the task completion degree, and the token difference; wherein the comprehensive reward function is R. t :R t =α×Acc+β×Sim +γ×Perf;α+β+γ=1;where, Acc is the semantic proximity, Sim is the task completion degree, Perf is the token difference, and α, β, and γ are the weights of the first, second, and third dimensions, respectively.

2. The reinforcement learning-based cascaded fusion model inference optimization method according to claim 1, characterized in that, The step of inputting the user input information into the policy network model to obtain the model scheduling policy includes: The user input information is received through the input layer of the policy network model, and the user input information is encoded into a dense vector representation to form a time series matrix; The temporal matrix is ​​used to extract features through the feature extraction layer of the policy network model to obtain local feature vectors containing local semantic information of different granularities. The self-attention layer of the policy network model performs self-attention calculation based on the correlation between different local feature vectors to obtain several global feature vectors containing global text context information. Several global feature vectors are input into a shared hidden layer for fusion to obtain several target feature vectors containing local details and overall semantics of the text; Several target feature vectors are respectively input into the policy head, and the policy head generates a model scheduling policy for a hybrid action space based on the several target feature vectors.

3. The reinforcement learning-based cascaded fusion model inference optimization method according to claim 2, characterized in that, The model scheduling strategy includes a first type of scheduling decision, a second type of scheduling decision, or a third type of scheduling decision; The first type of scheduling decision includes invoking the small parameter model, and the corresponding scheduling parameters include the relevant parameters for invoking the small parameter model; The second type of scheduling decision includes calling a large parameter model, and the corresponding scheduling parameters include the relevant parameters for calling the large parameter model; The third type of scheduling decision includes calling small parameter models and large parameter models, and the corresponding scheduling parameters include the relevant parameters for calling small parameter models and large parameter models.

4. The reinforcement learning-based cascaded fusion model inference optimization method according to claim 1, characterized in that, The step of determining the target model corresponding to the model scheduling strategy and generating an API request to call the target model according to the model scheduling strategy includes: If the decision corresponding to the model scheduling strategy is a first type of scheduling decision, the target model is determined to be a small parameter model, wherein the small parameter model is a model with a scale of less than 10B; If the decision corresponding to the model scheduling strategy is a second type of scheduling decision, the target model is determined to be a large parameter model, and the large parameter model is a model with a scale of more than 100B. If the decision corresponding to the model scheduling strategy is a third type of scheduling decision, the target model is determined to be a cascaded fusion model composed of a small parameter model and a large parameter model.

5. The reinforcement learning-based cascaded fusion model inference optimization method according to claim 1, characterized in that, The step of obtaining the token difference based on the token used in the inference process of the target model and the token used in the inference process of the large parameter model includes: Obtain the difference between the token used in the inference process of the target model and the token used in the inference process of the large parameter model; The difference is normalized to obtain the token difference.

6. The reinforcement learning-based cascaded fusion model inference optimization method according to claim 5, characterized in that, The step of obtaining the task completion score Sim based on the first reasoning result includes: A feedback score is obtained based on the user's feedback on the first reasoning result; The feedback score is normalized to obtain the task completion rate.

7. The reinforcement learning-based cascaded fusion model inference optimization method according to claim 5, characterized in that, The step of updating the parameters of the policy network model according to the comprehensive reward function includes: Determine the policy gradient loss function; The policy gradient loss function is: : , For the first One state, It is in the Actions to be taken in each state It is with state-action pairs ( , The corresponding comprehensive reward function, These are the parameters to be updated in the policy network model. For parameters are Under the policy network in state Take action The probability of; Obtain the gradient of the policy gradient loss function with respect to the model parameters; The gradient is : ; Update the parameters of the policy network model according to the gradient; The parameter update formula for the policy network model includes: ,in, For learning rate, To and The corresponding updated parameters.

8. A reasoning method, characterized in that, include: Obtain the rate of change of the current parameters in the latest policy network model compared to the parameters before the last update; If the rate of change does not meet the preset change conditions, the cascaded fusion model inference optimization method based on reinforcement learning as described in any one of claims 1-7 is invoked to optimize the latest policy network model; The optimized policy network model is used to analyze user input information and obtain the current model scheduling policy. Determine the current target model corresponding to the current model scheduling strategy; The reasoning results obtained based on the current target model are obtained from the user's input information and then fed back to the user.

9. A computer device, characterized in that, The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the reinforcement learning-based cascaded fusion model inference optimization method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the reinforcement learning-based cascaded fusion model inference optimization method as described in any one of claims 1-7.

11. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1-7.

Images