Battery purchase strategy adaptive optimization method fusing reinforcement learning

Abstract

The invention relates to the technical field of strategy optimization, in particular to a reinforcement learning-fused battery purchase strategy adaptive optimization method, which comprises the steps of generating an original battery purchase strategy by utilizing a pre-trained reinforcement learning model based on a current market state; inputting the original battery purchasing strategy into a strategy simulator for constraint conformity simulation, and obtaining a simulation execution result; wherein the strategy simulator is internally provided with a plurality of business constraint rules which can be dynamically configured; when the simulation execution result triggers any business constraint rule, parameter correction is carried out on the original battery purchasing strategy according to the triggered business constraint rule, and the corrected battery purchasing strategy is output and executed; and otherwise, directly executing the original battery purchasing strategy. According to the invention, by combining the comparison of the simulation execution track and the built-in business constraint rule, the battery purchasing strategy is effectively ensured to maximize the cost benefit and strictly comply with the business constraint.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of policy optimization, in particular to a battery procurement policy adaptive optimization method fusing reinforcement learning. BACKGROUND

[0002] With the rapid development of new energy vehicles and energy storage industries, battery procurement has become a core link of the supply chain management of related enterprises. Traditional battery procurement optimization methods are mainly based on economic order quantity model, static inventory control strategy and other classic theories. These methods have certain effect in stable market environment. However, the battery procurement environment has high dynamicity and complexity, which is characterized by frequent fluctuations in raw material prices, uncertain changes in supplier reliability and rapid technology iteration, which greatly reduces the adaptability of traditional static optimization methods.

[0003] The model training target of the existing procurement scheme based on reinforcement learning is usually limited to a single economic indicator such as cost minimization or revenue maximization. However, various non-quantitative constraints existing in the real business environment, such as supplier concentration risk, cash flow safety boundary, procurement rhythm stability and other key factors, are often ignored or only handled through simple reward function penalty terms. This handling method may generate procurement strategies that are optimal in economic indicators but have extremely high execution risks. At present, the industry's solution to this problem is to hard-code the constraint conditions into the reward function during model training, which makes the model training complex and opaque. Therefore, there is an urgent need for an innovative method that can maintain the efficiency of reinforcement learning decision-making while ensuring that the generated strategy meets the actual business constraints. SUMMARY

[0004] The present application provides a battery procurement policy adaptive optimization method fusing reinforcement learning. By introducing a policy simulator and a constraint compliance simulation mechanism, a closed-loop decision-making process of "generation-simulation-correction" is constructed, so that the reinforcement learning model can actively adapt to the dynamically changing business constraint rules while maintaining its optimization performance. This not only avoids the problem of hard-coding complex constraints into the reward function, which makes model training difficult, but also overcomes the efficiency bottleneck brought by manual review. Finally, under the premise of ensuring the business compliance and safety of the procurement strategy, the automation and intelligentization of the decision-making process are realized.

[0005] The battery procurement policy adaptive optimization method fusing reinforcement learning comprises the following steps: S1: Based on the current market state, a pre-trained reinforcement learning model is used to generate an original battery procurement policy; S2: The original battery procurement policy is input into the policy simulator for constraint compliance simulation, and the simulation execution result is obtained; wherein the policy simulator has multiple dynamically configurable business constraint rules built-in; S3: when the simulation execution result triggers any of the business constraint rules, then the original battery procurement strategy is parameterized according to the triggered business constraint rule, and the parameterized battery procurement strategy is output and executed; otherwise, the original battery procurement strategy is directly executed.

[0006] Optionally, the S1 comprises: S11: real-time collection and integration of multi-dimensional state information from internal inventory systems and external market data sources; S12: standardization processing of the multi-dimensional state information, to construct a current state vector matching the input layer dimension of the pre-trained reinforcement learning model; S13: inputting the current state vector into the pre-trained reinforcement learning model, performing forward inference in the model internal policy network, and outputting the procurement action with the highest probability from the pre-set discrete action space, which constitutes the original battery procurement strategy.

[0007] Optionally, the multi-dimensional state information comprises battery inventory level, raw material spot price, and supplier capacity utilization rate.

[0008] Optionally, the S13 comprises: S131: inputting the current state vector obtained after standardization processing into the pre-trained reinforcement learning model, performing forward inference process in the policy network inside the model, and obtaining the probability distribution of each procurement action in the pre-set discrete action space; S132: selecting the procurement action with the highest probability from the probability distribution output by the policy network as the original battery procurement strategy at the current time and outputting it.

[0009] Optionally, the S2 comprises: S21: loading the original battery procurement strategy as a driving instruction into the strategy simulator, and initializing the working condition based on the current market environment; S22: executing the original battery procurement strategy on the simulation time scale, and real-time tracking of the simulation execution trajectory, the simulation execution trajectory comprising procurement cost distribution, supplier selection distribution, and inventory level change curve; S23: real-time comparison of the simulation execution trajectory with the multiple dynamically configurable business constraint rules built-in the strategy simulator, when any index in the trajectory violates any business constraint rule, a constraint violation signal including specific violation rule identification and violation severity is generated; S24: summarizing all simulation execution trajectory data and the constraint violation signal within the entire simulation period to jointly constitute the simulation execution result.

[0010] Optionally, the S23 comprises: S231: In the simulation execution process, compare the generated simulation execution trajectory with the built-in commercial constraint rule set by time, and determine whether there is a violation of the commercial constraint rule in the simulation execution trajectory; S232: When any index in the simulation execution trajectory violates the commercial constraint rule, a constraint violation signal is generated, which includes the number of violated rules and the severity level of violation.

[0011] Optionally, the S24 comprises: S241: All simulation execution trajectory data and constraint violation signals in the entire simulation period are summarized to ensure that the execution process and constraint triggering information are complete; S242: The simulation execution trajectory data and the constraint violation signal obtained by summarizing are combined to form the final simulation execution result.

[0012] Optionally, the S3 comprises: S31: Analyze the simulation execution result to determine whether it includes a constraint violation signal; S32: If the constraint violation signal exists, according to the type of the triggered commercial constraint rule identified in the constraint violation signal, a pre-defined strategy correction rule library is called to select a corresponding parameter adjustment algorithm; S33: Based on the violation severity carried in the constraint violation signal, the parameter adjustment algorithm is used to quantitatively adjust the parameters in the original battery procurement strategy, and a corrected battery procurement strategy is generated; S34: The corrected battery procurement strategy is output to the procurement execution system; if the constraint violation signal does not exist, the original battery procurement strategy is directly output to the procurement execution system.

[0013] Optionally, the S32 comprises: S321: Analyze the type of the triggered commercial constraint rule identified in the constraint violation signal, and identify the specific constraint category by the rule number; S322: According to the identified constraint rule type, select the parameter adjustment algorithm corresponding to the constraint rule from the pre-defined strategy correction rule library.

[0014] Optionally, the S34 comprises: S341: When the constraint violation signal is detected, the corrected battery procurement strategy is output to the procurement execution system; S342: If no constraint violation signal is detected, the original battery procurement strategy is directly output to the procurement execution system to execute the original procurement decision.

[0015] The beneficial effects of the present application are: This invention achieves a high degree of adaptability in the generation and adjustment of battery procurement strategies by introducing reinforcement learning models and constraint compliance simulations. Through the analysis of simulation execution results and dynamic comparison of constraint rules, it can identify situations that may violate business constraints in procurement decisions in real time and automatically adjust the procurement strategy according to the degree of violation. This adaptive adjustment mechanism can ensure that procurement decisions are always consistent with market changes and business needs, thereby improving the flexibility of battery procurement strategies and minimizing the waste of procurement resources and cost losses caused by fixed rules.

[0016] This invention effectively ensures that battery procurement strategies not only maximize cost-effectiveness but also strictly adhere to business constraints by comparing simulated execution trajectories with built-in business constraint rules. Whether it's inventory levels, supplier selection, or other business constraint rules, all can be optimized through an automated strategy correction process. This invention significantly improves the compliance of procurement strategies, avoiding supply chain risks and over-purchasing caused by failure to adjust strategies in a timely manner, and effectively enhances the operational stability and decision-making efficiency of enterprises in the face of complex market environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Fig. 1 This is a flowchart of a method according to an embodiment of the present invention; Fig. 2 This is a flowchart illustrating the strategy correction process in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Those skilled in the art may employ other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0020] like Figs. 1-2 As shown, the adaptive optimization method for battery procurement strategy incorporating reinforcement learning includes the following steps: S1: Based on the current market situation, generate the original battery procurement strategy using a pre-trained reinforcement learning model; S1 specifically includes: S11: At time The multi-dimensional state information is obtained from an internal inventory system of the enterprise and an external market data source and fused to form an original state data set The original state data set includes three key elements, specifically including a battery inventory level , a raw material spot price , and a supplier capacity utilization rate , and is expressed as: ; wherein, is the number of battery inventories, directly read from an internal ERP / inventory management system of the enterprise, reflecting the current inventory level of the enterprise, and the value range is , is a market spot price of a main raw material, obtained through a raw material price API, a third-party market quotation platform, and a supplier quotation sheet, is a supplier capacity utilization rate, calculated according to the monthly capacity, order quantity, and delivery quantity provided by the supplier, , and the value range is , reflecting the degree of supply chain capacity tension; S12: Normalizing each dimension of the original state data to eliminate differences in different dimensions and mapping to an interval of to obtain a state vector conforming to the input requirements of a pre-trained reinforcement learning model , expressed as: ; wherein, denotes a standardization function, which can adopt linear normalization, Z-score standardization, or empirical boundary mapping, etc., to ensure input consistency, is the dimension of the state vector, determined by the input layer node number of the pre-trained model, consistent with the pre-trained model input, and the value range is - ; S13: Inputting the standardized state vector to a pre-trained reinforcement learning strategy network , calculating the probability distribution of each action in the action space through a forward inference process, and selecting the action with the highest probability as the original procurement strategy at the current time, expressed as: ; wherein, is a pre-trained reinforcement learning strategy network, is a parameter, and the reinforcement learning strategy network usually outputs an action probability distribution through a softmax layer, and the value range is while ensuring the probability sum is 1, so that the output can intuitively represent the relative optimality of each action, is a predefined discrete action space, each action corresponds to a typical combination of procurement decisions, the size of the action space needs to be balanced between expressing decision flexibility and computational complexity, If is too small, the procurement strategy is too rough to adapt to complex market changes; if is too large, the inference efficiency decreases and the training difficulty increases, so it is set to a combination of several typical procurement quantities, supplier selection and procurement time points, so that the action can cover common business needs and ensure computational feasibility, i.e. The value range is 5-20, is the optimal action output by the model at time , i.e. the original battery procurement strategy, which specifically embodies the decision content of procurement quantity, procurement time node and supplier selection. In the distribution output by the strategy network, selecting the optimal action (greedy decision) is the standard approach for reinforcement learning decision-making, which conforms to the inference mode of deep Q network and policy gradient method.

[0021] S2: input the original battery procurement strategy into the strategy simulator for constraint compliance simulation and obtain the simulation execution result; wherein the strategy simulator is built-in with multiple dynamically configurable business constraint rules; S2 specifically includes: S21: input the original battery procurement strategy as a driving instruction into the strategy simulator, and initialize the simulation working condition parameter set according to the current market environment state to form an initial input pair , wherein is the original battery procurement strategy, which is limited within the action space to ensure that the output is always an executable procurement strategy, avoiding the occurrence of unlandable results, and the number of actions is kept within a limited range, which can ensure strategy diversity and control computational complexity, is the market environment parameter set at time , including raw material prices, supplier capacity, and initial inventory, which comes from common market data distribution and enterprise inventory scale in the industry, to ensure that the simulation working condition is close to the actual operating environment; S22: execute the original battery procurement strategy on the set simulation time scale and generate a simulation execution trajectory , represented as: ; wherein ​is a set of key indicators evolving over time during the simulation, including the distribution curve of purchasing cost, the distribution of supplier selection, and the curve of inventory level over time, is the distribution curve of purchasing cost, i.e. the cumulative or unit purchasing cost at moment, the upper limit of purchasing cost considers the scenario of bulk material centralized procurement, and the value range is , is the distribution of supplier selection, i.e. the supplier selection vector at moment, reflecting the proportion of each supplier being selected, and the supplier distribution satisfies probability normalization, and the value range is , is the number of suppliers, is the curve of inventory level, i.e. the inventory level at moment, the upper limit of inventory level matches the warehousing capacity, and the value range is , is the simulation time step, and the value range is , the value range of is 30-365, and the simulation time range needs to cover at least one procurement period to observe the dynamic characteristics such as inventory consumption and supply change; S23: assuming that the built-in business constraint rule set in the strategy simulator is , the simulated execution trajectory needs to be compared with the business constraint rule set in real time during the simulation execution, when any indicator in the simulated execution trajectory violates the business constraint rule , a constraint violation signal is generated, which includes the rule number violated and the severity level of violation, and is represented as: ; wherein is the violation rule report output at time step , and is the set of business constraint rules, including cash flow constraints, upper limits of supplier share, upper limits of inventory safety, etc. is the constraint violation signal at moment, including the rule number violated and the severity level of violation , the severity level of violation is a discrete level, specifically including no violation, light level, and severe level; S24: all simulated execution trajectories and constraint violation signals in the entire simulation period are summarized to form the final simulation execution result The final simulation execution result includes complete execution trajectory and trigger information related to business constraints, and provides basis for subsequent policy revision, and is expressed as: ; Wherein, is the simulation execution result at time , including simulation execution trajectory and all constraint violation signal information in the whole cycle, is the simulation execution trajectory, is the constraint violation signal set.

[0022] S3 specifically includes: S31: Analyzing the simulation execution result , judging whether it includes constraint violation signal , if it exists, further analyzing the specific violation rule, expressed as: ; Wherein, is the simulation execution result, including all trajectory data and constraint violation signal, is the constraint violation signal, including the identifier of the violation rule and the severity information; In the battery procurement strategy optimization method of the application, the analysis step of the simulation execution result is very important, and the core purpose of the design is to ensure that the execution process of the procurement strategy can detect and judge whether the preset business constraint rule is violated in real time; the key of this process lies in predicting the strategy execution by using the simulation environment, judging whether there is a violation of the business rule by analyzing the simulation execution result; according to the type and severity of the violation, the system can timely trigger the revision mechanism to avoid the strategy that does not meet the business constraints being executed, so as to optimize the procurement decision; S32: If there is a constraint violation signal , according to the type of the triggered business constraint rule identified in the constraint violation signal , call the pre-defined policy revision rule library , select the corresponding parameter adjustment algorithm , expressed as: ; Wherein, is the number of triggered business constraint rule, is the pre-defined policy revision rule library, including various revision schemes and parameter adjustment methods, which is pre-defined according to business requirements, and the selection and management of the policy revision rule library helps to ensure that the most appropriate revision algorithm can be selected according to different situations when the rule is violated, is the parameter adjustment algorithm selected according to the business constraint rule ; When a violation of the business constraint rules is found during the simulation execution, the system needs to accurately identify the triggered rules in order to select the appropriate correction strategy, in order to ensure that the procurement decision can meet the actual needs of the enterprise while complying with the business rules; according to the specific type of the triggered rule, the system selects the appropriate parameter adjustment algorithm through the pre-defined correction rule library, different constraint rules require different correction measures, ensuring that the system can flexibly cope with different types of constraint violations and can adjust through the selection of appropriate methods in the correction rule library; S33: Based on the severity of the violation carried in the constraint violation signal , through the selected parameter adjustment algorithm Quantitative adjustment of the parameters in the original battery procurement strategy to generate the corrected battery procurement strategy , denoted as: ; Where, is the severity level of the violated rule, taking discrete values, indicates no violation, indicates a minor violation, indicates a serious violation, indicates the original battery procurement strategy, indicates the corrected battery procurement strategy, which satisfies all business constraint rules, aiming to make the procurement decision comply with the business rules, reduce constraint violations, and optimize cost and resource allocation; In the procurement strategy, different types of constraint violations may have different impacts, if the violation is minor, the adjustment to the procurement strategy may be small; while a serious violation requires significant correction of the strategy; this adjustment strategy based on severity can help the system effectively control the adjustment range of the strategy, avoiding excessive adjustment affecting normal operation; after the severity of the violation is determined, the appropriate parameter adjustment algorithm is selected for adjustment to ensure that the strategy can return to a reasonable range, generating the corrected battery procurement strategy, which can ensure that the strategy maintains business continuity while meeting the constraint requirements; S34: If there is a constraint violation signal, output the corrected battery procurement strategy to the procurement execution system, if there is no constraint violation signal, output the original battery procurement strategy directly to the procurement execution system, the final battery procurement strategy output to the procurement execution system is denoted as: ; Where, is the final battery procurement strategy output to the procurement execution system, which is selected according to whether there is a constraint violation signal, ensuring the flexibility and real-time adjustment of procurement execution; If there is a constraint violation signal during the simulation execution, the system will output a revised battery procurement strategy, which will ensure the balance between business demand and constraint rules, thereby improving the compliance and efficiency of procurement decisions. Through this method, enterprises can adjust their procurement strategies under changing market conditions, maintaining the flexibility and efficiency of the procurement process. When there is no constraint violation signal, the system directly executes the original battery procurement strategy, which can avoid unnecessary adjustments and maintain the consistency and stability of the procurement strategy. Through this mechanism, the system can efficiently execute the original strategy without adjustment, reducing the computational and execution burden of the system.

[0023] The present application encompasses any substitutions, modifications, equivalent methods and solutions made to the essence and scope of the present application. In order to fully understand the present application by the public, specific details are described in the following preferred embodiments of the present application, and the present application can be fully understood without these details by those skilled in the art. In addition, in order to avoid unnecessary confusion to the essence of the present application, well-known methods, processes, procedures, elements and circuits, etc. are not described in detail.

[0024] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. An adaptive optimization method for battery procurement strategy incorporating reinforcement learning, characterized in that, Includes the following steps: S1: Based on the current market situation, generate the original battery procurement strategy using a pre-trained reinforcement learning model; S2: Input the original battery procurement strategy into the strategy simulator to perform constraint compliance simulation and obtain the simulation execution results; wherein, the strategy simulator has multiple dynamically configurable business constraint rules built-in; S3: When the simulation execution result triggers any of the business constraint rules, the parameters of the original battery procurement strategy are modified according to the triggered business constraint rule, and the modified battery procurement strategy is output and executed. Otherwise, the original battery procurement strategy will be implemented directly.

2. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning as described in claim 1, characterized in that, S1 includes: S11: Real-time collection and integration of multi-dimensional status information from internal inventory systems and external market data sources; S12: Standardize the multi-dimensional state information to construct a current state vector that matches the dimensions of the input layer of the pre-trained reinforcement learning model; S13: Input the current state vector into the pre-trained reinforcement learning model, perform forward reasoning through the policy network inside the model, and output the purchase action with the highest probability from its preset discrete action space. The purchase action constitutes the original battery purchase strategy.

3. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning according to claim 2, characterized in that, The multi-dimensional status information includes battery inventory levels, raw material spot prices, and supplier capacity utilization rates.

4. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning according to claim 2, characterized in that, S13 includes: S131: Input the current state vector obtained after standardization into the pre-trained reinforcement learning model, and perform the forward inference process in the policy network inside the model to obtain the probability distribution of each procurement action in the preset discrete action space. S132: Based on the probability distribution output by the policy network, select the purchase action with the highest probability as the original battery purchase strategy at the current moment and output it.

5. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning according to claim 1, characterized in that, S2 includes: S21: Load the original battery procurement strategy as a driving instruction into the strategy simulator, and initialize the operating conditions based on the current market environment; S22: On the simulated time scale, execute the original battery procurement strategy and track its simulated execution trajectory in real time, the simulated execution trajectory including procurement cost distribution, supplier selection distribution and inventory level change curve; S23: The simulated execution trajectory is compared in real time with multiple dynamically configurable business constraint rules built into the strategy simulator. When any indicator in the trajectory violates any business constraint rule, a constraint violation signal including the specific violation rule identifier and the severity of the violation is generated. S24: Summarize all simulation execution trajectory data and constraint violation signals throughout the entire simulation cycle to form the simulation execution result.

6. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning according to claim 4, characterized in that, S23 includes: S231: During the simulation execution process, the generated simulation execution trajectory is compared with the built-in set of business constraint rules hourly to determine whether there are any violations of business constraint rules in the simulation execution trajectory; S232: When any indicator in the simulated execution trajectory violates a business constraint rule, a constraint violation signal is generated, which includes the number of the violation rule and the severity level of the violation.

7. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning according to claim 5, characterized in that, S24 includes: S241: Summarize all simulation execution trajectory data and constraint violation signals throughout the entire simulation cycle; S242: Combine the aggregated simulated execution trajectory data with the constraint violation signals to form the final simulated execution result.

8. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning according to claim 1, characterized in that, S3 includes: S31: Analyze the simulation execution results and determine whether they include constraint violation signals; S32: If the constraint violation signal exists, then according to the type of the triggered business constraint rule identified in the constraint violation signal, call the predefined strategy correction rule library and select the corresponding parameter adjustment algorithm; S33: Based on the severity of the violation carried in the constraint violation signal, the parameters in the original battery procurement strategy are quantitatively adjusted through the parameter adjustment algorithm to generate a revised battery procurement strategy; S34: Output the revised battery procurement strategy to the procurement execution system; if there is no constraint violation signal, output the original battery procurement strategy directly to the procurement execution system.

9. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning according to claim 7, characterized in that, S32 includes: S321: Analyze the type of triggered business constraint rule identified in the constraint violation signal, and identify the specific constraint category violated by rule number; S322: Based on the identified constraint rule type, select the parameter adjustment algorithm corresponding to the constraint rule from the predefined strategy correction rule base.

10. The adaptive optimization method for battery procurement strategy incorporating reinforcement learning according to claim 7, characterized in that, S34 includes: S341: When a constraint violation signal is detected, the corrected battery procurement strategy will be output to the procurement execution system; S342: If no constraint violation signal is detected, the original battery procurement strategy will be directly output to the procurement execution system to execute the original procurement decision.

Images