Project configuration updating method and system based on reinforcement learning

Abstract

The invention belongs to the field of machine learning, and discloses an item configuration updating method and system based on reinforcement learning. The method comprises the steps of collecting project feature data; the project feature data are analyzed, and a reinforcement learning model is adopted to predict real-time budget; calculating a budget deviation according to the real-time budget, and evaluating a budget execution state; judging whether an early warning instruction is generated or not according to the budget execution state; the budget deviation corresponding to the early warning instruction is analyzed, and a corresponding deviation reason is fed back; according to the invention, the real-time monitoring and early warning of the budget execution deviation can be realized, the adopted reinforcement learning model can perform autonomous learning optimization, and the prediction effect and the early warning accuracy are improved; and deviation reasons can be automatically analyzed, so that project configuration updating can be better realized, project related personnel can be ensured to timely find and correct budget deviation, the situations of project hyperbranched, delay and even failure are avoided, the project risk is reduced, and the project configuration updating efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and more specifically, to a project configuration updating method and system based on reinforcement learning. Background Art

[0002] In the process of project configuration update, budget control is one of the key factors to ensure project success; accurate budget execution is not only related to the economic benefits of the project, but also directly affects the progress and quality of the project; however, due to the complexity and uncertainty of the project environment, deviations often occur in the budget execution process. If these deviations are not discovered and corrected in time, they may lead to project overspending, delays or even failure; traditional budget control methods mainly rely on manual monitoring and regular audits, which cannot achieve real-time warnings and dynamic adjustments, and therefore cannot meet the needs of modern project configuration updates;

[0003] The Chinese patent application number CN117993854A discloses a method for cost management of industrial park power engineering projects based on data analysis; the method comprises the following steps: making an investment budget for the main distribution network of the industrial park power engineering project under construction; making investment accounting for the industrial park power engineering project under construction, wherein the accounting content includes real-time monitoring of the cost information and progress information of the industrial park power engineering project under construction, judging whether there is abnormal deviation data based on the data of the project and giving early warning for the deviation data, making cost accounting based on the cost information of the industrial park power engineering project under construction and forming a project report; making investment evaluation and analysis for the industrial park power engineering project under construction; and solving the problems of many cost indicators, large differences and difficulty in estimating of the industrial park power engineering project, thereby achieving the effect of providing a more scientific and reliable basis for cost prediction and real-time cost monitoring of power grid construction projects;

[0004] Although the above technology can achieve real-time early warning, it is limited by the manually designed analysis framework and cannot learn and optimize autonomously, that is, it cannot fully utilize the real-time feedback information of the current project environment to continuously improve the accuracy of early warning. At the same time, it is difficult to automatically analyze the causes of deviations in project budgets. It is necessary to rely on the supervision personnel to find and analyze the specific causes of the deviations on site, which reduces the response speed and project configuration update efficiency, and has certain subjectivity and limitations.

[0005] In view of this, the present invention proposes a project configuration update method and system based on reinforcement learning to solve the above problems. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a project configuration update method based on reinforcement learning, comprising:

[0007] Collect project characteristics data;

[0008] Analyze project feature data and use reinforcement learning models to predict real-time budgets;

[0009] Calculate budget deviations and evaluate budget execution status based on real-time budget;

[0010] Determine whether to generate an early warning instruction based on the budget execution status;

[0011] Analyze the budget deviations corresponding to the early warning instructions and provide feedback on the corresponding reasons for the deviations.

[0012] Furthermore, the project characteristic data includes project budget data, project expenditure data and project progress data; the project budget data includes the budget of each sub-project in each stage of the project; the project expenditure data includes the expenditure of each sub-project in each completed stage of the project; the project progress data includes the current stage and stage progress; the current stage is the current stage of the project, and the stage progress is the degree of completion of the current stage;

[0013] The method for predicting real-time budget includes:

[0014] Filter out the sub-project budgets corresponding to the current stage from the project pre-selected data and mark them as stage budgets; obtain the payment progress data corresponding to the current stage, which includes the advance payment and final payment for each sub-project;

[0015] Set different digital labels for each stage of the project; obtain the digital label corresponding to the current stage and mark it as the stage label; use the stage label, stage progress, stage budget and payment progress data as analysis data, input the analysis data into the trained budget prediction model, and predict the real-time budget, which is the sub-project budget corresponding to the stage progress of the current stage of the project.

[0016] Furthermore, the steps of building the budget forecasting model include:

[0017] Step 1: Define the state space;

[0018] The state S includes the stage label, stage progress, stage budget, payment progress data and real-time budget. The stage label, stage progress, stage budget, payment progress data and real-time budget are all represented by real numbers. The state vector dimension is set to 3n+3, where n is the number of sub-projects in the current stage.

[0019] Step 2: Define the action space;

[0020] Action A is the amplitude and direction of the next adjustment of the analysis data; the value of each analysis data is adjusted according to the real number interval [-B, B]; the absolute value of the real number represents the percentage of the amplitude of the adjustment of the analysis data, the positive and negative signs represent the direction of the adjustment of the analysis data, and the action vector dimension is 2n+3;

[0021] Step 3: Design the Actor network structure;

[0022] The Actor network structure includes 1 first input layer, 2 hidden layers and 1 first output layer; the state vector of the first input layer is 3n+3; the 2 hidden layers include the first hidden layer and the second hidden layer, the first hidden layer includes 32 nodes, the activation function is the Relu function, the second hidden layer includes 16 nodes, the activation function is also the Relu function; the first output layer has 3-dimensional continuous action values ​​and no activation function;

[0023] Step 4: Design the Critic network structure;

[0024] The Critic network structure includes 1 second input layer, 2 hidden layers and 1 second output layer; the state vector of the second input layer has 3n+3; the 2 hidden layers include the third hidden layer and the fourth hidden layer, the third hidden layer includes 64 nodes, the activation function is the Relu function, the fourth hidden layer includes 32 nodes, the activation function is the Relu function; the second output layer includes 1 node, without activation function;

[0025] Step 5: Design training configuration:

[0026] Actor network learning rate α1 = 0.0001; Critic network learning rate α2 = 0.0001; experience replay memory size E = 10000; batch size F = 128; update network step size τ = 0.0001; discount factor γ = 0.99.

[0027] Furthermore, the training process of the budget prediction model includes:

[0028] Randomly select action A and obtain environmental feedback (M′, m, ψ), where M′ is the next state, m is the reward value obtained after selecting action A, and ψ is a flag indicating whether it is a terminal state; if the task is completed or failed, ψ = True, and if the task is still continuing, ψ = False; save the action A and the corresponding environmental feedback after each training to the experience replay memory;

[0029] Divide all actions A and corresponding environmental feedback stored in the experience replay memory into a training set and a validation set;

[0030] Get the evaluation Q value PQ of the current Critic network;

[0031] Calculate the target Q value MQ of the Critic network;

[0032] MQ = m + γ × C (M');

[0033] Where C(M') is the quality value obtained by the Critic network evaluating the next state M';

[0034] Calculate the Actor network gradient ▽θ;

[0035] ▽θ=▽AQ(M,A)|M=M',A=A(M')▽θA(M');

[0036] Where A(M') is the action generated according to M'; ▽AQ(M,A) is the Jacobian matrix, and each element in the Jacobian matrix is ​​the partial derivative of MQ learned by the Critic network with respect to each dimension of the Actor network parameter θ; ▽θA(M') represents the gradient of the action with respect to the Actor network parameter θ;

[0037] Calculate the loss value loos; loos = PQ-MQ;

[0038] The model is trained continuously using the training set until the loss value loos continues to decrease and converges, then the training ends.

[0039] Furthermore, the method for calculating the budget deviation includes:

[0040] Filter out the sub-project expenditures corresponding to the current stage from the project expenditure data and mark them as stage expenditures;

[0041] Subtract the corresponding sub-project budget in the real-time budget from each sub-project expenditure in the stage expenditure to obtain the budget deviation of each sub-project;

[0042] The method for evaluating the budget execution status comprises:

[0043] Set different digital labels for each sub-project in each stage of the project and mark them as sub-project labels; set up a sub-project-threshold mapping table to obtain the corresponding deviation threshold according to the sub-project label corresponding to the current stage;

[0044] Compare each budget deviation with the corresponding deviation threshold;

[0045] If the budget deviation is greater than or equal to the deviation threshold, the corresponding budget deviation is marked as budget overrun;

[0046] If the budget deviation is less than the deviation threshold and greater than the opposite of the deviation threshold, the corresponding budget deviation will not be marked;

[0047] If the budget deviation is less than or equal to the opposite of the deviation threshold, the corresponding budget deviation is marked as the budget surplus;

[0048] The budget execution status includes budget surplus status, budget overspending status and budget balance status;

[0049] Add up each budget overrun in sequence to obtain the total budget overrun; add up each budget surplus in sequence to obtain the total budget surplus; then add up the total budget overrun and the total budget surplus to obtain the total budget deviation;

[0050] If the total budget deviation is less than 0, the budget execution status is the budget surplus status;

[0051] If the total budget deviation is greater than 0, the budget execution status is budget overspending;

[0052] If the total budget deviation is equal to 0, the budget execution status is the budget breakeven status.

[0053] Furthermore, the method for determining whether to generate a warning instruction includes:

[0054] Early warning instructions include low-level early warning instructions, medium-level early warning instructions and high-level early warning instructions;

[0055] If the budget execution status is the budget balance status, no warning instruction will be generated;

[0056] If the budget execution status is a budget surplus status or a budget overspending status, the budget surplus and budget overspending are counted respectively, and the budget surplus is added to the budget overspending to obtain the budget anomaly number; the budget deviation is counted, and the budget anomaly number is divided by the budget deviation number to obtain the anomaly ratio D; a ratio threshold C is preset, C∈(0,0.5), and the anomaly ratio D is compared with the ratio threshold C;

[0057] If 0<D<C, a low-level warning instruction is generated;

[0058] If C≤D≤2C, an intermediate warning instruction is generated;

[0059] If 2C<D≤1, an advanced warning instruction is generated.

[0060] Furthermore, the step of feeding back the corresponding deviation cause includes:

[0061] Step a: Preset initialization temperature T max , minimum temperature T min , cooling coefficient δ and maximum number of iterations Let the current temperature T = T max ; A preset cause set, the cause set includes m causes of deviation, where m is an integer greater than 1;

[0062] Step b: Randomly set a feasible solution χ, the feasible solution χ is a deviation cause, and the range of feasible solutions χ is the cause set;

[0063] Step c: determine the fitness function;

[0064] The expression of fitness function is: f = PP;

[0065] In the formula, f is the fitness, YP is the matching degree;

[0066] Step d: Calculate the fitness f corresponding to the feasible solution χ; take the feasible solution χ as the current point, perform random perturbations in the neighborhood of the current point, obtain a new feasible solution χ′, and calculate the fitness f′ corresponding to the new feasible solution χ′;

[0067] Step e: Calculate the fitness difference f″, the expression of the fitness difference f″ is f″=f′-f;

[0068] If the fitness difference f″>0, let χ=χ′, that is, assign the value of the new feasible solution χ′ to the feasible solution χ; if the fitness difference f″≤0, calculate the probability p″, and let χ=χ′ according to the probability p″; the expression of probability p″ is: In the formula, e is a natural constant;

[0069] Step f: Repeat steps d to e until the number of loops reaches the maximum number of iterations. When , the loop ends and enters step g;

[0070] Step g: Set the current temperature T = T × δ, that is, cool down the current temperature in step a, and assign the value after cooling to the current temperature; set the maximum number of iterations The value of the maximum number of iterations after the reduction is assigned to the maximum number of iterations;

[0071] Step h: Repeat steps d to g until the current temperature T < T min When , the loop ends, the deviation cause corresponding to the feasible solution χ is obtained, and feedback is given.

[0072] Furthermore, the method for obtaining the matching degree includes:

[0073] Different digital labels are set for the m types of deviation reasons in the reason set, and marked as reason labels; the sub-project labels corresponding to the budget overrun and budget surplus are marked as abnormal labels; the stage label, stage process, abnormal label, sub-project expenditure corresponding to the abnormal label, and the sub-project budget corresponding to the abnormal label are used as evaluation data, and the evaluation data and the cause label of the deviation cause corresponding to the feasible solution χ are used as test data. The evaluation data and the abnormal label are in one-to-one correspondence, that is, a group of evaluation data corresponds to one abnormal label; the test data is input into the trained matching prediction model to predict the corresponding matching degree;

[0074] The training process of the matching prediction model includes:

[0075] Pre-collect the matching degrees corresponding to b groups of test data, and convert the test data and the corresponding matching degrees into a corresponding set of feature vectors, where b is an integer greater than 1;

[0076] Each group of feature vectors is used as the input of the matching prediction model. The matching prediction model takes a group of predicted matching degrees corresponding to each test data as output, and takes the actual matching degree corresponding to each test data as the prediction target, where the actual matching degree is the pre-collected matching degree corresponding to the test data; minimizing the sum of prediction errors of all test data is used as the training goal; the matching prediction model is trained until the sum of prediction errors reaches convergence and the training is stopped; the matching prediction model is a deep neural network model.

[0077] Further, the deviation reasons of the feedback are analyzed to determine whether to update the deviation reasons;

[0078] The deviation cause of the feedback is marked as the feedback cause; a fitness threshold is preset, the fitness of the feedback cause is marked as the cause fitness, and the cause fitness is compared with the fitness threshold;

[0079] If the cause fitness is less than or equal to the fitness threshold, the deviation cause is updated and the updated deviation cause is added to the cause set;

[0080] If the cause fitness is greater than the fitness threshold, the deviation cause is not updated;

[0081] The method for updating the deviation cause is: input the evaluation data into the trained cause update model, predict the corresponding update label, obtain the corresponding update cause according to the update label, and provide feedback; the update label is the digital label corresponding to the update cause, and the update cause is the updated deviation cause; the training process of the cause update model is consistent with the training process of the matching prediction model, and it is also a deep neural network model; the difference is that during the use of the cause update model, new evaluation data and corresponding deviation causes will be continuously collected to train the cause update model.

[0082] A project configuration updating system based on reinforcement learning implements the project configuration updating method based on reinforcement learning, including:

[0083] Data collection module, used to collect project characteristic data;

[0084] Data analysis module, used to analyze project feature data and predict real-time budget using reinforcement learning model;

[0085] Status evaluation module, used to calculate budget deviation and evaluate budget execution status based on real-time budget;

[0086] The early warning module is used to determine whether to generate an early warning instruction based on the budget execution status;

[0087] The cause feedback module is used to analyze the budget deviation corresponding to the early warning instruction and feedback the corresponding deviation cause.

[0088] The technical effects and advantages of the project configuration update method and system based on reinforcement learning of the present invention are as follows:

[0089] 1. By collecting and analyzing project feature data, using the reinforcement learning model to predict the real-time budget, calculate the budget deviation and evaluate the budget execution status, and determine whether to generate an early warning instruction based on the budget execution status; achieve real-time monitoring and early warning of budget execution deviations, and the reinforcement learning model can autonomously learn and optimize to improve the prediction effect and the accuracy of the early warning; at the same time, it can also automatically analyze the cause of the deviation without manual search and analysis of the cause of the deviation, so as to better realize the project configuration update, ensure that project-related personnel can discover and correct budget deviations in a timely manner, avoid project overspending, delays or even failures, thereby reducing project risks, improving the efficiency of project configuration updates, and providing technical support for the smooth completion of projects.

[0090] 2. Realize online learning and updating of the causes of deviations in project budget execution; by continuously collecting new evaluation data and corresponding deviation causes, the cause update model can continuously learn in the actual project configuration update application process, continuously optimize and improve the built-in cause knowledge base, and overcome the problem that the cause set is difficult to automatically update; therefore, it can more accurately update the deviation causes with low fitness, has stronger adaptive capabilities and continuous optimization potential, and can effectively improve the accuracy and efficiency of project budget execution deviation warnings. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 This is a schematic diagram of a project configuration update system based on reinforcement learning according to Embodiment 1 of the present invention;

[0092] Figure 2 This is a schematic diagram of a project configuration update system based on reinforcement learning according to Embodiment 2 of the present invention;

[0093] Figure 3 This is a flow chart of a project configuration update method based on reinforcement learning according to Embodiment 3 of the present invention;

[0094] Figure 4 This is a schematic diagram of an electronic device according to Embodiment 4 of the present invention;

[0095] Figure 5 This is a schematic diagram of a storage medium according to Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0097] Example 1

[0098] See also Figure 1 As shown, the project configuration update system based on reinforcement learning described in this embodiment includes a data collection module, a data analysis module, a status evaluation module, an early warning module and a cause feedback module; each module is connected by wired and / or wireless means to realize data transmission between modules;

[0099] Data collection module, used to collect project characteristic data.

[0100] Project characteristic data includes project budget data, project expenditure data and project progress data; all project data are obtained from the project information system; various data in the project information system are recorded and updated by project-related personnel;

[0101] The project budget data includes the budget of each sub-project in each stage of the project; for example, a real estate development project can be divided into the infrastructure construction stage, the building construction stage, the supporting facilities construction stage, etc. Among them, the sub-projects in the infrastructure construction stage include site leveling, road and pipe network construction, water, electricity and other public facilities construction, etc., the sub-projects in the building construction stage include the main structure construction, interior decoration materials, labor recruitment, etc., and the sub-projects in the supporting facilities construction stage include parking lot construction, greening project construction, property management site construction, etc.;

[0102] Project expenditure data includes the expenditure of each sub-project in each stage of the completed project; for example, site leveling expenditure, main structure construction expenditure, parking lot construction expenditure, etc.

[0103] The project progress data includes the current stage and stage progress; the current stage is the current stage of the project, and the stage progress is the degree of completion of the current stage, such as 90%, 50%, etc.

[0104] The data analysis module is used to analyze project feature data and predict real-time budget using a reinforcement learning model.

[0105] Methods for forecasting real-time budgets include:

[0106] Filter out the sub-project budget corresponding to the current stage from the project pre-selected data and mark it as the stage budget; obtain the payment progress data corresponding to the current stage, which includes the advance payment and final payment of each sub-project; the payment progress data corresponding to the current stage is also obtained from the project information system;

[0107] Different digital labels are set for each stage of the project; the digital label corresponding to the current stage is obtained and marked as the stage label; the stage label, stage progress, stage budget and payment progress data are used as analysis data, and the analysis data is input into the trained budget prediction model to predict the real-time budget. The real-time budget is the sub-project budget corresponding to the stage progress of the current stage of the project, such as the sub-project budget corresponding to 90% of the completion of the infrastructure construction stage, and the sub-project budget corresponding to 50% of the completion of the supporting facilities construction stage.

[0108] The steps to build a budget forecasting model include:

[0109] Step 1: Define the state space;

[0110] The state S includes stage label, stage progress, stage budget, payment progress data and real-time budget. The stage label, stage progress, stage budget, payment progress data and real-time budget are all represented by real numbers; the state vector dimension is set to 3n+3, where n is the number of sub-projects in the current stage.

[0111] Step 2: Define the action space;

[0112] Action A is the amplitude and direction of the next adjustment of the analysis data; the value of each analysis data is adjusted according to the real number interval [-B, B], and B is preferably 0.05 in this embodiment; wherein the absolute value of the real number represents the percentage of the amplitude of the adjustment of the analysis data once, the positive and negative signs represent the direction of the adjustment of the analysis data once, and the action vector dimension is 2n+3.

[0113] Step 3: Design the Actor network structure;

[0114] The Actor network structure includes 1 first input layer, 2 hidden layers and 1 first output layer; the state vector of the first input layer is 3n+3; the 2 hidden layers include the first hidden layer and the second hidden layer, the first hidden layer includes 32 nodes, the activation function is the Relu function, the second hidden layer includes 16 nodes, and the activation function is also the Relu function; the first output layer has a 3-dimensional continuous action value and no activation function.

[0115] Step 4: Design the Critic network structure;

[0116] The Critic network structure includes 1 second input layer, 2 hidden layers and 1 second output layer; the state vector of the second input layer has 3n+3; the 2 hidden layers include the third hidden layer and the fourth hidden layer, the third hidden layer includes 64 nodes, the activation function is the Relu function, the fourth hidden layer includes 32 nodes, the activation function is the Relu function; the second output layer includes 1 node and has no activation function.

[0117] Step 5: Design training configuration:

[0118] Actor network learning rate α1 = 0.0001; Critic network learning rate α2 = 0.0001; experience replay memory size E = 10000; batch size F = 128; update network step size τ = 0.0001; discount factor γ = 0.99.

[0119] The specific training process of the budget prediction model includes:

[0120] Randomly select action A and obtain environmental feedback (M′, m, ψ), where M′ is the next state, m is the reward value obtained after selecting action A, and ψ is a flag indicating whether it is a terminal state; if the task is completed or failed, ψ = True, if the task is still continuing, ψ = False; save the action A and the corresponding environmental feedback after each training to the experience playback memory; the reward value obtained after selecting action A is preset by the project-related personnel based on the actual situation of the project and combined with actual experience. The reward value corresponding to action A is obtained according to the reward function;

[0121] All actions A and the corresponding environment feedback stored in the experience replay memory are divided into a training set and a validation set.

[0122] Get the evaluation Q value PQ of the current Critic network;

[0123] Calculate the target Q value MQ of the Critic network;

[0124] MQ = m + γ × C (M');

[0125] In the formula, C(M') is the quality value obtained by the Critic network to evaluate the next state M', which provides a stable learning goal for training.

[0126] Calculate the Actor network gradient ▽θ;

[0127] ▽θ=▽AQ(M,A)|M=M',A=A(M')▽θA(M');

[0128] Where A(M') is the action generated according to M'; ▽AQ(M,A) is the Jacobian matrix, each element in the Jacobian matrix is ​​the partial derivative of MQ learned by the Critic network with respect to each dimension of the Actor network parameter θ; ▽θA(M') represents the gradient of the action with respect to the Actor network parameter θ.

[0129] Calculate the loss value loos; loos = PQ-MQ;

[0130] The model is trained continuously using the training set until the loss value loos continues to decrease and converges, then the training ends.

[0131] It should be noted that the purpose of calculating the Actor network gradient ▽θ is to: guide the direction of strategy update. The Actor network gradient represents the update direction of the Actor network parameter θ, promoting continuous optimization of the strategy; control the learning speed. The size of the gradient determines the magnitude of each step of the strategy update, avoiding divergence or overfitting, and achieving stable convergence of the strategy.

[0132] It should be understood that the budget forecasting model is a reinforcement learning model. The purpose of using the reinforcement learning model is to efficiently process multidimensional data: by defining the state space, the reinforcement learning model can process complex multidimensional data, such as stage labels, stage progress, stage budget and payment progress data; it can learn deep patterns and relationships from these data to improve the accuracy of budget forecasting; strong dynamic adaptability: by defining the action space, the reinforcement learning model is continuously updated and optimized through interaction with the environment, which can better adapt to changes and uncertainties in the project progress; it enables the model to adjust the budget forecast in real time when facing complex project configuration update scenarios; continuous improvement: by designing the training configuration and continuously adjusting the learning rate and other parameters, the model can continuously improve during the training process, so as the project progresses and new data is continuously added, the reinforcement learning model can continue to learn and improve; that is, the model's prediction ability will continue to increase over time, and it can better cope with various challenges in project configuration updates; automation capability: by designing the Actor network structure, that is, the deep neural network, the reinforcement learning model can automatically analyze and process large amounts of data, reducing the need for manual intervention and improving the automation of the prediction process; it not only reduces the risk of human error, but also saves human resources and time;

[0133] The status evaluation module is used to calculate the budget deviation and evaluate the budget execution status based on the real-time budget.

[0134] Methods for calculating budget variance include:

[0135] Filter out the sub-project expenditures corresponding to the current stage from the project expenditure data and mark them as stage expenditures;

[0136] Subtract the corresponding sub-project budget in the real-time budget from each sub-project expenditure in the stage expenditure to obtain the budget deviation of each sub-project, so as to accurately reflect the budget execution status of each sub-project in the current stage.

[0137] Methods for assessing the status of budget execution include:

[0138] Different digital labels are set for each sub-project in each stage of the project and marked as sub-project labels; a sub-project-threshold mapping table is set to obtain the corresponding deviation threshold according to the sub-project label corresponding to the current stage, thereby improving the efficiency of obtaining the deviation threshold.

[0139] The sub-project-threshold mapping table is stored in the project information system and may be a two-dimensional mapping table, which includes sub-project labels and deviation thresholds corresponding to the sub-project labels; according to the sub-project labels, the corresponding deviation thresholds may be obtained through the sub-project-threshold mapping table;

[0140] The method for setting the sub-project-threshold mapping table is as follows: project-related personnel set corresponding deviation thresholds for each different sub-project in each stage of the project in turn according to the actual situation of the project and their own experience; set the sub-project-threshold mapping table according to the sub-project labels corresponding to each different sub-project in each stage of the project and the deviation thresholds corresponding to each different sub-project in each stage of the project.

[0141] Compare each budget deviation with the corresponding deviation threshold. By comparing the budget deviation with the preset deviation threshold, sub-projects with budget overruns or underruns can be identified intuitively and quickly, improving assessment accuracy.

[0142] If the budget deviation is greater than or equal to the deviation threshold, the corresponding budget deviation will be marked as the budget overrun amount, indicating that the expenditure of the sub-project corresponding to the budget deviation far exceeds the corresponding sub-project budget, resulting in budget overrun;

[0143] If the budget deviation is less than the deviation threshold and greater than the opposite of the deviation threshold, the corresponding budget deviation will not be marked;

[0144] If the budget deviation is less than or equal to the opposite of the deviation threshold, the corresponding budget deviation will be marked as the budget surplus; this means that the sub-project budget corresponding to the budget deviation far exceeds the corresponding sub-project expenditure, resulting in a budget surplus.

[0145] The budget execution status includes budget surplus status, budget overspending status and budget balance status;

[0146] Each budget overrun is added up in sequence to obtain the total budget overrun; each budget surplus is added up in sequence to obtain the total budget surplus; the total budget overrun is then added to the total budget surplus to obtain the total budget deviation; the total budget deviation takes into account the deviations of all sub-projects and provides a comprehensive assessment of the overall budget execution;

[0147] If the total budget deviation is less than 0, the budget execution status is the budget surplus status;

[0148] If the total budget deviation is greater than 0, the budget execution status is budget overspending;

[0149] If the total budget deviation is equal to 0, the budget execution status is the budget breakeven status.

[0150] The early warning module is used to determine whether to generate an early warning instruction based on the budget execution status.

[0151] Methods for determining whether to generate an early warning instruction include:

[0152] Early warning instructions include low-level early warning instructions, medium-level early warning instructions and high-level early warning instructions;

[0153] If the budget execution status is the budget flat status, no warning instruction will be generated, indicating that when the project reaches the current stage, the actual expenditure is within the normal range and there is no need to analyze the cause of the deviation;

[0154] If the budget execution status is a budget surplus status or a budget overspending status, the number of budget surplus and budget overspending are counted respectively, and the number of budget surplus is added to the number of budget overspending to obtain the number of budget anomalies; the number of budget deviations is counted, and the number of budget anomalies is divided by the number of budget deviations to obtain the anomaly ratio D; a ratio threshold C is preset, and the anomaly ratio D is compared with the ratio threshold C; the ratio threshold C is preset by project-related personnel according to the number of sub-projects in the current stage of the project and the actual situation of the project, C∈(0,0.5), and C is preferably 0.4 in this embodiment;

[0155] If 0<D<C, a low-level warning instruction is generated, indicating that the value of the abnormal proportion is small and the number of abnormal budget deviations is small, so the corresponding warning level is the lowest;

[0156] If C≤D≤2C, an intermediate warning instruction is generated, indicating that the value of the abnormal proportion is large and the number of abnormal budget deviations is large, so the corresponding warning level is high;

[0157] If 2C<D≤1, a high-level warning instruction is generated, indicating that the value of the abnormal proportion is too large and the number of abnormal budget deviations is too large, so the corresponding warning level is the highest.

[0158] It should be noted that different levels of warning instructions are generated according to the abnormal proportion, and graded warnings are provided to facilitate project-related personnel to take corresponding adjustment measures according to the warning level, thereby improving the initiative and effectiveness of project configuration updates.

[0159] The cause feedback module is used to analyze the budget deviation corresponding to the early warning instruction and feedback the corresponding deviation cause.

[0160] The steps for providing feedback on the corresponding deviation reasons include:

[0161] Step a: Preset initialization temperature T max , minimum temperature T min , cooling coefficient δ and maximum number of iterations Let the current temperature T = T max ; A preset cause set includes m causes of deviation, where m is an integer greater than 1, and the m causes of deviation are all causes of budget deviation determined by project-related personnel in the process of judging budget deviations of historical projects; for example, during site leveling, it is found that the groundwater level is high or there are a large number of geological problems, resulting in earthwork volume and drainage system construction costs exceeding the budget; during pipeline laying, unexpected pipeline conflicts are encountered, requiring the pipeline route to be redesigned, increasing the pipeline construction cost, etc.

[0162] It should be noted that by presetting the parameters and the cause set, a structured framework is constructed, which facilitates the subsequent implementation of the simulated annealing algorithm with clear goals and scope, improves the operability and efficiency of the simulated annealing algorithm, and uses the budget deviation causes in historical projects to help quickly find the deviation causes in new projects and improve analysis efficiency;

[0163] Step b: Randomly set a feasible solution χ, where the feasible solution χ is a deviation cause, and the range of the feasible solution χ is the cause set. By randomly initializing the feasible solution, we can fully explore different deviation causes, prevent falling into the local optimum, and increase the probability of finding the global optimal solution.

[0164] Step c: determine the fitness function;

[0165] The expression of fitness function is: f = PP;

[0166] In the formula, f is the fitness and YP is the matching degree.

[0167] The matching degree can be obtained by:

[0168] Different digital labels are set for the m deviation reasons in the reason set and marked as reason labels; the sub-project labels corresponding to the budget overrun and budget surplus are marked as abnormal labels; the stage label, stage process, abnormal label, sub-project expenditure corresponding to the abnormal label and the sub-project budget corresponding to the abnormal label are used as evaluation data, and the evaluation data and the cause label of the deviation cause corresponding to the feasible solution χ are used as test data. The evaluation data and the abnormal label correspond one-to-one, that is, a group of evaluation data corresponds to one abnormal label; the test data is input into the trained matching prediction model to predict the corresponding matching degree.

[0169] The specific training process of the matching prediction model includes:

[0170] Pre-collect the matching degrees corresponding to the test data of group b, and convert the test data and the corresponding matching degrees into a corresponding set of feature vectors, where b is an integer greater than 1; the matching degrees corresponding to the test data are analyzed by project-related personnel based on the actual situation of the project, and the matching degree between the deviation cause and the evaluation data under the conditions of each evaluation data is analyzed, so as to obtain the matching degrees corresponding to the test data of group b;

[0171] Each set of feature vectors is used as the input of the matching prediction model. The matching prediction model takes a set of predicted matching degrees corresponding to each test data as output, and takes the actual matching degree corresponding to each test data as the prediction target. The actual matching degree is the matching degree corresponding to the test data collected in advance. The training goal is to minimize the sum of the prediction errors of all test data. The calculation formula of the prediction error is Z k =(α k -μ k ) 2 , where Zk is the prediction error, k is the group number of the feature vector corresponding to the test data, αk is the predicted matching degree corresponding to the k-th test data, and μk is the actual matching degree corresponding to the k-th test data; the matching prediction model is trained until the sum of the prediction errors reaches convergence and the training is stopped;

[0172] The above-mentioned matching prediction model is specifically a deep neural network model, which includes an input layer, a hidden layer and an output layer; each hidden layer includes multiple neurons, each neuron is connected to the neurons in the next layer, and the connection contains weights, which determine the importance and influence of data transmission in the neural network; an activation function is applied to each neuron between the hidden layer and the output layer, and the activation function reflects nonlinearity, allowing the network to learn more complex patterns and features.

[0173] It should be noted that the purpose of calculating the matching degree is to quantify the fitness so that the judgment of the cause of the deviation no longer relies on pure experience, but is based on a data-driven method, which improves the reliability and scientificity of the judgment.

[0174] Step d: Calculate the fitness f corresponding to the feasible solution χ; take the feasible solution χ as the current point, perform random perturbations in the neighborhood of the current point, obtain a new feasible solution χ′, and calculate the fitness f′ corresponding to the new feasible solution χ′.

[0175] Step e: Calculate the fitness difference f″, the expression of the fitness difference f″ is f″=f′-f;

[0176] If the fitness difference f″>0, let χ=χ′, that is, assign the value of the new feasible solution χ′ to the feasible solution χ; if the fitness difference f″≤0, calculate the probability p″, and let χ=χ′ according to the probability p″; the expression of probability p″ is: In the formula, e is a natural constant. Through the fitness difference and probability update mechanism, the feasible solution is continuously adjusted and the solution is continuously optimized. It can not only quickly converge to the local optimal solution, but also jump out of the local optimal solution through random perturbations, conduct global search, and improve the quality of the final solution.

[0177] Step f: Repeat steps d to e until the number of loops reaches the maximum number of iterations. When , the loop ends and enters step g; through multiple iterative optimizations, the feasible solution is continuously improved, which increases the possibility and accuracy of finding the best deviation cause.

[0178] Step g: Set the current temperature T = T × δ, that is, cool down the current temperature in step a, and assign the value after cooling to the current temperature; set the maximum number of iterations The value of the reduced maximum number of iterations is assigned to the maximum number of iterations; if the reduced maximum number of iterations is not an integer, the reduced maximum number of iterations is rounded up to make it an integer; this helps to gradually converge to the global optimal solution in the later stage and prevents too fast convergence to a suboptimal solution in the early stage.

[0179] Step h: Repeat steps d to g until the current temperature T < T min When , the loop ends, the deviation cause corresponding to the feasible solution χ is obtained, and feedback is given.

[0180] It should be noted that the initialization temperature T max , minimum temperature T min , cooling coefficient δ and maximum number of iterations As preset parameters, the preset parameters are collected by technicians in this field during the historical project budget deviation judgment process. c groups of test data are collected, where c is an integer greater than 1. For the same test data, multiple groups of different preset parameters are pre-set in sequence, and the deviation causes are obtained by using the simulated annealing algorithm in sequence. The corresponding fitness is calculated according to the obtained multiple deviation causes, and the preset parameter corresponding to the maximum fitness is used as the preset parameter corresponding to the group of test data. The preset parameters corresponding to c groups of test data are obtained by analogy, and the mean values ​​of multiple preset parameters (i.e., the initialization temperature mean, the minimum temperature mean, the cooling coefficient mean, and the maximum number of iterations mean) are used as the preset initialization temperature T in step a. max , minimum temperature T min , cooling coefficient δ and maximum number of iterations

[0181] It should be understood that the process of steps a to h above is the simulated annealing algorithm. Combining the simulated annealing algorithm with the deep neural network model can effectively analyze the causes of budget deviations and improve the intelligence and scientific level of budget management.

[0182] This embodiment collects and analyzes project feature data, uses a reinforcement learning model to predict real-time budgets, calculates budget deviations and evaluates budget execution status, and determines whether to generate warning instructions based on the budget execution status; it realizes real-time monitoring and early warning of budget execution deviations, and the reinforcement learning model can autonomously learn and optimize to improve the prediction effect and the accuracy of early warning; at the same time, it can also automatically analyze the causes of deviations without manual search and analysis of the causes of deviations, thereby better realizing project configuration updates, ensuring that project-related personnel can promptly discover and correct budget deviations, avoid project overspending, delays or even failures, thereby reducing project risks, improving project configuration update efficiency, and providing technical support for the smooth completion of projects.

[0183] Example 2

[0184] See also Figure 2 As shown, this embodiment further improves the design on the basis of Embodiment 1. In Embodiment 1, when feedback is given on the cause of deviation, it is obtained from a preset cause set. Since the cause set cannot be automatically updated, it is difficult to optimize the cause set. Therefore, the deviation cause feedback mechanism lacks adaptability, resulting in the feedback deviation cause being inconsistent with the actual cause, thereby reducing the reliability and stability of the cause analysis after the real-time warning of the project budget execution deviation. Therefore, this embodiment provides a project configuration update system based on reinforcement learning, which also includes a cause update module.

[0185] The cause update module is used to analyze the feedback deviation cause and determine whether to update the deviation cause.

[0186] The feedback deviation cause is marked as the feedback cause; a fitness threshold is preset, the fitness of the feedback cause is marked as the cause fitness, and the cause fitness is compared with the fitness threshold; the fitness threshold is the fitness corresponding to the accurate feedback of the deviation cause collected by project-related personnel multiple times during the historical project budget deviation judgment process, and the average of multiple fitnesses is used as the fitness threshold;

[0187] If the cause fitness is less than or equal to the fitness threshold, the deviation cause is updated and the updated deviation cause is added to the cause set;

[0188] If the cause fitness is greater than the fitness threshold, the deviation cause is not updated.

[0189] The method for updating the deviation cause is: input the evaluation data into the trained cause update model, predict the corresponding update label, obtain the corresponding update cause according to the update label, and provide feedback; the update label is the digital label corresponding to the update cause, and the update cause is the updated deviation cause; the specific training process of the cause update model is consistent with the training process of the matching prediction model, and it is also a deep neural network model; the difference is that during the use of the cause update model, new evaluation data and corresponding deviation causes will be continuously collected to train the cause update model, and the new evaluation data and corresponding deviation causes are obtained from the abnormal event log in the project information system. The abnormal event log is written by project-related personnel based on the feedback deviation cause and the actual situation of the project, so that the cause update model can continuously learn online to improve the prediction effect and adaptability.

[0190] This embodiment realizes online learning and updating of the causes of project budget execution deviations; by continuously collecting new evaluation data and corresponding deviation causes, the cause update model can continuously learn in the actual project configuration update application process, continuously optimize and improve the built-in cause knowledge base, and overcome the problem that the cause set is difficult to automatically update; therefore, it can more accurately update the deviation causes with lower fitness, has stronger adaptive capabilities and continuous optimization potential, and can effectively improve the accuracy and efficiency of project budget execution deviation warnings.

[0191] Example 3

[0192] See also Figure 3 As shown, the part not described in detail in this embodiment can be found in the description of embodiments 1 and 2, and a method for updating project configuration based on reinforcement learning is provided, the method comprising:

[0193] Collect project characteristics data;

[0194] Analyze project feature data and use reinforcement learning models to predict real-time budgets;

[0195] Calculate budget deviations and evaluate budget execution status based on real-time budget;

[0196] Determine whether to generate an early warning instruction based on the budget execution status;

[0197] Analyze the budget deviations corresponding to the early warning instructions and provide feedback on the corresponding reasons for the deviations.

[0198] Furthermore, the project characteristic data includes project budget data, project expenditure data and project progress data; the project budget data includes the budget of each sub-project in each stage of the project; the project expenditure data includes the expenditure of each sub-project in each completed stage of the project; the project progress data includes the current stage and stage progress; the current stage is the current stage of the project, and the stage progress is the degree of completion of the current stage;

[0199] The method for predicting real-time budget includes:

[0200] Filter out the sub-project budgets corresponding to the current stage from the project pre-selected data and mark them as stage budgets; obtain the payment progress data corresponding to the current stage, which includes the advance payment and final payment for each sub-project;

[0201] Set different digital labels for each stage of the project; obtain the digital label corresponding to the current stage and mark it as the stage label; use the stage label, stage progress, stage budget and payment progress data as analysis data, input the analysis data into the trained budget prediction model, and predict the real-time budget, which is the sub-project budget corresponding to the stage progress of the current stage of the project.

[0202] Furthermore, the steps of building the budget forecasting model include:

[0203] Step 1: Define the state space;

[0204] The state S includes the stage label, stage progress, stage budget, payment progress data and real-time budget. The stage label, stage progress, stage budget, payment progress data and real-time budget are all represented by real numbers. The state vector dimension is set to 3n+3, where n is the number of sub-projects in the current stage.

[0205] Step 2: Define the action space;

[0206] Action A is the amplitude and direction of the next adjustment of the analysis data; the value of each analysis data is adjusted according to the real number interval [-B, B]; the absolute value of the real number represents the percentage of the amplitude of the adjustment of the analysis data, the positive and negative signs represent the direction of the adjustment of the analysis data, and the action vector dimension is 2n+3;

[0207] Step 3: Design the Actor network structure;

[0208] The Actor network structure includes 1 first input layer, 2 hidden layers and 1 first output layer; the state vector of the first input layer is 3n+3; the 2 hidden layers include the first hidden layer and the second hidden layer, the first hidden layer includes 32 nodes, the activation function is the Relu function, the second hidden layer includes 16 nodes, the activation function is also the Relu function; the first output layer has 3-dimensional continuous action values ​​and no activation function;

[0209] Step 4: Design the Critic network structure;

[0210] The Critic network structure includes 1 second input layer, 2 hidden layers and 1 second output layer; the state vector of the second input layer has 3n+3; the 2 hidden layers include the third hidden layer and the fourth hidden layer, the third hidden layer includes 64 nodes, the activation function is the Relu function, the fourth hidden layer includes 32 nodes, the activation function is the Relu function; the second output layer includes 1 node, without activation function;

[0211] Step 5: Design training configuration:

[0212] Actor network learning rate α1 = 0.0001; Critic network learning rate α2 = 0.0001; experience replay memory size E = 10000; batch size F = 128; update network step size τ = 0.0001; discount factor γ = 0.99.

[0213] Furthermore, the training process of the budget prediction model includes:

[0214] Randomly select action A and obtain environmental feedback (M′, m, ψ), where M′ is the next state, m is the reward value obtained after selecting action A, and ψ is a flag indicating whether it is a terminal state; if the task is completed or failed, ψ = True, and if the task is still continuing, ψ = False; save the action A and the corresponding environmental feedback after each training to the experience replay memory;

[0215] Divide all actions A and corresponding environmental feedback stored in the experience replay memory into a training set and a validation set;

[0216] Get the evaluation Q value PQ of the current Critic network;

[0217] Calculate the target Q value MQ of the Critic network;

[0218] MQ = m + γ × C (M');

[0219] Where C(M') is the quality value obtained by the Critic network evaluating the next state M';

[0220] Calculate the Actor network gradient ▽θ;

[0221] ▽θ=▽AQ(M,A)|M=M',A=A(M')▽θA(M');

[0222] Where A(M') is the action generated according to M'; ▽AQ(M,A) is the Jacobian matrix, and each element in the Jacobian matrix is ​​the partial derivative of MQ learned by the Critic network with respect to each dimension of the Actor network parameter θ; ▽θA(M') represents the gradient of the action with respect to the Actor network parameter θ;

[0223] Calculate the loss value loos; loos = PQ-MQ;

[0224] The model is trained continuously using the training set until the loss value loos continues to decrease and converges, then the training ends.

[0225] Furthermore, the method for calculating the budget deviation includes:

[0226] Filter out the sub-project expenditures corresponding to the current stage from the project expenditure data and mark them as stage expenditures;

[0227] Subtract the corresponding sub-project budget in the real-time budget from each sub-project expenditure in the stage expenditure to obtain the budget deviation of each sub-project;

[0228] The method for evaluating the budget execution status comprises:

[0229] Set different digital labels for each sub-project in each stage of the project and mark them as sub-project labels; set up a sub-project-threshold mapping table to obtain the corresponding deviation threshold according to the sub-project label corresponding to the current stage;

[0230] Compare each budget deviation with the corresponding deviation threshold;

[0231] If the budget deviation is greater than or equal to the deviation threshold, the corresponding budget deviation is marked as budget overrun;

[0232] If the budget deviation is less than the deviation threshold and greater than the opposite of the deviation threshold, the corresponding budget deviation will not be marked;

[0233] If the budget deviation is less than or equal to the opposite of the deviation threshold, the corresponding budget deviation is marked as the budget surplus;

[0234] The budget execution status includes budget surplus status, budget overspending status and budget balance status;

[0235] Add up each budget overrun in sequence to obtain the total budget overrun; add up each budget surplus in sequence to obtain the total budget surplus; then add up the total budget overrun and the total budget surplus to obtain the total budget deviation;

[0236] If the total budget deviation is less than 0, the budget execution status is the budget surplus status;

[0237] If the total budget deviation is greater than 0, the budget execution status is budget overspending;

[0238] If the total budget deviation is equal to 0, the budget execution status is the budget breakeven status.

[0239] Furthermore, the method for determining whether to generate a warning instruction includes:

[0240] Early warning instructions include low-level early warning instructions, medium-level early warning instructions and high-level early warning instructions;

[0241] If the budget execution status is the budget balance status, no warning instruction will be generated;

[0242] If the budget execution status is a budget surplus status or a budget overspending status, the budget surplus and budget overspending are counted respectively, and the budget surplus is added to the budget overspending to obtain the budget anomaly number; the budget deviation is counted, and the budget anomaly number is divided by the budget deviation number to obtain the anomaly ratio D; a ratio threshold C is preset, C∈(0,0.5), and the anomaly ratio D is compared with the ratio threshold C;

[0243] If 0<D<C, a low-level warning instruction is generated;

[0244] If C≤D≤2C, an intermediate warning instruction is generated;

[0245] If 2C<D≤1, an advanced warning instruction is generated.

[0246] Furthermore, the step of feeding back the corresponding deviation cause includes:

[0247] Step a: Preset initialization temperature T max , minimum temperature T min , cooling coefficient δ and maximum number of iterations Let the current temperature T = T max ; A preset cause set, the cause set includes m causes of deviation, where m is an integer greater than 1;

[0248] Step b: Randomly set a feasible solution χ, the feasible solution χ is a deviation cause, and the range of feasible solutions χ is the cause set;

[0249] Step c: determine the fitness function;

[0250] The expression of fitness function is: f = PP;

[0251] In the formula, f is the fitness, YP is the matching degree;

[0252] Step d: Calculate the fitness f corresponding to the feasible solution χ; take the feasible solution χ as the current point, perform random perturbations in the neighborhood of the current point, obtain a new feasible solution χ′, and calculate the fitness f′ corresponding to the new feasible solution χ′;

[0253] Step e: Calculate the fitness difference f″, the expression of the fitness difference f″ is f″=f′-f;

[0254] If the fitness difference f″>0, let χ=χ′, that is, assign the value of the new feasible solution χ′ to the feasible solution χ; if the fitness difference f″≤0, calculate the probability p″, and let χ=χ′ according to the probability p″; the expression of probability p″ is: In the formula, e is a natural constant;

[0255] Step f: Repeat steps d to e until the number of loops reaches the maximum number of iterations. When , the loop ends and enters step g;

[0256] Step g: Set the current temperature T = T × δ, that is, cool down the current temperature in step a, and assign the value after cooling to the current temperature; set the maximum number of iterations The value of the maximum number of iterations after the reduction is assigned to the maximum number of iterations;

[0257] Step h: Repeat steps d to g until the current temperature T < T min When , the loop ends, the deviation cause corresponding to the feasible solution χ is obtained, and feedback is given.

[0258] Furthermore, the method for obtaining the matching degree includes:

[0259] Different digital labels are set for the m types of deviation reasons in the reason set, and marked as reason labels; the sub-project labels corresponding to the budget overrun and budget surplus are marked as abnormal labels; the stage label, stage process, abnormal label, sub-project expenditure corresponding to the abnormal label, and the sub-project budget corresponding to the abnormal label are used as evaluation data, and the evaluation data and the cause label of the deviation cause corresponding to the feasible solution χ are used as test data. The evaluation data and the abnormal label are in one-to-one correspondence, that is, a group of evaluation data corresponds to one abnormal label; the test data is input into the trained matching prediction model to predict the corresponding matching degree;

[0260] The training process of the matching prediction model includes:

[0261] Pre-collect the matching degrees corresponding to b groups of test data, and convert the test data and the corresponding matching degrees into a corresponding set of feature vectors, where b is an integer greater than 1;

[0262] Each group of feature vectors is used as the input of the matching prediction model. The matching prediction model takes a group of predicted matching degrees corresponding to each test data as output, and takes the actual matching degree corresponding to each test data as the prediction target, where the actual matching degree is the pre-collected matching degree corresponding to the test data; minimizing the sum of prediction errors of all test data is used as the training goal; the matching prediction model is trained until the sum of prediction errors reaches convergence and the training is stopped; the matching prediction model is a deep neural network model.

[0263] Further, the deviation reasons of the feedback are analyzed to determine whether to update the deviation reasons;

[0264] The deviation cause of the feedback is marked as the feedback cause; a fitness threshold is preset, the fitness of the feedback cause is marked as the cause fitness, and the cause fitness is compared with the fitness threshold;

[0265] If the cause fitness is less than or equal to the fitness threshold, the deviation cause is updated and the updated deviation cause is added to the cause set;

[0266] If the cause fitness is greater than the fitness threshold, the deviation cause is not updated;

[0267] The method for updating the deviation cause is: input the evaluation data into the trained cause update model, predict the corresponding update label, obtain the corresponding update cause according to the update label, and provide feedback; the update label is the digital label corresponding to the update cause, and the update cause is the updated deviation cause; the training process of the cause update model is consistent with the training process of the matching prediction model, and it is also a deep neural network model; the difference is that during the use of the cause update model, new evaluation data and corresponding deviation causes will be continuously collected to train the cause update model.

[0268] Example 4

[0269] See also Figure 4 As shown, according to another aspect of the present application, an electronic device 500 is also provided. The electronic device 500 may include one or more processors and one or more memories. The memories store computer readable codes, and when the computer readable codes are executed by one or more processors, the project configuration update method based on reinforcement learning as described above may be executed.

[0270] The method or system according to the embodiment of the present application can also be used by Figure 4 The electronic device architecture shown in FIG. Figure 4 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, ROM 503, RAM 504, a communication port 505 connected to a network, an input / output 506, a hard disk 507, etc. The storage device in the electronic device 500, such as ROM 503 or hard disk 507, may store the project configuration update method based on reinforcement learning provided in the present application. Further, the electronic device 500 may also include a user interface 508. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of an electronic device are shown.

[0271] Example 5

[0272] See also Figure 5 As shown, a computer-readable storage medium 600 according to an embodiment of the present application is shown. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are executed by the processor, the project configuration update method based on reinforcement learning according to the embodiment of the present application described with reference to the above figures can be executed. The storage medium 600 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0273] In addition, according to the embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided by the present application, for example: a project configuration update method based on reinforcement learning. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0274] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0275] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Project configuration update method based on reinforcement learning, It is characterized in that include: Collect project characteristics data; Analyze project feature data and use reinforcement learning models to predict real-time budgets; Calculate budget deviations and evaluate budget execution status based on real-time budget; Determine whether to generate an early warning instruction based on the budget execution status; Analyze the budget deviations corresponding to the early warning instructions and provide feedback on the corresponding reasons for the deviations.

2. The project configuration updating method based on reinforcement learning according to claim 1, It is characterized in that The project characteristic data includes project budget data, project expenditure data and project progress data; the project budget data includes the budget of each sub-project in each stage of the project; the project expenditure data includes the expenditure of each sub-project in each completed stage of the project; the project progress data includes the current stage and stage progress; The current stage is the current stage of the project, and the stage progress is the completion degree of the current stage; The method for predicting the real-time budget comprises: Filter out the sub-project budgets corresponding to the current stage from the project pre-selected data and mark them as stage budgets; obtain the payment progress data corresponding to the current stage, which includes the advance payment and final payment for each sub-project; Set different digital labels for each stage of the project; obtain the digital label corresponding to the current stage and mark it as the stage label; use the stage label, stage progress, stage budget and payment progress data as analysis data, input the analysis data into the trained budget prediction model, and predict the real-time budget, which is the sub-project budget corresponding to the stage progress of the current stage of the project.

3. The project configuration updating method based on reinforcement learning according to claim 2, It is characterized in that The steps to build a budget forecasting model include: Step 1: Define the state space; The state S includes the stage label, stage progress, stage budget, payment progress data and real-time budget. The stage label, stage progress, stage budget, payment progress data and real-time budget are all represented by real numbers. The state vector dimension is set to 3n+3, where n is the number of sub-projects in the current stage. Step 2: Define the action space; Action A is the amplitude and direction of the next adjustment of the analysis data; the value of each analysis data is adjusted according to the real number interval [-B, B]; the absolute value of the real number represents the percentage of the amplitude of the adjustment of the analysis data, the positive and negative signs represent the direction of the adjustment of the analysis data, and the action vector dimension is 2n+3; Step 3: Design the Actor network structure; The Actor network structure includes 1 first input layer, 2 hidden layers and 1 first output layer; the state vector of the first input layer is 3n+3; the 2 hidden layers include the first hidden layer and the second hidden layer, the first hidden layer includes 32 nodes, the activation function is the Relu function, the second hidden layer includes 16 nodes, the activation function is also the Relu function; the first output layer has 3-dimensional continuous action values ​​and no activation function; Step 4: Design the Critic network structure; The Critic network structure includes 1 second input layer, 2 hidden layers and 1 second output layer; the state vector of the second input layer has 3n+3; the 2 hidden layers include the third hidden layer and the fourth hidden layer, the third hidden layer includes 64 nodes, the activation function is the Relu function, the fourth hidden layer includes 32 nodes, the activation function is the Relu function; the second output layer includes 1 node, without activation function; Step 5: Design training configuration: Actor network learning rate α 1 =0.0001; Critic network learning rate α 2 =0.0001; experience replay memory size E=10000; batch size F=128; update network step size τ=0.0001; discount factor γ=0.

99.

4. The project configuration updating method based on reinforcement learning according to claim 3, It is characterized in that The training process of the budget forecasting model includes: Randomly select action A and obtain environmental feedback (M′, m, ψ), where M′ is the next state, m is the reward value obtained after selecting action A, and ψ is a flag indicating whether it is a terminal state; if the task is completed or failed, ψ = True, and if the task is still continuing, ψ = False; save the action A and the corresponding environmental feedback after each training to the experience replay memory; All actions A and corresponding environmental feedback stored in the experience replay memory are divided into a training set and a validation set; Get the evaluation Q value PQ of the current Critic network; Calculate the target Q value MQ of the Critic network; MQ = m + γ × C (M'); Where C(M') is the quality value obtained by the Critic network evaluating the next state M'; Calculate the Actor network gradient ▽θ; ▽θ=▽AQ(M,A)|M=M',A=A(M')▽θA(M'); Where A(M') is the action generated according to M'; ▽AQ(M,A) is the Jacobian matrix, and each element in the Jacobian matrix is ​​the partial derivative of MQ learned by the Critic network with respect to each dimension of the Actor network parameter θ; ▽θA(M') represents the gradient of the action with respect to the Actor network parameter θ; Calculate the loss value loos; loos = PQ-MQ; The model is trained continuously using the training set until the loss value loos continues to decrease and converges, then the training ends.

5. The project configuration updating method based on reinforcement learning according to claim 4, It is characterized in that The method for calculating the budget deviation includes: Filter out the sub-project expenditures corresponding to the current stage from the project expenditure data and mark them as stage expenditures; Subtract the corresponding sub-project budget in the real-time budget from each sub-project expenditure in the stage expenditure to obtain the budget deviation of each sub-project; The method for evaluating the budget execution status comprises: Set different digital labels for each sub-project in each stage of the project and mark them as sub-project labels; set up a sub-project-threshold mapping table to obtain the corresponding deviation threshold according to the sub-project label corresponding to the current stage; Compare each budget deviation with the corresponding deviation threshold; If the budget deviation is greater than or equal to the deviation threshold, the corresponding budget deviation is marked as budget overrun; If the budget deviation is less than the deviation threshold and greater than the opposite of the deviation threshold, the corresponding budget deviation will not be marked; If the budget deviation is less than or equal to the opposite of the deviation threshold, the corresponding budget deviation is marked as the budget surplus; The budget execution status includes budget surplus status, budget overspending status and budget balance status; Add up each budget overrun in sequence to obtain the total budget overrun; add up each budget surplus in sequence to obtain the total budget surplus; then add up the total budget overrun and the total budget surplus to obtain the total budget deviation; If the total budget deviation is less than 0, the budget execution status is the budget surplus status; If the total budget deviation is greater than 0, the budget execution status is budget overspending; If the total budget deviation is equal to 0, the budget execution status is the budget breakeven status.

6. The project configuration updating method based on reinforcement learning according to claim 5, It is characterized in that Methods for determining whether to generate an early warning instruction include: Early warning instructions include low-level early warning instructions, medium-level early warning instructions and high-level early warning instructions; If the budget execution status is the budget balance status, no warning instruction will be generated; If the budget execution status is a budget surplus status or a budget overspending status, the budget surplus and budget overspending are counted respectively, and the budget surplus is added to the budget overspending to obtain the budget anomaly number; the budget deviation is counted, and the budget anomaly number is divided by the budget deviation number to obtain the anomaly ratio D; a ratio threshold C is preset, C∈(0,0.5), and the anomaly ratio D is compared with the ratio threshold C; If 0<D<C, a low-level warning instruction is generated; If C≤D≤2C, an intermediate warning instruction is generated; If 2C<D≤1, an advanced warning instruction is generated.

7. The project configuration updating method based on reinforcement learning according to claim 6, It is characterized in that The step of feeding back the corresponding deviation cause includes: Step a: Preset initialization temperature T max , minimum temperature T min , cooling coefficient δ and maximum number of iterations Let the current temperature T = T max ; A preset cause set, the cause set includes m causes of deviation, where m is an integer greater than 1; Step b: Randomly set a feasible solution χ, the feasible solution χ is a deviation cause, and the range of feasible solutions χ is the cause set; Step c: determine the fitness function; The expression of fitness function is: f = PP; In the formula, f is the fitness, YP is the matching degree; Step d: Calculate the fitness f corresponding to the feasible solution χ; take the feasible solution χ as the current point, perform random perturbations in the neighborhood of the current point, obtain a new feasible solution χ′, and calculate the fitness f′ corresponding to the new feasible solution χ′; Step e: Calculate the fitness difference f″, the expression of the fitness difference f″ is f″=f′-f; If the fitness difference f″>0, let χ=χ′, that is, assign the value of the new feasible solution χ′ to the feasible solution χ; if the fitness difference f″≤0, calculate the probability p″, and let χ=χ′ according to the probability p″; the expression of probability p″ is: In the formula, e is a natural constant; Step f: Repeat steps d to e until the number of loops reaches the maximum number of iterations. When , the loop ends and enters step g; Step g: Set the current temperature T = T × δ, that is, cool down the current temperature in step a, and assign the value after cooling to the current temperature; set the maximum number of iterations The value of the maximum number of iterations after the reduction is assigned to the maximum number of iterations; Step h: Repeat steps d to g until the current temperature T < T min When , the loop ends, the deviation cause corresponding to the feasible solution χ is obtained, and feedback is given.

8. The project configuration updating method based on reinforcement learning according to claim 7, It is characterized in that The matching degree can be obtained by: Different digital labels are set for the m types of deviation reasons in the reason set, and marked as reason labels; the sub-project labels corresponding to the budget overrun and budget surplus are marked as abnormal labels; the stage label, stage process, abnormal label, sub-project expenditure corresponding to the abnormal label, and the sub-project budget corresponding to the abnormal label are used as evaluation data, and the evaluation data and the cause label of the deviation cause corresponding to the feasible solution χ are used as test data. The evaluation data and the abnormal label are in one-to-one correspondence, that is, a group of evaluation data corresponds to one abnormal label; the test data is input into the trained matching prediction model to predict the corresponding matching degree; The training process of the matching prediction model includes: Pre-collect the matching degrees corresponding to b groups of test data, and convert the test data and the corresponding matching degrees into a corresponding set of feature vectors, where b is an integer greater than 1; Each group of feature vectors is used as the input of the matching prediction model. The matching prediction model takes a group of predicted matching degrees corresponding to each test data as output, and takes the actual matching degree corresponding to each test data as the prediction target, where the actual matching degree is the pre-collected matching degree corresponding to the test data; minimizing the sum of prediction errors of all test data is used as the training goal; the matching prediction model is trained until the sum of prediction errors reaches convergence and the training is stopped; the matching prediction model is a deep neural network model.

9. The project configuration updating method based on reinforcement learning according to claim 8, It is characterized in that Analyze the causes of feedback deviations and determine whether to update the causes of deviations; The deviation cause of the feedback is marked as the feedback cause; a fitness threshold is preset, the fitness of the feedback cause is marked as the cause fitness, and the cause fitness is compared with the fitness threshold; If the cause fitness is less than or equal to the fitness threshold, the deviation cause is updated and the updated deviation cause is added to the cause set; If the cause fitness is greater than the fitness threshold, the deviation cause is not updated; The method for updating the deviation cause is as follows: input the evaluation data into the trained cause update model, predict the corresponding update label, obtain the corresponding update cause according to the update label, and provide feedback; the update label is the digital label corresponding to the update cause, and the update cause is the updated deviation cause; the training process of the cause update model is consistent with the training process of the matching prediction model, and it is also a deep neural network model; The difference is that during the use of the cause update model, new evaluation data and corresponding deviation causes are continuously collected to train the cause update model.

10. A project configuration update system based on reinforcement learning, implementing the project configuration update method based on reinforcement learning according to any one of claims 1 to 9, It is characterized in that include: Data collection module, used to collect project characteristic data; Data analysis module, used to analyze project feature data and predict real-time budget using reinforcement learning model; Status evaluation module, used to calculate budget deviation and evaluate budget execution status based on real-time budget; The early warning module is used to determine whether to generate an early warning instruction based on the budget execution status; The cause feedback module is used to analyze the budget deviation corresponding to the early warning instruction and feedback the corresponding deviation cause.

Images