Method and medium for training a construction management decision model based on reinforcement learning
By generating training data and optimizing the decision-making model using a reinforcement learning-based construction project simulator, the problem of insufficient training data was solved, and the decision-making ability regarding worker hours and material purchase quantities in construction projects was improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GLODON CO LTD
- Filing Date
- 2022-08-10
- Publication Date
- 2026-05-29
AI Technical Summary
Insufficient training data in existing technologies results in low decision-making capabilities of decision models used for planning work hours and material purchase quantities.
By training a construction management decision-making model based on reinforcement learning, simulating training data using a construction project simulator, and optimizing the decision-making model through a policy gradient algorithm, highly realistic and diverse training data is generated, thus solving the problem of insufficient training data.
This improves the decision support capabilities of the decision-making model, enabling it to provide better decision recommendations on worker hours and material purchase quantities in complex and uncertain construction environments.
Smart Images
Figure CN115330182B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a method for training a construction management decision-making model based on reinforcement learning, a method for planning worker hours and material purchase quantities based on reinforcement learning, a construction project simulator, and a computer-readable storage medium. Background Technology
[0002] In construction projects, project managers frequently need to make resource planning decisions, primarily planning the working hours of workers in various trades and determining the purchase quantities of various materials. Errors in planning working hours and material purchase quantities can lead to losses in project costs and timelines. However, the complexity and uncertainty of construction projects often make making sound decisions extremely difficult.
[0003] To alleviate the decision-making burden on project managers, some scholars have used machine learning methods to train decision-making models based on deep neural networks, providing suggestions for planning work hours and material purchase quantities. Theoretically, a well-trained decision-making model can overcome the uncertainties and complexities of construction projects and make reasonable decisions. However, deep learning-based methods rely on decision-making data from real construction projects for training. The high cost of collecting such data leads to insufficient training data, thus reducing the decision-making ability of the trained model.
[0004] There is currently no effective solution to the technical problem that insufficient training data in existing technologies leads to low decision-making ability of decision models used for planning work hours and material purchase quantities. Summary of the Invention
[0005] The purpose of this invention is to provide a method for training a construction management decision-making model based on reinforcement learning, a method for planning worker hours and material purchase quantities based on reinforcement learning, a construction project simulator, and a computer-readable storage medium, which can solve the technical problem that insufficient training data in the prior art leads to low decision-making ability of the decision-making model used for planning worker hours and material purchase quantities.
[0006] One aspect of the present invention provides a method for training a construction management decision model based on reinforcement learning, comprising: acquiring initial construction parameters of a target construction project and historical environmental parameters of the region where the target construction project is located; wherein the target construction project is a construction project for which worker hours and material purchase quantities are to be planned; constructing a construction project simulator based on the initial construction parameters and the historical environmental parameters as basic data; simulating observation trajectories, action trajectories, state trajectories, and incentive sequences related to the construction process based on the construction project simulator; and optimizing a decision model with a preset structure based on the observation trajectories, action trajectories, state trajectories, and incentive sequences, so that the optimized decision model can be used to plan worker hours and material purchase quantities.
[0007] Optionally, the step of simulating the observation trajectory, action trajectory, state trajectory, and incentive sequence related to the construction process using the construction project simulator includes: inputting the state parameters of day T before construction into a preset observation function to simulate the observation parameters of day T of construction; inputting the observation parameters of day T of construction into the decision model to simulate the action parameters of day T of construction; inputting the state parameters and action parameters of day T of construction into the construction project simulator to simulate the state parameters of day T+1 of construction; inputting the state parameters of day T and day T+1 of construction into a preset incentive function to simulate the incentive value of day T of construction; and constructing the observation trajectory, the action trajectory, the state trajectory, and the incentive sequence according to the simulated observation parameters, action parameters, state parameters, and incentive values, respectively; wherein T is an integer greater than or equal to 1 and less than or equal to N, N is the total construction period of the target construction project and N is less than or equal to the maximum number of construction days specified in the construction contract of the target construction project; wherein, when T is 1, the state parameters of the first day of construction are obtained from a preset database.
[0008] Optionally, the step of inputting the observation parameters of construction day T into the decision model to output the action parameters of construction day T includes: inputting the observation parameters of construction day T into the standardization module of the decision model to obtain standardized observation parameters; inputting the standardized observation parameters into the deep neural network of the decision model to obtain the expected values corresponding to worker hours and material purchase quantities, and obtaining the variances corresponding to worker hours and material purchase quantities from the model parameters of the decision model; constructing normal distributions of worker hours and material purchase quantities based on the expected values and variances corresponding to worker hours and material purchase quantities, respectively; sampling the normal distributions of worker hours and material purchase quantities through the sampling module of the decision model to obtain standardized worker hours and standardized material purchase quantities; and inputting the standardized worker hours and standardized material purchase quantities into the inverse standardization module of the decision model to obtain the action parameters of construction day T.
[0009] Optionally, the step of inputting the state parameters and action parameters of construction day T into the construction project simulator to simulate the state parameters of construction day T+1 includes: constructing a stochastic process based on the historical environmental parameters using the environmental model of the construction project simulator, and independently sampling the environmental parameters of each day of construction based on the constructed stochastic process during the simulation to simulate the environmental state of each day of construction; and simulating the construction state of construction day T+1 using the project model of the construction project simulator based on the state parameters and action parameters of construction day T.
[0010] Optionally, the step of constructing a stochastic process based on the historical environmental parameters using the environmental model of the construction project simulator, and independently sampling the daily environmental parameters of the construction project according to the constructed stochastic process during the simulation to simulate the daily environmental state of the construction project, includes: constructing a stochastic process based on the historical meteorological index in the historical environmental parameters using the environmental model, and independently sampling the daily meteorological index of the construction project according to the constructed stochastic process during the simulation to simulate the daily meteorological index of the construction project; constructing a stochastic process based on the historical material prices in the historical environmental parameters using the environmental model, and sampling the daily material prices of the construction project according to the constructed stochastic process during the simulation to simulate the daily environmental state of the construction project and the daily material prices of the construction project; and determining the simulated daily meteorological index and material prices of the construction project as the daily environmental state of the construction project using the environmental model.
[0011] Optionally, the simulation of the construction state on day T+1 using the project model of the construction project simulator based on the state parameters and action parameters of day T includes: calculating the remaining construction funds on day T+1 based on the remaining construction funds in the construction state on day T, the material prices in the environmental state, the worker hours and material purchase quantity in the action parameters, and the hourly wage of workers stipulated in the construction contract; calculating the worker efficiency on day T based on the weather index in the environmental state on day T and the worker hours on day T before construction; and calculating the remaining construction funds on day T based on the remaining construction funds in the construction state on day T. The construction progress increment on day T is calculated using labor resources, worker efficiency, and worker hours in the action parameters. The sum of the construction progress on day T and the construction progress increment is then used as the construction progress on day T+1. The construction resource consumption is calculated based on the progress increment on day T using the project model. The remaining construction resources on day T+1 are calculated based on the remaining construction resources in the construction status on day T, the material purchase quantity in the action parameters, and the construction resource consumption. The worker efficiency on day T, the remaining construction funds, construction progress, and remaining construction resources on day T+1 are used as the construction status on day T+1 using the project model.
[0012] Optionally, the excitation function is:
[0013] r T =ω P ×pr T +ω M ×mr T +ω S ×sr T
[0014] Where, r TLet pr be the incentive value on day T of construction. T For the incremental construction progress on day T, mr T For the net income on day T of construction, sr T To assess worker safety on day T of construction, ω P ω M and ω S pr T ,mr T and sr T The weight value and ω P ω M and ω S The sum is 1.
[0015] Another aspect of the present invention provides a method for planning worker hours and material purchase quantities based on reinforcement learning, characterized by comprising: collecting today's observation parameters from the construction project for which worker hours and material purchase quantities are to be planned; wherein, the today's observation parameters include the current construction funding status, the current construction resource status, the current construction progress status, the current environmental status, and a predicted value for the future environmental status; inputting the today's observation parameters into a decision model that can be used to plan worker hours and material purchase quantities so that the decision model plans today's worker hours and material purchase quantities; wherein, the decision model is obtained by the method of training a construction management decision model based on reinforcement learning as described in any of the above embodiments.
[0016] Another aspect of the present invention provides a construction project simulator, including a project model and an environment model. The project model includes a funding module, a resource module, an efficiency module, and a schedule module. The environment model includes an environment module. The funding module is used to determine remaining construction funds, the resource module is used to determine remaining construction resources, the efficiency module is used to determine worker efficiency, the schedule module is used to determine construction progress, and the environment module is used to determine weather indicators and material prices. The interaction between the various modules is controlled through the project model and the environment model to determine the status parameters for the next day of construction based on the status parameters of the target construction project on the current construction day. The target construction project is a construction project for which worker hours and material purchase quantities need to be planned, and the status parameters include remaining construction funds, remaining construction resources, worker efficiency, construction progress, weather indicators, and material prices.
[0017] Another aspect of the present invention provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for training a construction management decision model based on reinforcement learning and / or the method for planning worker hours and material purchase quantities based on reinforcement learning as described in any of the above embodiments.
[0018] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for training a construction management decision model based on reinforcement learning and / or the method for planning worker hours and material purchase quantities based on reinforcement learning as described in any of the above embodiments.
[0019] To overcome the deficiency in existing technologies where insufficient training data leads to low decision-making capabilities in models used for planning worker hours and material purchases, this invention provides a method for training a construction management decision-making model based on reinforcement learning. Training data is simulated using real construction projects, and the decision-making model is trained based on this simulated data, thus solving the problem of insufficient training data in existing technologies. Furthermore, training the decision-making model using reinforcement learning allows the model to find optimal decisions through repeated trial and error in realistic and diverse construction simulation environments. Compared to deep learning, it can be trained without relying on real decision-making data. Therefore, this further addresses the technical problem of low decision-making capabilities in existing technologies due to insufficient real training data for planning worker hours and material purchases, enabling the final trained decision-making model to provide better decision-making suggestions for planning worker hours and material purchases. Attached Figure Description
[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0021] Figure 1 Here is a flowchart of the method for training a construction management decision-making model based on reinforcement learning, as described in Example 1.
[0022] Figure 2 This is a structural schematic diagram of the construction project simulator in Example 1;
[0023] Figure 3 This is a schematic diagram of the decision model in Example 1;
[0024] Figure 4 This is a schematic diagram of the overall process of training a construction management decision-making model based on reinforcement learning, as shown in Example 1.
[0025] Figure 5 This is a flowchart of the method for planning worker hours and material purchase quantities based on reinforcement learning in Example 2;
[0026] Figure 6 This is a schematic diagram illustrating the process of planning worker hours and material purchase quantities based on reinforcement learning in Example 2.
[0027] Figure 7 This is a block diagram of the device for training a construction management decision-making model based on reinforcement learning, as described in Example 3.
[0028] Figure 8 This is a block diagram of the device for planning worker hours and material purchase quantities based on reinforcement learning, as described in Example 4.
[0029] Figure 9 A block diagram of a computer device provided in Embodiment 5 is shown, which is suitable for implementing a method for training a construction management decision model based on reinforcement learning and / or a method for planning worker hours and material purchase quantities based on reinforcement learning. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.
[0031] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0032] Example 1
[0033] Figure 1 Here is a flowchart of the method for training a construction management decision-making model based on reinforcement learning, as described in Example 1. Figure 1 As shown, the method for training a construction management decision-making model based on reinforcement learning includes steps S1 to S4, wherein:
[0034] Step S1: Obtain the initial construction parameters of the target construction project and the historical environmental parameters of the region where the target construction project is located; wherein, the target construction project is a construction project for which worker hours and material purchase quantities are to be planned.
[0035] Data from the construction contract for the target project, specifying the construction process, can be used as initial construction parameters. These initial parameters include: total project volume, start date, maximum number of construction days, hourly wages for various types of workers, and maximum storage capacity for various materials. Worker types include: steelworkers, formwork workers, and concrete workers; material types include: steel bars, formwork, and concrete.
[0036] Historical weather indices and historical material prices for the target construction project area can be used as historical environmental parameters. Historical weather indices include the daily average maximum and minimum weather indices for each day of the historical year. When the historical year is one year, the daily average maximum and minimum weather indices refer to the highest and lowest weather indices for each day of that year, respectively. When the historical year spans multiple years, the daily average maximum and minimum weather indices refer to the average of the highest and lowest weather indices for each day of those multiple years, respectively. Weather indices include temperature, rainfall, and wind speed. Historical material prices include the daily average maximum and minimum material prices for each day of the historical year. When the historical year is one year, the daily average maximum and minimum material prices refer to the highest and lowest material prices for each day of that year, respectively. When the historical year spans multiple years, the daily average maximum and minimum material prices refer to the average of the highest and lowest material prices for each day of those multiple years, respectively. Material prices include steel reinforcement prices, formwork prices, and concrete prices.
[0037] It should be noted that all parameters included in the above initial construction parameters and historical environmental parameters are obtained from real construction contracts and historical records, ensuring the authenticity of the simulation.
[0038] Step S2: Construct a construction project simulator based on the initial construction parameters and the historical environmental parameters.
[0039] Step S3: Simulate the observation trajectory, action trajectory, state trajectory, and excitation sequence related to the construction process using the construction project simulator.
[0040] The observation trajectory is {o1, ..., o} T-1 o T , ..., o N}, o T The observation parameters are for day T of construction; the movement trajectory is {a1, ..., a...} T-1 a T , ..., a N}, a T Let {s1, ..., s2} be the action parameters for day T of construction; the state trajectory is {s1, ..., s3}. T, ..., s N+1}, s T Let r be the state parameters on day T of construction; the excitation sequence be {r1, ..., r...}. T-1 r T ,...,r N}, r T This is the incentive value on day T of construction.
[0041] Where T is an integer greater than or equal to 1 and less than or equal to N, and N is the total construction period of the target construction project, and N is less than or equal to the maximum number of construction days specified in the construction contract of the target construction project.
[0042] Observational parameters include quantitative indicators used to plan worker hours and material purchase quantities; action parameters include the planned worker hours and material purchase quantities; state parameters include construction status and environmental status; and incentive values are used to evaluate action parameters. It should be noted that the specific parameters included in the observational, action, and state parameters must be defined in advance before formal training. Observational parameters include: current construction funding status, current construction resource status, current construction progress status, current environmental status, historical weather and work hour status, and future environmental status. Examples of the specific types of observational parameters can be found in Table 1 below. Action parameters include: worker hours and material purchase quantities. Examples of the specific types of action parameters can be found in Table 2 below. The construction status in the state parameters includes: remaining construction funding, remaining construction resources, worker efficiency, and construction progress; the environmental status in the state parameters includes: weather indicators and material prices in the target construction project area.
[0043] Table 1
[0044] Parameter types Specific parameters Current construction funding status Current cash balance and project progress payments expected to be received within the next 3 days. Current construction resource status Inventory of reinforcing steel, formwork, and concrete Current construction progress status The amount of work completed in processes such as tying reinforcing bars, erecting formwork, and pouring concrete should be... Current environmental status Current temperature, rainfall, wind speed, and material prices Historical status of weather and working hours Recent 3 days' actual working hours for each job type and weather information records Future environmental state Weather forecasts and material price forecasts for the next 5 days
[0045] Table 2
[0046]
[0047] Furthermore, the structure of the decision-making model needs to be determined before formal training. Specifically, the input and output of the decision-making model are defined according to the decision-making task and objective, i.e., which parameters should be included in the observation parameters and action parameters; the structure of the decision-making model is determined based on the number of parameter terms in the observation parameters and action parameters, and the parameters of the decision-making model are initialized. The decision-making model is a neural network model. The number of nodes in the input layer of the decision-making model equals the number of parameter terms in the observation parameters, the number of nodes in the output layer equals the number of parameter terms in the action parameters, and the number of hidden layers in the decision-making model is determined using empirical formulas to analyze the number of parameter terms in the observation parameters and action parameters.
[0048] Among them, the construction project simulator Mε (s T a T ) is used to simulate the state parameters on day T+1 of construction, i.e., s T+1 =M ε (s T a T Construction Project Simulator M ε (s T a T The project model PM(ε) is included. p s T a T ) and environmental model EM(ε E Project model PM(ε) p s T a T The state parameter s used to simulate the construction on day T+1 T+1 Construction status in ps T+1 Environmental model EM(ε) E The state parameter s used to simulate the construction on day T+1 T+1 Environment state in es T+1 ε is a model parameter of the construction project simulator. It is an invariant that reflects the characteristics of the target construction project being simulated and is related to state updates, including the initial construction parameter ε. p and historical environmental parameters ε E .
[0049] like Figure 2 As shown, the project model in the construction project simulator includes a funding module, a resource module, an efficiency module, and a schedule module, while the environment model includes an environment module. The funding module determines the remaining construction funds, the resource module determines the remaining construction resources, the efficiency module determines worker efficiency, the schedule module determines the construction schedule, and the environment module determines weather indicators and material prices. The interaction between these modules is controlled through the project model and the environment model to determine the status parameters for the next day's construction based on the status parameters of the target construction project on the current day.
[0050] As an optional embodiment, step S3 includes steps S31 to S35, wherein:
[0051] Step S31: Input the state parameters of construction day T into the preset observation function to simulate the observation parameters of construction day T;
[0052] Step S32: Input the observation parameters of construction day T into the decision model to simulate the action parameters of construction day T;
[0053] Step S33: Input the status parameters and action parameters of construction day T into the construction project simulator to simulate the status parameters of construction day T+1.
[0054] Step S34: Input the state parameters of construction day T and construction day T+1 into the preset excitation function to simulate the excitation value of construction day T;
[0055] Step S35: Construct the observation trajectory, the action trajectory, the state trajectory, and the excitation sequence based on the simulated observation parameters, action parameters, state parameters, and excitation values, respectively.
[0056] When T is 1, the pre-set status parameters for the first day of construction are obtained from the preset database.
[0057] Repeat steps S31 to S34 until the daily observation parameters, action parameters, and stimulus values from day 1 to day N of construction are simulated, and the daily state parameters from day 2 to day N+1 of construction are simulated. Then, in chronological order, all observation parameters from day 1 to day N of construction are put into a set to obtain the observation trajectory, all action parameters from day 1 to day N of construction are put into a set to obtain the action trajectory, all state parameters from day 1 to day N+1 of construction are put into a set to obtain the state trajectory, and all stimulus values from day 1 to day N of construction are put into a set to obtain the stimulus sequence.
[0058] Wherein, the observation function O(S) T Used to collect observation parameters in simulated real construction projects. T The process, i.e., o T =O(S) T ), O(S T S is a set of historical states. T ={s1, ..., s T}
[0059] As an optional embodiment, setting the status parameters for the first day of construction includes:
[0060] The preset remaining construction funds, preset construction resources, preset worker efficiency, and preset construction progress corresponding to the first day of construction are respectively used as the remaining construction funds, remaining construction resources, worker efficiency, and construction progress for the first day of construction.
[0061] The weather index for the first day of construction is estimated based on the historical weather index in the historical environmental parameters, and the material price for the first day of construction is estimated based on the historical material price in the historical environmental parameters.
[0062] The remaining construction funds, remaining construction resources, worker efficiency, construction progress, weather index, and material prices on the first day of construction are used as the status parameters for the first day of construction.
[0063] Among them, the preset construction resources and preset construction progress can be 0, and the preset worker efficiency can be 100%.
[0064] The weather index for the first day of construction is estimated based on historical weather indices from the historical environmental parameters, and the material price for the first day of construction is estimated based on historical material prices from the historical environmental parameters. This includes: selecting the average highest and average lowest weather indices corresponding to the date of the first day of construction from the historical weather indices, and randomly selecting a value from the range of weather indices formed by the selected average highest and average lowest weather indices as the weather index for the first day of construction; and selecting the average highest and average lowest material prices corresponding to the date of the first day of construction from the historical material prices, and randomly selecting a value from the range of material prices formed by the selected average highest and average lowest material prices as the material price for the first day of construction. For example, if the date of the first day of construction is April 1st, then values can be randomly selected from the temperature index range, rainfall index range, wind speed index range, steel reinforcement price range, formwork price range, and concrete price range corresponding to April 1st as the temperature, rainfall, wind speed, steel reinforcement price, formwork price, and concrete price for the first day of construction, respectively.
[0065] As an optional embodiment, step S32 includes:
[0066] Input the observation parameters of day T of construction into the standardization module of the decision model to obtain the standardized observation parameters;
[0067] The standardized observation parameters are input into the deep neural network in the decision model to obtain the expected values of worker hours and material purchase quantity, and the variances of worker hours and material purchase quantity are obtained from the model parameters of the decision model.
[0068] We construct normal distributions for worker hours and material purchase quantities based on the expected and variance of worker hours and the expected and variance of material purchase quantities, respectively.
[0069] The normal distribution of worker hours and the normal distribution of material purchase quantity are sampled by the sampling module in the decision model to obtain the standardized worker hours and standardized material purchase quantity respectively.
[0070] The standardized worker hours and standardized material purchase quantities are input into the inverse standardization module of the decision model to obtain the action parameters for day T of construction.
[0071] Among them, such as Figure 3 As shown, the decision model P θ (o T The model parameters θ include: the weights and biases of the deep neural network, the variance corresponding to worker hours, and the variance corresponding to material purchase quantities. The purpose of training is to obtain the optimal decision model parameters θ. * Decision model P θ (o T This is used to predict the optimal action under the current observation. This embodiment introduces a probabilistic mechanism into the decision model, meaning the action parameters output by the decision model are randomly selected from the corresponding normal distributions, ensuring that each action parameter has a unique characteristic. Specifically, the decision model determines the expected value corresponding to each type of worker's working hours and the expected value corresponding to each type of material purchase quantity. Based on the expected value and variance of each type of worker's working hours, a normal distribution for each worker's working hours is constructed, and samples are taken from the normal distribution of each worker's working hours to obtain the worker's working hours on day T of construction. Similarly, based on the expected value and variance of each type of material purchase quantity, the decision model constructs a normal distribution for each material purchase quantity, and samples are taken from the normal distribution of each material purchase quantity to obtain the material purchase quantity on day T of construction.
[0072] As an optional embodiment, step S33 includes:
[0073] Step S331: Construct a random process based on the historical environmental parameters using the environmental model of the construction project simulator, and independently sample the environmental parameters of each day of construction according to the constructed random process during the simulation process to simulate the environmental state of each day of construction.
[0074] Step S332: Simulate the construction status on day T+1 using the project model of the construction project simulator based on the status parameters and action parameters of day T.
[0075] Among them, the environmental state and construction state on the construction day T+1 can be used as the state parameters on the construction day T+1.
[0076] As an optional embodiment, step S331 includes:
[0077] The environmental model is used to construct a stochastic process based on historical meteorological indices from the historical environmental parameters. During the simulation, the meteorological indices for each day of construction are independently sampled based on the constructed stochastic process to simulate the meteorological indices for each day of construction.
[0078] The environmental model is used to construct a stochastic process based on historical material prices in the historical environmental parameters. During the simulation, the material prices of each construction day are sampled according to the constructed stochastic process to simulate the environmental conditions and material prices of each construction day.
[0079] The environmental model is used to determine the daily environmental conditions of construction based on the simulated daily weather index and material prices.
[0080] Specifically, when simulating the environmental conditions of each day of construction, the average highest and lowest meteorological indices corresponding to the date of construction can be selected from the historical meteorological indices. A value is then randomly selected from the range of meteorological indices formed by the selected average highest and lowest meteorological indices as the meteorological index for that day of construction. Similarly, the average highest and lowest material prices corresponding to the date of construction can be selected from the historical material prices. A value is then randomly selected from the range of material prices formed by the selected average highest and lowest material prices as the material price for that day of construction. For example, if the second day of construction is May 31st, values can be randomly selected from the temperature index range, rainfall index range, wind speed index range, steel reinforcement price range, formwork price range, and concrete price range corresponding to May 31st as the temperature, rainfall, wind speed, steel reinforcement price, formwork price, and concrete price for the second day of construction.
[0081] As an optional embodiment, step S332 includes:
[0082] Step S3321: Calculate the remaining construction funds for construction day T+1 using the project model based on the remaining construction funds in the construction status on day T, the material prices in the environmental status, the worker hours and material purchase quantity in the action parameters, and the hourly wage of workers stipulated in the construction contract.
[0083] Step S3322: Calculate the worker efficiency on day T of construction using the project model based on the meteorological index in the environmental conditions on day T of construction and the worker hours on day T before construction.
[0084] Step S3323: Calculate the construction progress increment for day T using the project model based on the remaining construction resources and worker efficiency in the construction status on day T and the worker hours in the action parameters, and calculate the sum of the construction progress on day T and the construction progress increment as the construction progress for day T+1.
[0085] Step S3324: Calculate the construction resource consumption based on the progress increment on day T of construction using the project model, and calculate the remaining construction resources on day T+1 of construction based on the remaining construction resources in the construction status on day T, the material purchase quantity in the action parameters, and the construction resource consumption.
[0086] Step S3325: Using the project model, the worker efficiency on construction day T, the remaining construction funds, construction progress, and remaining construction resources on day T+1 are taken as the construction status on day T+1.
[0087] Step S3322 includes: determining the worker fatigue index for day T of construction based on the meteorological index of the environmental conditions on day T and the worker hours on day T prior to construction; and estimating the worker efficiency for day T based on the worker fatigue index. Both the meteorological index and worker hours can affect worker efficiency, especially when worker hours are too long, leading to greater worker fatigue and potentially causing absenteeism or exhaustion the following day, thus impacting worker efficiency. For example, two normal distributions can be determined based on the meteorological index and worker fatigue index. The absenteeism rate is sampled from the first normal distribution, and the baseline worker efficiency is sampled from the second normal distribution. The baseline worker efficiency is then reduced according to the absenteeism rate to obtain the final worker efficiency.
[0088] Step S3323 includes: estimating the first alternative construction progress increment for construction day T based on the product of worker efficiency in the construction status on construction day T and worker hours in the action parameters on construction day T; estimating the second alternative construction progress increment for construction day T based on the sum of remaining construction resources in the construction status on construction day T and material purchase quantity in the action parameters on construction day T; estimating the third alternative construction progress increment based on the preset construction procedures; calculating the difference between the total amount of work stipulated in the construction contract and the total construction progress on day T-1 before construction as the fourth alternative construction progress increment; and selecting the minimum value from the first alternative construction progress increment, the second alternative construction progress increment, the third alternative construction progress increment, and the fourth alternative construction progress increment as the construction progress increment for construction day T.
[0089] It should be noted that, since worker efficiency is random and is taken into account when determining the incremental construction progress, the incremental construction progress is inherently random; furthermore, since the incremental construction progress is taken into account when determining the remaining construction resources, the remaining construction resources are inherently random.
[0090] As an optional embodiment, the excitation function R in step S34 T (s T s T+1 )for:
[0091] r T =ω P ×pr T +ω M ×mr T +ω S ×sr T
[0092] Where, r T Let pr be the incentive value on day T of construction. T For the incremental construction progress on day T, mr T For the net income on day T of construction, sr T To assess worker safety on day T of construction, ω P ω M and ω S pr T ,mr T and sr T The weight value and ω P ω M and ω S The sum is 1. This can be determined based on pr. T ,mr T and sr T The importance of these three objectives is adjusted by weighting the values ω. p ω m and ω s Incentive values include dense incentives and sparse incentives, which correspond to process objectives and outcome objectives, respectively. Table 3 provides examples of each type of incentive value.
[0093] Table 3
[0094]
[0095]
[0096] Step S4: Optimize the decision model based on the observation trajectory, the action trajectory, the state trajectory, and the stimulus sequence, so that the optimized decision model can be used to plan worker hours and material purchase quantities.
[0097] This involves simulating the entire construction process M times to generate M observation trajectories, M action trajectories, M state trajectories, and M excitation sequences. These M observation trajectories, M action trajectories, M state trajectories, and M excitation sequences are then used in a policy gradient algorithm to optimize the decision model, yielding the optimal decision model parameters θ. * Where θ * It is the model parameter with the highest expected cumulative incentive value obtained by simulating the entire construction process of the target construction project, that is... Furthermore, through multiple optimization training iterations, the optimal decision model parameters θ of the optimized decision model are obtained. * To achieve the desired outcome, the optimized decision-making model is used to plan worker hours and material purchases.
[0098] During the optimization process, each training data point is {o} T a T r T s T+1}
[0099] like Figure 4 As shown, the state parameters of the next day of construction are simulated using a construction project simulator to obtain the state trajectory, the observation trajectory is simulated using an observation function, the stimulus sequence is simulated using an activation function, and the action trajectory is simulated using a decision model. Then, the simulated observation trajectory, action trajectory, state trajectory, and stimulus sequence are stored in the training data storage for subsequent model training.
[0100] It should be noted that, to ensure the diversity of training data, the M "state trajectories, action trajectories, and observation trajectories" obtained throughout the M simulated construction processes must be different. Specifically, firstly, this embodiment constructs a stochastic process based on historical environmental parameters, and randomly samples within a selected range according to the constructed stochastic process in each simulation to ensure that the meteorological indicators and material prices sampled each time are different; secondly, a probabilistic mechanism is adopted for worker efficiency. For example, due to absenteeism or other reasons, the worker's work efficiency on day T is a random variable, and the project progress increment on day T is a random variable concerning the planned worker hours on day T; thirdly, a probabilistic mechanism is introduced into the decision-making model, and the decided worker hours and material purchase quantities are randomly sampled within their respective normal distributions to ensure that the output worker hours and material purchase quantities are different each time.
[0101] The beneficial effects of this invention are: 1. By simulating training data and training the decision model based on the policy gradient algorithm in reinforcement learning, the problem of insufficient training data caused by the high cost of collecting real training data in existing deep learning-based methods is solved; 2. By simulating training data through relevant objective laws and collecting relevant parameters from corresponding real construction sites and environments, the high fidelity of the training data is ensured; 3. By introducing a random process in the simulation of training data, the diversity of the simulated training data is ensured; 4. Because highly realistic and diverse training data is simulated, and the amount of training data is much larger than the real training data that can be collected, the decision-making assistance capability of the decision model trained according to the method proposed in this invention is higher than that of the decision model trained based on deep learning methods.
[0102] Example 2
[0103] Embodiment 2 of the present invention also provides a method for planning worker hours and material purchase quantities based on reinforcement learning. Some technical features of this method correspond to those of the method provided in Embodiment 1 above. The corresponding technical features and effects will not be detailed in this embodiment, but relevant details can be found in Embodiment 1 above. Specifically, Figure 5 This is a flowchart illustrating the reinforcement learning-based method for planning worker hours and material purchase quantities in Example 2. Figure 5 As shown, the method for planning worker hours and material purchase quantities based on reinforcement learning may include steps A1 and A2, wherein:
[0104] Step A1: Collect today's observation parameters from the construction projects for which worker hours and material purchase quantities are to be planned; wherein, the today's observation parameters include the current construction funding status, the current construction resource status, the current construction progress status, the current environmental status, and the predicted value of the future environmental status;
[0105] Step A2: Input the observed parameters of today into a decision model that can be used to plan worker hours and material purchases so that the decision model can plan today's worker hours and material purchases; wherein, the decision model is obtained through any of the implementation methods in Example 1.
[0106] Specifically, today's observed parameters can be understood as the observed parameters for construction day T, and the planned worker hours and material purchase quantities for today can be understood as the action parameters for construction day T. The project manager can determine whether to accept the decision suggestion output in step A2. If accepted, the worker hours and material purchase quantities are planned based on this output; if not accepted or not fully accepted, the project manager decides the planned worker hours and material purchase quantities for today. Furthermore, steps A1 and A2 can be repeated until the actual construction project is completed. Figure 6 As shown, after collecting real environmental parameters and real construction parameters, the observation parameters for today are determined. Then, the observation parameters are input into a decision model that can be used to plan worker hours and material purchase quantities so that the decision model can output decision recommendations. The project manager then makes actual plans based on the decision recommendations.
[0107] Example 3
[0108] Embodiment 3 of the present invention also provides an apparatus for training a construction management decision-making model based on reinforcement learning. Some technical features of this apparatus correspond to the method provided in Embodiment 1 above. The corresponding technical features and effects will not be detailed in this embodiment, but relevant details can be found in Embodiment 1 above. Specifically, Figure 7 This is a block diagram of the device for training a construction management decision-making model based on reinforcement learning, as described in Example 3. Figure 7 As shown, the device 700 for training a construction management decision-making model based on reinforcement learning may include:
[0109] The acquisition module 701 is used to acquire the initial construction parameters of the target construction project and the historical environmental parameters of the area where the target construction project is located; wherein, the target construction project is a construction project for which worker hours and material purchase quantities are to be planned.
[0110] The construction module 702 is used to construct a construction project simulator based on the initial construction parameters and the historical environmental parameters.
[0111] Simulation module 703 is used to simulate observation trajectories, action trajectories, state trajectories and excitation sequences related to the construction process based on the construction project simulator;
[0112] The optimization module 704 is used to optimize the decision model with a preset structure based on the observation trajectory, the action trajectory, the state trajectory and the stimulus sequence, so that the optimized decision model can be used to plan worker hours and material purchase quantities.
[0113] Example 4
[0114] Embodiment 4 of the present invention also provides a device for planning worker hours and material purchase quantities based on reinforcement learning. Some technical features of this device correspond to the methods provided in Embodiments 1 and 2 above. The corresponding technical features and effects will not be detailed in this embodiment, but relevant details can be found in Embodiments 1 and 2 above. Specifically, Figure 8 This is a block diagram of the device for planning worker hours and material purchase quantities based on reinforcement learning, as described in Embodiment 4. Figure 8 As shown, the apparatus 800 for planning worker hours and material purchases based on reinforcement learning may include:
[0115] The collection module 801 is used to collect today's observation parameters from construction projects with planned worker hours and material purchase quantities; wherein, the today's observation parameters include the current construction funding status, the current construction resource status, the current construction progress status, the current environmental status, and the predicted value of the future environmental status;
[0116] The planning module 802 is used to input the observed parameters of today into a decision model that can be used to plan the working hours and material purchases of today, so that the decision model can plan the working hours and material purchases of today; wherein, the decision model is obtained by the method described in Embodiment 1.
[0117] Example 5
[0118] Figure 9A block diagram of a computer device, as provided in Embodiment 5, suitable for implementing a method for training a construction management decision-making model based on reinforcement learning and / or a method for planning worker hours and material purchase quantities based on reinforcement learning. In this embodiment, the computer device 900 can be a smartphone, tablet, laptop, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc., that executes a program. Figure 9 As shown, the computer device 900 in this embodiment includes, but is not limited to, a memory 901, a processor 902, and a network interface 903 that are communicatively connected to each other via a system bus. It should be noted that... Figure 9 Only a computer device 900 with components 901-903 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0119] In this embodiment, the memory 903 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 901 may be an internal storage unit of the computer device 900, such as the hard disk or memory of the computer device 900. In other embodiments, the memory 901 may also be an external storage device of the computer device 900, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 900. Of course, the memory 901 may also include both the internal storage unit and the external storage device of the computer device 900. In this embodiment, the memory 901 is typically used to store the operating system and various application software installed on the computer device 900, such as the program code for methods of training construction management decision models based on reinforcement learning and / or methods of planning worker hours and material purchase quantities based on reinforcement learning.
[0120] In some embodiments, processor 902 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 902 is typically used to control the overall operation of computer device 900. For example, it performs control and processing related to data interaction or communication with computer device 900. In this embodiment, processor 902 is used to run program code stored in memory 901 for methods based on reinforcement learning to train construction management decision-making models and / or methods based on reinforcement learning to plan worker hours and material purchase quantities.
[0121] In this embodiment, the method for training a construction management decision model based on reinforcement learning and / or the method for planning worker hours and material purchase quantities based on reinforcement learning, stored in memory 901, can be further divided into one or more program modules and executed by one or more processors (processor 902 in this embodiment) to complete the present invention.
[0122] Network interface 903 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 900 and other computer devices. For example, network interface 903 is used to connect computer device 900 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 900 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.
[0123] Example 6
[0124] This embodiment also provides a computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. When the computer program is executed by a processor, it implements the steps of a method for training a construction management decision model based on reinforcement learning and / or a method for planning worker hours and material purchase quantities based on reinforcement learning.
[0125] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.
[0126] It should be noted that the sequence numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0128] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A method for training a construction management decision-making model based on reinforcement learning, characterized in that, include: Obtain the initial construction parameters of the target construction project and the historical environmental parameters of the region where the target construction project is located; wherein, the target construction project is a construction project for which worker hours and material purchase quantities are to be planned; A construction project simulator is constructed based on the initial construction parameters and the historical environmental parameters. The state parameters of construction day T are input into a preset observation function to simulate the observation parameters of construction day T. The state parameters include construction status and environmental status. The construction status includes: remaining construction funds, remaining construction resources, worker efficiency, and construction progress. The environmental status includes: meteorological indicators and material prices in the area where the target construction project is located. The observation parameters include: current construction fund status, current construction resource status, current construction progress status, current environmental status, historical meteorological and working hour status, and future environmental status. The observation parameters of construction day T are input into a pre-defined decision model to simulate the action parameters for construction day T. These action parameters include the planned worker hours and material purchase quantities. The decision model is a neural network model. It determines the expected value and variance of each worker hour and each material purchase quantity. A normal distribution for each worker hour is constructed based on its expected value and variance, and samples are taken from this normal distribution to obtain the worker hours for construction day T. Similarly, a normal distribution for each material purchase quantity is constructed based on its expected value and variance, and samples are taken from this normal distribution to obtain the material purchase quantity for construction day T. Input the status parameters and action parameters of construction day T into the construction project simulator to simulate the status parameters of construction day T+1. The state parameters of construction day T and construction day T+1 are input into a preset excitation function to simulate the excitation value of construction day T; the excitation value is used to evaluate the action parameters. The observation trajectory, the action trajectory, the state trajectory, and the excitation sequence are constructed based on the simulated observation parameters, action parameters, state parameters, and excitation values, respectively; wherein, the observation trajectory is... , The observation parameters are for day T of construction; the movement trajectory is... , The action parameters for day T of construction; the state trajectory is... , The state parameters are for day T of construction; the excitation sequence is... , The incentive value is for day T of construction; T is an integer greater than or equal to 1 and less than or equal to N, N is the total construction period of the target construction project and N is less than or equal to the maximum number of construction days specified in the construction contract of the target construction project; wherein, when T is 1, the pre-set status parameters for the first day of construction are obtained from the preset database; The decision model with a preset structure is optimized based on the observation trajectory, the action trajectory, the state trajectory, and the stimulus sequence, so that the optimized decision model can be used to plan worker hours and material purchase quantities.
2. The method according to claim 1, characterized in that, The step of inputting the observation parameters of construction day T into the decision model and outputting the action parameters of construction day T includes: Input the observation parameters of day T of construction into the standardization module of the decision model to obtain the standardized observation parameters; The standardized observation parameters are input into the deep neural network in the decision model to obtain the expected values of worker hours and material purchase quantity, and the variances of worker hours and material purchase quantity are obtained from the model parameters of the decision model. We construct normal distributions for worker hours and material purchase quantities based on the expected and variance of worker hours and the expected and variance of material purchase quantities, respectively. The normal distribution of worker hours and the normal distribution of material purchase quantity are sampled by the sampling module in the decision model to obtain the standardized worker hours and standardized material purchase quantity respectively. The standardized worker hours and standardized material purchase quantities are input into the inverse standardization module of the decision model to obtain the action parameters for day T of construction.
3. The method according to claim 1, characterized in that, The step of inputting the state parameters and action parameters of construction day T into the construction project simulator to simulate the state parameters of construction day T+1 includes: The construction project simulator uses an environment model to construct a stochastic process based on historical environmental parameters. During the simulation, the environmental parameters of each day of construction are independently sampled according to the constructed stochastic process to simulate the environmental state of each day of construction. The project model of the construction project simulator simulates the construction status on day T+1 based on the status parameters and action parameters on day T of construction.
4. The method according to claim 3, characterized in that, The process of constructing a stochastic process based on historical environmental parameters using the environmental model of the construction project simulator, and independently sampling daily environmental parameters during the simulation based on the constructed stochastic process to simulate the daily environmental state of the construction, includes: The environmental model is used to construct a stochastic process based on historical meteorological indices from the historical environmental parameters. During the simulation, the meteorological indices for each day of construction are independently sampled based on the constructed stochastic process to simulate the meteorological indices for each day of construction. The environmental model is used to construct a stochastic process based on historical material prices in the historical environmental parameters. During the simulation, the material prices of each construction day are sampled according to the constructed stochastic process to simulate the environmental conditions and material prices of each construction day. The environmental model is used to determine the daily environmental conditions of construction based on the simulated daily weather index and material prices.
5. The method according to claim 3, characterized in that, The process of simulating the construction state on day T+1 using the project model in the construction project simulator based on the state parameters and action parameters of day T includes: The remaining construction funds for construction day T+1 are calculated using the project model based on the remaining construction funds in the construction status on day T, the material prices in the environmental status, the worker hours and material purchase quantity in the action parameters, and the hourly wage of workers stipulated in the construction contract. The project model calculates the worker efficiency on day T of construction based on the meteorological index in the environmental conditions on day T of construction and the worker hours on day T before construction. The project model calculates the construction progress increment for day T based on the remaining construction resources and worker efficiency in the construction status on day T, as well as the worker hours in the action parameters. The sum of the construction progress on day T and the construction progress increment is then calculated as the construction progress for day T+1. The project model calculates the construction resource consumption based on the progress increment on day T of construction, and calculates the remaining construction resources on day T+1 of construction based on the remaining construction resources in the construction status on day T, the material purchase quantity in the action parameters, and the construction resource consumption. The project model uses the worker efficiency on day T, the remaining construction funds, construction progress, and remaining construction resources on day T+1 as the construction status on day T+1.
6. The method according to claim 1, characterized in that, The excitation function is: in, The incentive value is the value on day T of construction. This represents the incremental construction progress on day T. The net income on day T of construction. The level of worker safety on day T of construction. , and They are respectively , and weight values and , and The sum is 1.
7. The method according to claim 1, characterized in that, The construction project simulator includes a project model and an environment model. The project model includes a funding module, a resource module, an efficiency module, and a schedule module. The environment model includes an environment module. The funding module determines remaining construction funds, the resource module determines remaining construction resources, the efficiency module determines worker efficiency, the schedule module determines construction progress, and the environment module determines weather indicators and material prices. The interaction between these modules is controlled through the project model and the environment model to determine the status parameters for the next day's construction based on the status parameters of the target construction project on the current construction day. The target construction project is a project for which worker hours and material purchase quantities need to be planned. The status parameters include remaining construction funds, remaining construction resources, worker efficiency, construction progress, weather indicators, and material prices.
8. A method for planning worker hours and material purchase quantities based on reinforcement learning, characterized in that, include: Collect today's observation parameters from construction projects that require planned worker hours and material purchases; wherein, today's observation parameters include current construction funding status, current construction resource status, current construction progress status, current environmental status, and predicted values for future environmental status; The observed parameters for today are input into a decision model that can be used to plan worker hours and material purchases so that the decision model can plan today's worker hours and material purchases; wherein the decision model is obtained by the method described in any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it is used to implement the method of any one of claims 1 to 7 or the method of claim 8.