A method, device and medium for concrete multi-objective proportioning optimization based on reinforcement learning

Abstract

The present application relates to the technical field of concrete multi-objective proportioning design, and discloses a concrete multi-objective proportioning optimization method, device and medium based on reinforcement learning, the optimization method comprising candidate gene screening on a data set based on elastic network regression; a prediction model is obtained based on reinforcement learning optimization; the prediction model outputs a concrete multi-objective proportioning. The present application constructs a concrete original data set containing raw material composition, microstructure characteristics and typical performance indicators, realizes the coexistence of material composition, microstructure and typical performance, screens key genes through an elastic network sparse modeling mechanism, constructs an initial concrete multi-objective proportioning prediction model, and embodies the nonlinear mapping and coupling principle between multiple performance indicators; the feature contribution degree in the prediction model is used to construct a concrete material knowledge graph, and the contribution degree is used to adjust the strategy based on the contribution degree, so as to realize the performance-driven optimization closed loop of data, model and strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of concrete multi-objective proportioning design, and particularly relates to a concrete multi-objective proportioning optimization method based on reinforcement learning, a device and a medium. BACKGROUND

[0002] In concrete material engineering, the mix proportion design needs to comprehensively consider performance objectives such as strength, durability and cost. Existing methods mainly include traditional empirical methods and data-driven optimization methods, and although a relatively mature technical system has been formed, there are certain technical bottlenecks in the design of high-performance concrete and multi-objective trade-off, and it is difficult to meet the urgent needs of intelligent and correct proportioning objectives.

[0003] Traditional methods are based on specification standards and empirical formulas, such as the absolute volume method in ACI 211.1-91, which has the advantages of simple operation and strong applicability. However, such methods highly depend on static parameters and empirical charts, and are difficult to dynamically adapt to material performance fluctuations and changes in site construction, and lack the ability to model the coupling relationship between performance. When faced with multi-objective design tasks, traditional methods are difficult to balance each performance indicator, and the target performance adjustment process mainly relies on manual implementation, which is costly and inefficient.

[0004] Data-driven methods model the nonlinear relationship between raw material proportioning and performance through machine learning techniques, improving the prediction accuracy and parameter optimization ability of the model. Although existing machine learning methods have achieved good results, they generally rely on a large amount of high-quality data, and the final results are easily affected by the training samples. In addition, most models use a black box mechanism, lack transparency, and are difficult for people to understand. The training and optimization process is mostly offline, lacking real-time adaptive optimization ability, and is difficult to flexibly respond to current environmental and target changes.

[0005] In summary, although traditional methods have stable advantages in standardization and practicality, they have deficiencies in multi-objective coordination and complex environment adaptation. While data-driven methods have improved the level of intelligence, they are limited by data quality, model transparency and system adaptability.

[0006] In summary, there is an urgent need for a concrete multi-objective proportioning optimization method that takes into account performance integration, environmental adaptation and real-time optimization to solve the problems in the prior art. SUMMARY

[0007] The present application aims to provide a concrete multi-objective proportioning optimization method based on reinforcement learning, and the specific technical solutions are as follows:

[0008] A concrete multi-objective proportioning optimization method based on reinforcement learning, comprising the following steps:

[0009] Step S100: concrete dataset construction and data preprocessing, obtaining a complete representation of the concrete original dataset including input samples;

[0010] Step S200: candidate gene screening based on elastic network regression on the concrete original dataset obtained in step S100, obtaining a concrete optimized dataset;

[0011] Step S300: selecting an initial concrete performance prediction submodel, training the initial concrete performance prediction submodel based on the concrete optimized dataset obtained in step S200, obtaining an initial concrete multi-objective proportion prediction model;

[0012] Step S400: optimizing the initial concrete multi-objective proportion prediction model based on reinforcement learning, generating a concrete proportion strategy that meets engineering constraints and achieves balance under multiple objectives, obtaining a final concrete multi-objective proportion prediction model;

[0013] Step S500: predicting based on the final concrete multi-objective proportion prediction model obtained in step S400, outputting the concrete multi-objective proportion.

[0014] Preferably, step S100 comprises the following steps:

[0015] Step S101, constructing a multi-source and multi-scale concrete material gene dataset, including: ①, concrete basic feature genes; ②, coarse and fine aggregate structure genes; ③, concrete microstructure genes; ④, typical performance genes of concrete;

[0016] Step S102, using a standard deviation-based statistical method to identify and process outliers in the numerical type features in the concrete material gene dataset;

[0017] Step S103, using the maximum-minimum normalization method to normalize all numerical type features in the concrete material gene dataset;

[0018] Step S104, performing one-hot encoding on the type features in the concrete material gene dataset; concatenating all one-hot vectors after encoding to the original feature vector to form a complete representation of the concrete original dataset including input samples.

[0019] Preferably, the concrete basic feature genes include cement, water, water-binder ratio / water-cement ratio, admixture, fly ash, slag, ultra-fine slag, admixture, silica fume, limestone, and oxides; the oxides include at least one of CaO, SiO2, Al2O3, MgO, and Fe2O3;

[0020] The coarse aggregate structure genes include coarse aggregate particle size, natural sand, and artificial sand; the coarse aggregate particle size includes at least one of 5-10 mm, 5-12.5 mm, 5-16 mm, 5-20 mm, 5-30 mm, 10-20 mm, and 20-30 mm of gravel;

[0021] The concrete microstructure genes include porosity, diffusion coefficient, pore structure, water reducing agent, and air entraining agent.

[0022] The typical performance genes of the concrete include impermeability, compressive strength, and carbonation resistance.

[0023] Preferably, the step S200 includes the following steps:

[0024] The step S201 defines the following objective function based on the elastic net regression model combining L1 regularization and L2 regularization:

[0025] ;

[0026] Wherein: is a feature matrix of a concrete original data set, is the number of concrete samples in the concrete material gene data set, is the number of feature genes in the concrete material gene data set; is the three target performance indicators of electric flux, 28d compressive strength, and carbonation depth corresponding to the impermeability, compressive strength, and carbonation resistance of the concrete; is a regression coefficient vector, representing the linear contribution of each feature to the performance indicator; is an intercept term; is a regularization strength hyperparameter, controlling the magnitude of the overall penalty term; is a regularization proportion coefficient, controlling the mixing proportion of L1 regularization and L2 regularization; represents a minimization operation; the operation represents the L1 norm of the vector; the operation represents the square of the L2 norm of the vector;

[0027] The step S202 trains the concrete original data set obtained in the step S100 to obtain a trained elastic net regression model; and performs candidate gene screening based on the trained elastic net regression model to obtain a concrete optimized data set.

[0028] Preferably, the candidate genes include CaO, SiO2, Al2O3, MgO, Fe2O3, cement, admixture, water, water-binder ratio, water reducing agent, air entraining agent, natural sand, artificial sand, 5-10 mm gravel, 10-20 mm gravel, 20-30 mm gravel, porosity, and diffusion coefficient.

[0029] Preferably, step S300 includes the following steps:

[0030] Step S301: Establish prediction sub-models for the initial concrete performance using the random forest regression method, defining the following multi-objective function. :

[0031] ;

[0032] ;

[0033] in: Input the proportions. , including filtered Numerical features and One categorical variable; These represent the predicted outputs for carbonization depth, 28-day compressive strength, and electrical flux, respectively. Indicates that for the first A random forest regression model is constructed based on several target performance parameters. Indicates the first The regression tree in the 1st Fitting function on each target The total number of trees;

[0034] Step S302: Based on the concrete optimization dataset obtained in step S200, train the prediction sub-model of the initial concrete performance to obtain the initial concrete multi-objective mix proportion prediction model.

[0035] Preferably, step S400 includes the following steps:

[0036] Step S401: Formalize the concrete mix design problem into a Markov decision process;

[0037] Step S402: Design the policy network and value function network;

[0038] Step S403: Construct the policy loss function; obtain the value function loss; obtain the final multi-objective loss function;

[0039] Step S404: Generate a concrete mix design strategy that satisfies engineering constraints and achieves balance under multiple objectives, and obtain the final concrete multi-objective mix design prediction model.

[0040] Preferably, the policy loss function in step S403 as follows:

[0041] ;

[0042] in: It is a time step Expectations; is a policy ratio function; is a advantage function; is a clipping threshold; is an operation for limiting a policy step;

[0043] value function loss as follows:

[0044] ;

[0045] wherein: is a value function network; is a multi-objective weighted loss function;

[0046] final multi-objective loss function as follows:

[0047] .

[0048] The technical scheme of the present application has the following beneficial effects:

[0049] The concrete multi-objective proportioning optimization method based on reinforcement learning comprises: concrete data set construction and data preprocessing, to obtain a complete representation of the input sample of a concrete original data set; candidate gene screening of the concrete original data set based on elastic network regression, to obtain a concrete optimization data set; selection of an initial concrete performance prediction submodel, training of the initial concrete performance prediction submodel based on the concrete optimization data set, to obtain an initial concrete multi-objective proportioning prediction model; optimization of the initial concrete multi-objective proportioning prediction model based on reinforcement learning, to generate a concrete proportioning strategy that satisfies engineering constraints and achieves balance under multiple objectives, and to obtain a final concrete multi-objective proportioning prediction model; and prediction based on the final concrete multi-objective proportioning prediction model, to output the concrete multi-objective proportioning. In combination with the characteristics of concrete as a multiphase composite material, a structured material gene data set (the concrete original data set including a complete representation of the input sample) containing raw material composition, microstructure characteristics (such as porosity, fly ash particle size distribution, etc.) and typical performance indicators is constructed, to realize data modeling based on the coexistence of material composition, microstructure and typical performance; in the concrete original high-dimensional characteristics, low-contribution variables are eliminated and key genes are screened through the elastic network sparse modeling mechanism, and a high-precision prediction model (i.e., the initial concrete multi-objective proportioning prediction model) is constructed to predict multiple objective performances, to reflect the nonlinear mapping and coupling principle between multiple performance indicators; the feature contribution degree in the initial concrete multi-objective proportioning prediction model is used to construct a concrete material knowledge graph, and the contribution degree is used to affect strategy adjustment (such as increasing the sampling density of high-contribution variables and reducing the action space of low-contribution variables), to obtain the final concrete multi-objective proportioning prediction model, and to realize performance-driven optimization of the trinity of data, model and strategy.

[0050] In addition, the application further discloses a computer device, which comprises a memory and a processor.

[0051] The memory is used for storing a computer program which can run on the processor.

[0052] The processor is used for implementing the steps of the concrete multi-objective proportioning optimization method based on reinforcement learning when the computer program is executed.

[0053] In addition, the application further discloses a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is used for implementing the steps of the concrete multi-objective proportioning optimization method based on reinforcement learning when the computer program is executed by a processor.

[0054] In addition to the purposes, features and advantages described above, the application has other purposes, features and advantages. The application will be further described below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to make the technical scheme of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the description are only some embodiments of the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative work on the basis of the accompanying drawings also belong to the protection scope of the present application.

[0056] Figure 1 is a step flow chart of the concrete multi-objective proportioning optimization method based on reinforcement learning in the preferred embodiment of the present application;

[0057] Figure 2 is a schematic diagram of a concrete material gene data set structure in the present application;

[0058] Figure 3 is a principle diagram of the concrete multi-objective proportioning optimization method based on reinforcement learning. DETAILED DESCRIPTION

[0059] In order to make the technical scheme of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the description are only some embodiments of the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative work on the basis of the accompanying drawings also belong to the protection scope of the present application.

[0060] Embodiment:

[0061] The embodiment provides a concrete multi-objective proportioning optimization method based on reinforcement learning, as shown in Figure 1 and Figure 3 , specifically comprising the following steps: collecting concrete original data, completing outlier elimination, feature normalization and one-hot encoding, and constructing an original data set D ; performing elastic network regression on high-dimensional features to obtain sparse feature coefficients and eliminate redundant / low-correlation variables to form an optimized data set retaining only key material genes; constructing a multi-objective prediction model to respectively predict carbonation depth, compressive strength and chloride ion permeability, uniformly packaging as a performance feedback function and constructing a material knowledge graph; designing and training a strategy network and a value function network, jointly weighting rewards to learn a collaborative optimal strategy; using the trained strategy to generate a high-quality proportioning solution set satisfying constraints from an arbitrary initial state, and performing performance evaluation and screening through the prediction model.

[0062] The specific steps of the concrete multi-objective proportioning optimization method provided in the embodiment include: step S100, concrete dataset construction and data preprocessing, to obtain a complete representation of the input sample; step S200, candidate gene screening based on elastic network regression on the concrete original dataset obtained in step S100, to obtain a concrete optimization dataset; step S300, selecting an initial concrete performance prediction submodel, training the initial concrete performance prediction submodel based on the concrete optimization dataset obtained in step S200, to obtain an initial concrete multi-objective proportioning prediction model; step S400, optimizing the initial concrete multi-objective proportioning prediction model based on reinforcement learning, to generate a concrete proportioning strategy that satisfies engineering constraints and achieves balance under multi-objective, and to obtain a final concrete multi-objective proportioning prediction model; and step S500, prediction based on the final concrete multi-objective proportioning prediction model obtained in step S400, to output a concrete multi-objective proportioning.

[0063] In the embodiment, preferably, before the proportioning optimization modeling, a concrete dataset with engineering pertinence and holographic feature expression capability is constructed, and systematic data preprocessing is performed. Step S100 includes the following steps:

[0064] Step S101, constructing a multi-source and multi-scale concrete material gene dataset. Considering that concrete is a high-dimensional, multi-objective coupled complex composite material, its performance is influenced by multiple factors, and has the following significant characteristics: high-dimensional proportioning space: concrete proportioning involves multiple variables, the composition space is complex, and the variables significantly influence each other; multi-scale structural characteristics: the performance is not only influenced by the ingredients, but also depends on the microstructure state, and structural characterization means (such as pore scanning image technology) need to be combined for modeling and characterization; performance coupling and conflict: there is a significant nonlinear relationship and synergy / conflict among three typical performance indicators, i.e., compressive strength, impermeability, and carbonation resistance, and it is difficult to achieve comprehensive optimization through traditional single-objective methods. Based on the material genome concept, the key composition and structural characteristics of concrete are abstracted as structured “genes”, and details are shown in Figure 2; Specifically, the concrete basic characteristic genes include cement, water, water-binder ratio / water-cement ratio, admixture, fly ash, slag, ultra-fine slag, admixture, silica fume, limestone, and oxides; the oxides include at least one of CaO, SiO2, Al2O3, MgO, and Fe2O3; the coarse-fine aggregate structure genes include coarse aggregate particle size, natural sand, and artificial sand; the coarse aggregate particle size includes at least one of 5-10 mm, 5-12.5 mm, 5-16 mm, 5-20 mm, 5-30 mm, 10-20 mm, and 20-30 mm of gravel; the concrete microstructure genes include porosity, diffusion coefficient, pore structure, water reducing agent, and air entraining agent; and the concrete typical performance genes include impermeability, compressive strength, and carbonation resistance. Finally, a concrete sample set including , a concrete original data set including genes is obtained D , each record corresponding to an actual concrete mixing proportion scheme and a corresponding performance index:

[0065] ;

[0066] wherein: represents raw materials of the th concrete sample, composed of numerical features and categorical features ; is a typical performance of the concrete.

[0067] In step S102, a numerical feature in the concrete material gene data set is subjected to abnormal value identification and processing by using a standard deviation-based statistical method. Specifically, the numerical feature is subjected to abnormal value identification and processing by using a standard deviation-based statistical method, and for the th numerical feature , a sample mean and a standard deviation are calculated, and an upper boundary and a lower boundary of the abnormal value are:

[0068] ;

[0069] ;

[0070] wherein: The tolerance multiple is usually 2, which corresponds to about 95% of the normal distribution coverage range. According to the upper and lower bounds, the abnormal values in the data are removed to ensure that the data distribution is reasonable and the extreme points do not interfere with the subsequent model training, thereby improving the stability and generalization ability of the model fitting.

[0071] In step S103, all numerical features in the concrete material gene data set are normalized by using the maximum-minimum normalization method, that is, dimensionless normalization. In order to eliminate the influence of different dimensions on model training, all numerical features are normalized to the interval [0, 1] by using the maximum-minimum normalization method:

[0072] ;

[0073] Wherein: is the th numerical feature of the th concrete sample; is the normalized value of ; , are the maximum and minimum values of the th numerical feature, respectively. The normalized data facilitates model convergence and improves training efficiency.

[0074] In step S104, the categorical features in the concrete material gene data set are one-hot encoded; all the encoded one-hot vectors are spliced into the original feature vector to form a concrete original data set including the complete representation of the input sample. Specifically:

[0075] For categorical features, they are converted into structured 0 / 1 vector representation. For the th categorical feature , there are possible values , and the one-hot encoding form is:

[0076] ;

[0077] Wherein: is the th categorical feature of the th concrete sample;

[0078] Only the class position corresponding to the current sample is 1, and the rest are 0. All the encoded one-hot vectors will be spliced into the original feature vector to form the complete representation of the final input sample .

[0079] Since there are many high-dimensional features in the concrete data set, in order to further identify the key features that contribute to the multi-objective performance of the concrete, the application constructs a candidate gene screening mechanism based on elastic net regression. The method has the ability of sparse modeling, can effectively control overfitting and eliminate redundant or low correlation features while preserving the performance prediction ability, and provides high-quality input variables for subsequent modeling and optimization. The preferred step S200 in the embodiment includes the following steps:

[0080] Step S201, based on the elastic net regression model combining L1 regularization and L2 regularization, define the objective function:

[0081] ;

[0082] Wherein: is the feature matrix of the original concrete data set, is the number of concrete samples in the concrete material gene data set, is the number of feature genes in the concrete material gene data set; is the carbonation depth, compressive strength and chloride ion permeability of the concrete material; is the regression coefficient vector, indicating the linear contribution of each feature to the performance index; is the intercept term; is the regularization strength hyperparameter, which controls the amplitude of the overall penalty term; is the regularization proportion coefficient, which controls the mixing proportion of L1 regularization and L2 regularization. denotes the minimization operation; the operation represents the L1 norm of the vector, that is, the sum of the absolute values of each element; the operation represents the L2 norm square of the vector, that is, the sum of the squares of each element.

[0083] The elastic net regression model combines the advantages of Lasso (L1 regularization) and Ridge (L2 regularization), and is suitable for engineering scenarios with high variable dimension and multicollinearity. Based on the preprocessed concrete data set , the proportion of the training set, the test set and the validation set for training the elastic net regression model is set to 4:3:3. The model takes the raw material features of the concrete sample as input, and realizes the sparse screening of concrete features through the regularization mechanism.

[0084] Step S202, based on the obtained concrete original data set in step S100, training is performed to obtain an elastic network regression model after training; based on the trained elastic network regression model, candidate gene screening is performed to obtain a concrete optimization data set. Specifically, after the elastic network regression model is trained, the final output is a set of sparse feature coefficient vectors, and the mathematical expression is:

[0085] ;

[0086] For each feature in the concrete data set, the regression coefficient represents the linear regression coefficient of the feature on the target performance. If , it means that the feature has no significant effect on the concrete performance index, and is determined as a redundant or weakly correlated feature, which is removed from the data set; if , it means that the feature has a significant effect on the concrete performance, and is defined as a “candidate gene” (i.e., a key input feature). After applying the above screening logic to the concrete original data set , all key variables (i.e., candidate genes) that satisfy are retained, and redundant or weakly correlated variables are removed, and finally the concrete optimization data set is obtained, as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] Among them: represents the key input features of the th sample retained by the elastic network screening, including numerical features and categorical features; represents the target performance index of the sample, corresponding to the carbonation depth , 28d compressive strength , and electric flux .

[0091] The candidate genes obtained by the elastic network screening of the concrete composition features in the specific experiment include: CaO, SiO2, Al2O3, MgO, Fe2O3, cement, admixture, water, water-binder ratio, water-reducing agent, air entraining agent, natural sand, artificial sand, 5-10mm crushed stone, 10-20mm crushed stone, 20-30mm crushed stone, porosity, and diffusion coefficient gene. Therefore, the above components are used as candidate genes (input variables) for subsequent experimental modeling.

[0092] In the preferred embodiment, step S300 is a multi-objective prediction model modeling based on coupling performance driving and construction of a concrete knowledge graph. To accurately represent the multi-performance coupling characteristics and the conflict relationship between the objectives existing in the optimization process of the concrete material, a multi-objective performance prediction model with concrete material perception ability is constructed in this embodiment. Combined with the nonlinear correlation characteristics between the concrete structure and performance, the mapping relationship between the mixing ratio parameters and the key performance indicators can be effectively established to provide high-quality environmental feedback for reinforcement learning strategy optimization. Based on the constructed concrete optimization dataset , the proportions of the training set, the test set and the validation set for training the model are set to 4:3:3. The input features are the screened concrete candidate genes, which not only represent the material composition parameters, but also reflect the influence path of the microstructure on the macro performance. Specifically, the following steps are included:

[0093] Step S301, a random forest regression method is used to establish a prediction sub-model of the initial concrete performance, and the following multi-objective function is defined: :

[0094] ;

[0095] ;

[0096] Wherein: is the mixing ratio input, , including numerical value features and classification variables; represent the prediction outputs of the carbonation depth, 28d compressive strength and electric flux, respectively; is a random forest regression model constructed for the th objective performance, is the fitting function of the th regression tree on the th objective, is the total number of trees.

[0097] During the training process, the mean square error is used as the objective function for optimization.

[0098] To reflect the performance of the concrete in mechanics and durability, the output contains three key indicators: ① : carbonation depth (mm), representing the carbonation resistance of concrete, and the optimization goal is to minimize; ② : 28-day compressive strength (MPa), representing the bearing capacity, which is converted to a negative value to conform to the minimization direction; ③ : electric flux (C), representing the resistance to chloride ion penetration, and the optimization goal is to minimize.

[0099] Step S302, based on the concrete optimization dataset obtained in step S200, the prediction model of the initial concrete performance is trained to obtain the initial concrete multi-objective proportioning prediction model. As a performance evaluation interface in the reinforcement learning model, the support strategy network can quickly simulate and feedback the target performance of the concrete after each round of proportioning generation, forming a closed-loop optimization mechanism.

[0100] Meanwhile, the dataset is optimized according to the contribution of the concrete component characteristics output by the prediction model to the target performance. For example, water, cement, oxides (CaO, SiO2, Al2O3, MgO, Fe2O3), admixtures, and porosity have high contribution to the target performance, while the remaining characteristics have relatively low contribution to the target performance. Specifically, the data density in the proportioning area represented by the above high-contribution characteristics is increased, and a concrete material knowledge graph is constructed. The high-contribution characteristics are selected as key nodes of the graph, and the relationship and weight of the graph edges are defined according to the influence direction and strength of the variables on the performance, and a concrete material knowledge graph is constructed with (material composition / microstructure, relationship, performance index) as the basic triplets, and finally a concrete material knowledge graph is obtained, which is used to guide the optimization of data and model in reverse. For the remaining low-contribution input characteristics, according to their physical correlation with the performance of the concrete, their adjustment frequency is reduced in subsequent strategy optimization, thereby improving efficiency and avoiding focusing on irrelevant variables.

[0101] In this embodiment, to achieve the collaborative optimization of concrete proportioning among multiple performance targets, the present application proposes a multi-objective optimization strategy based on reinforcement learning, which constructs an intelligent decision-making model to learn to achieve performance balance optimization in a high-dimensional continuous space. Step S400 is a multi-objective optimization strategy based on reinforcement learning, which includes the following steps:

[0102] Step S401, formalize the concrete proportioning optimization problem as a Markov decision process;

[0103] Step S402, design a strategy network and a value function network;

[0104] Step S403, construct a strategy loss function; obtain a value function loss; obtain a final multi-objective loss function;

[0105] Step S404, generate a concrete proportioning strategy that satisfies the engineering constraints and achieves balance under multiple objectives, and obtain a final concrete multi-objective proportioning prediction model.

[0106] The specific measures of this embodiment are as follows:

[0107] The concrete proportioning optimization problem is essentially a continuous decision-making problem under multiple objectives, multiple variables, and high constraints, which is formalized as a Markov decision process As follows:

[0108] State space : represents the current state of concrete mix proportioning.

[0109] Action space : In continuous action space, action vector represents the adjustment amount of numerical features of the current concrete; while the categorical features are not adjusted by action, and in actual engineering, the material category is determined and usually fixed in the short term.

[0110] State transition function : Since concrete proportioning is deterministic, the state transition function is simplified as , which limits each variable to the physical / norm feasible region through the clip operation, ensuring that the search process of RL strictly follows the engineering implementability of concrete materials. is the minimum value of numerical features, ; is the maximum value of numerical features, .

[0111] Reward function : Based on the prediction results of the trained multi-objective prediction model on target performance, a multi-objective weighted loss function is constructed, that is, the better the performance, the greater the reward, where: the weight coefficient reflects the constraints on the preference for concrete performance under different engineering scenarios.

[0112] Design policy network and value function network.

[0113] Policy network . The policy network outputs the action distribution in the continuous action space, which is set as a multi-dimensional Gaussian distribution, and each dimension is independent of each other:

[0114] ;

[0115] where: is a multi-dimensional normal distribution, and the policy network outputs the action mean and variance , matching the continuous adjustability of concrete proportioning and the material properties of different variable uncertainty differences.

[0116] After that, the action can be clipped to fall into the physical feasible region, ensuring that the generated proportioning meets the engineering specifications.

[0117] Value function network . The expected discounted cumulative return of the agent in state following the policy where the discount factor For regulating the importance of target performance, suitable for concrete materials with much longer life span than training step; represents the immediate reward of the step from the current time step.

[0118] Step S403 is specifically as follows:

[0119] Constructing the strategy loss function As follows:

[0120] ;

[0121] Wherein: is the strategy ratio function, which measures the relative preference degree of new and old strategies for the same action; the advantage function reflects the relative goodness of taking action compared to the average behavior; the clipping threshold and The operation is used to limit the strategy step length, so as to avoid extreme step length update of the strategy in the concrete performance space with significant target conflict.

[0122] The training target of the value function network is to minimize the error between the predicted value and the instantaneous value of the reward function , so the value function loss is as follows:

[0123] .

[0124] The final multi-objective loss function is as follows:

[0125] .

[0126] The model training can be optimized by cross-validation or Bayesian optimization. The joint training of the loss function is updated on each mini-batch sample, and the Adam optimizer is used to ensure the stability and convergence speed of the training.

[0127] The above is the complete multi-objective optimization model training process based on reinforcement learning.

[0128] (1) Initialize the strategy network , the value function network and the experience buffer , and set the reinforcement learning hyperparameters, such as learning rate, discount factor, entropy coefficient, clipping range , etc.

[0129] (2) Sample action from the current state , execute action get the next state and calculate three performance indicators using a multi-objective prediction model, and calculate the immediate reward based on a weighted reward function ; store the samples into the buffer pool , and update the current state.

[0130] (3) For each target , maintain the prediction value of the last steps, normalize it to a new target weight according to the variance , and automatically focus on material performance targets with large fluctuations (such as durability indicators, which are often less stable in the early stages), achieving dynamic adaptive control.

[0131] (4) Randomly extract a small batch of samples from the experience pool, calculate the advantage function to update the policy loss ; at the same time, minimize the mean square error between the prediction value and the instantaneous reward to update the value function loss .

[0132] (5) Construct a joint loss function , use the Adam optimizer to perform backpropagation and parameter update on the policy network and value function network respectively;

[0133] (6) Repeat steps (2) to (5) until the policy converges, and finally obtain a concrete mixing strategy that satisfies the engineering constraints and achieves a balance under multiple objectives .

[0134] After completing the reinforcement learning model training, the agent can generate a mixing scheme that meets the multi-performance coordination requirements within the actual feasible region of the concrete material.

[0135] Specifically, let the initial state be , call the policy network to generate action , update the state to a new concrete mixing scheme , and evaluate its three key performances through the multi-objective prediction model , and the comprehensive performance of the mixing solution can be calculated.

[0136] Collect 5774 data for three typical performances, after preprocessing such as missing value filling, outlier removal, feature normalization and one-hot encoding, there are 1315 data left, and the original data set with material semantics is formed accordingly.

[0137] Subsequently, the elastic network is used to sparsely screen the concrete component features, to obtain candidate genes highly correlated with the compressive strength, impermeability and chloride ion permeability: CaO, SiO2, Al2O3, MgO, Fe2O3, cement, admixture, water, water-binder ratio (w / b), water-reducing agent content, air entraining agent, natural sand, artificial sand, gravel 5-10 mm, gravel 10-20 mm, gravel 20-30 mm, porosity, diffusion coefficient; then, the random forest regression model of the three typical performances is trained based on this, as the performance feedback function in the reinforcement learning optimization stage.

[0138] Objective function (minimization framework, measured by carbonation depth, 28d compressive strength and electric flux of three performances): carbonation depth (mm), the smaller the better; 28d compressive strength (MPa), the greater the strength, the better; electric flux (C), the smaller the better (i.e. the stronger the chloride ion permeability), the schemes (RL1 and RL2) of the embodiment and the prior art are shown in Table 1:

[0139] Table 1 Three performance index statistics of the scheme of the embodiment and the ant colony algorithm in the prior art

[0140]

[0141] Under the same data and constraints, the mixing ratio scheme (RL1, RL2) obtained based on the method of the embodiment is significantly improved in carbonation depth, 28d compressive strength and electric flux compared with the commonly used ant colony algorithm scheme. The mixing ratio scheme is shown in Table 2:

[0142] Table 2 Concrete formula obtained by the scheme of the embodiment and the ant colony algorithm in the prior art

[0143]

[0144] Based on the above, the concrete material multi-objective collaborative mixing ratio optimization method based on reinforcement learning of the embodiment realizes the following improvements compared with the traditional method:

[0145] 1. Concrete material gene data set construction method

[0146] Combined with the characteristics of concrete as a multiphase composite material, a structured material gene data set containing raw material composition, microstructure features (such as porosity, fly ash particle size distribution, etc.) and typical performance indicators is constructed, realizing the coexistence of material composition, microstructure and typical performance in data modeling, so that the optimization model has material understanding ability, and the prediction accuracy and material explanation are improved.

[0147] 2. Candidate gene screening mechanism and collaborative construction of prediction model

[0148] In the original high-dimensional characteristics of concrete, through the elastic network sparse modeling mechanism, low contribution variables are eliminated, key genes are screened, and a high-precision prediction model is constructed to predict multiple target performance, reflecting the nonlinear mapping and coupling principle between multiple performance indicators. The proportioning optimization problem is formalized as a Markov process, and a reinforcement learning algorithm is introduced to train the strategy network, so that the model has the ability of dynamic target balance and high-dimensional continuous action regulation, which is suitable for collaborative regulation under multiple performance conflicts, and is obviously superior to the fixed target weighting scheme of traditional algorithms.

[0149] 3. Feature contribution driven material knowledge graph construction and adaptive adjustment mechanism

[0150] The feature contribution in the performance prediction model is used to construct the concrete material knowledge graph, and the contribution is used to adjust the strategy (such as increasing the sampling density of high-contribution variables and reducing the action space of low-contribution variables), realizing the performance-driven optimization closed loop of data, model and strategy. By introducing feature contribution evaluation and material knowledge graph construction, a feedback closed loop from data to model to strategy is realized, which can dynamically focus on key features and improve data utilization efficiency.

[0151] The embodiment of the application further discloses a computer device comprising a memory and a processor;

[0152] The memory is used to store a computer program which can run on the processor;

[0153] The processor is used to execute the computer program to realize the steps of the legal knowledge intelligent question and answer method and the concrete multi-objective proportioning optimization method based on reinforcement learning.

[0154] Illustratively, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device.

[0155] The computer device can be a mobile phone, a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The computer device can include, but is not limited to, a processor, a memory. For example, the computer device can also include an input / output device, a network access device, a bus, etc.

[0156] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the computer device, and connects various parts of the computer device through various interfaces and lines.

[0157] The memory can be used to store the computer program and / or modules, and the processor realizes the computer program by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0158] The modules / units integrated in the computer device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0159] In addition, the embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the above-mentioned concrete multi-objective proportioning optimization method based on reinforcement learning.

[0160] Compared with the prior art, the embodiment method processes text and image respectively, processes multi-modal features, and extracts medical relations by combining visual features. The embodiment method makes up for the lack of entity information by combining external knowledge, reduces the influence of irrelevant information by layering and aligning and fusing multiple medical pictures, and improves the accuracy of relation extraction.

[0161] It should be noted that the apparatus embodiments described above are only schematic and that the units as described separately can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme. In addition, the apparatus embodiments provided by the present application in the drawings represent that they have communication connection between the modules. Specifically, it can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0162] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. The present application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for concrete multi-objective proportioning optimization based on reinforcement learning, characterized in that, Comprising the following steps: Step S100: concrete dataset construction and data preprocessing, obtaining a complete representation of the input sample of the concrete original dataset; Step S200: based on the elastic network regression, the candidate gene screening of the concrete original dataset obtained in step S100 is carried out, and a concrete optimization dataset is obtained; Step S300: selecting an initial concrete performance prediction submodel, training the initial concrete performance prediction submodel based on the concrete optimization dataset obtained in step S200, and obtaining an initial concrete multi-objective proportion prediction model; Step S400: based on reinforcement learning, the initial concrete multi-objective proportion prediction model is optimized, a concrete proportion strategy that meets the engineering constraints and achieves balance under multiple objectives is generated, and a final concrete multi-objective proportion prediction model is obtained; Step S500: based on the final concrete multi-objective proportion prediction model obtained in step S400, prediction is carried out, and the concrete multi-objective proportion is output; Step S100 comprises the following steps: Step S101, constructing a multi-source and multi-scale concrete material gene dataset, comprising: ①, concrete basic feature genes; ②, coarse and fine aggregate structure genes; ③, concrete microstructure genes; ④, concrete typical performance genes, including impermeability, compressive strength and carbonation resistance; Step S102, using a standard deviation-based statistical method to identify and process abnormal values in the numerical type features in the concrete material gene dataset; Step S103, using the maximum-minimum normalization method to normalize all numerical type features in the concrete material gene dataset; Step S104, encoding the type features in the concrete material gene dataset; all the encoded one-hot vectors are spliced into the original feature vector, and all the encoded one-hot vectors are spliced into the original feature vector, to form a complete representation of the input sample of the concrete original dataset; Step S200 comprises the following steps: Step S201, based on the elastic network regression model combining L1 regularization and L2 regularization, the objective function is defined as follows: ； wherein: is a feature matrix of the concrete original dataset, is the number of concrete samples in the concrete material gene dataset, is the number of feature genes in the concrete material gene dataset; is the three target performance indicators corresponding to the impermeability, compressive strength and carbonation resistance of the concrete, namely, electric flux, 28d compressive strength and carbonation depth; is a regression coefficient vector, representing the linear contribution of each feature to the performance indicators; is an intercept term; is a regularization strength hyperparameter, controlling the magnitude of the overall penalty term; is a regularization proportion coefficient, controlling the mixing proportion of L1 regularization and L2 regularization; represents a minimization operation; the operation represents the L1 norm of the vector; the operation represents the L2 norm square of the vector; Step S202, based on the concrete original dataset obtained in step S100, the trained elastic network regression model is obtained; based on the trained elastic network regression model, candidate gene screening is carried out, and a concrete optimization dataset is obtained.

2. The method of claim 1, wherein, The concrete basic feature genes include cement, water, water-binder ratio / water-cement ratio, admixture, fly ash, slag, ultra-fine slag, admixture, silica fume, limestone and oxide; the oxide includes at least one of CaO, SiO2, Al2O3, MgO and Fe2O3; The coarse and fine aggregate structure genes include coarse aggregate particle size, natural sand and artificial sand; the coarse aggregate particle size includes at least one of 5-10mm, 5-12.5mm, 5-16mm, 5-20mm, 5-30mm, 10-20mm and 20-30mm gravel; The concrete microstructure genes include porosity, diffusion coefficient, pore structure, water reducing agent and air entraining agent.

3. The method of claim 2, wherein, The candidate genes include CaO, SiO2, Al2O3, MgO, Fe2O3, cement, admixture, water, water-binder ratio, water reducing agent, air entraining agent, natural sand, artificial sand, 5-10mm gravel, 10-20mm gravel, 20-30mm gravel, porosity, and diffusion coefficient.

4. The method of claim 1, wherein, The step S300 comprises the following steps: Step S301, a random forest regression method is used to establish a prediction sub-model of the initial concrete performance, and the following multi-objective function is defined : ； ； wherein: is a ratio input, , comprising the filtered number of value features and categorical variables; respectively represent the predicted outputs for carbonation depth, 28d compressive strength, and electric flux; represents a random forest regression model constructed for the target performance, represents the fitting function of the th regression tree on the th target, is the total number of trees; The step S302 comprises the following steps:

5. The method of claim 4, wherein, The step S400 comprises the following steps: The step S401 comprises the following steps: The step S402 comprises the following steps: The step S403 comprises the following steps: The step S404 comprises the following steps:

6. The method of claim 5, wherein the method further comprises: Policy loss function in step S403 As follows: ； wherein: is a time step is a desired; is a policy ratio function; is a merit function; is a clipping threshold; operates to limit a policy step size; Value function loss As follows: ； wherein: is a value function network; is a multi-objective weighted loss function; Final multi-objective loss function As follows: 。 7. A computer device, characterized by The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the concrete multi-objective proportioning optimization method based on reinforcement learning in any one of claims 1 to 6. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the concrete multi-objective proportioning optimization method based on reinforcement learning in any one of claims 1 to 6. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the concrete multi-objective proportioning optimization method based on reinforcement learning in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the concrete multi-objective proportioning optimization method based on reinforcement learning in any one of claims 1 to 6. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the concrete multi-objective proportioning optimization method based on reinforcement learning in any one of claims 1 to 6. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the concrete multi-objective proportioning optimization method based on reinforcement learning in any one of claims 1 to 6.

Images