Concrete multi-target proportioning optimization method and equipment based on reinforcement learning and medium

Abstract

The invention relates to the technical field of concrete multi-target ratio design, and discloses a reinforcement learning-based concrete multi-target ratio optimization method and device and a medium, and the optimization method comprises the steps: carrying out candidate gene screening on a data set based on elastic network regression; obtaining a prediction model based on reinforcement learning optimization; and the prediction model outputs a concrete multi-target ratio. According to the method, a concrete original data set containing raw material composition, microstructure characteristics and typical performance indexes is constructed, and material composition, microstructure and typical performance coexist; key genes are screened through an elastic network sparse modeling mechanism, an initial concrete multi-target proportion prediction model is constructed, and the nonlinear mapping and coupling principle among multiple performance indexes is embodied; the feature contribution degree in the prediction model is used for constructing a concrete material knowledge graph, strategy adjustment is counteracted on the basis of the contribution degree, and a data, model and strategy three-in-one performance-driven optimization closed loop is achieved.

Description

Technical Field

[0001] This invention relates to the field of multi-objective mix design technology for concrete, specifically to a method, equipment, and medium for multi-objective mix optimization of concrete based on reinforcement learning. Background Technology

[0002] In concrete materials engineering, mix design must comprehensively consider performance objectives such as strength, durability, and cost. Existing methods are mainly divided into traditional empirical methods and data-driven optimization methods. Although these methods have formed relatively mature technical systems, they all have certain technical bottlenecks in complex design scenarios such as high-performance concrete and multi-objective trade-offs, making it difficult to meet the urgent need for intelligent and accurate mix design objectives.

[0003] Traditional methods, based on standards and empirical formulas such as the absolute volume method in ACI 211.1-91, offer advantages in terms of ease of operation and wide applicability. However, these methods heavily rely on static parameters and empirical charts, making it difficult to dynamically adapt to fluctuations in material properties and changes during on-site construction. They also lack the ability to model the coupling relationships between performance parameters. When faced with multi-objective design tasks, traditional methods struggle to balance various performance indicators, and the adjustment of target performance primarily relies on manual processes, resulting in high costs and low efficiency.

[0004] Data-driven approaches, on the other hand, use machine learning techniques to model the non-linear relationship between raw material ratios and performance, improving the model's prediction accuracy and parameter optimization capabilities. While existing machine learning methods have achieved good results, they generally rely on large amounts of high-quality data, and their final results are easily influenced by the training samples. Furthermore, most models employ a black-box mechanism, lacking transparency and interpretability, making them difficult for humans to understand effectively. Their training and optimization processes are mostly offline, lacking real-time adaptive optimization capabilities and struggling to flexibly respond to changes in the current environment and objectives.

[0005] In summary, while traditional methods have stable advantages in terms of standardization and practicality, they are insufficient in multi-objective coordination and adaptation to complex environments; while data-driven methods improve the level of intelligence, they are limited by data quality, model transparency and system adaptability.

[0006] In summary, there is an urgent need for a multi-objective concrete mix design optimization method that takes into account performance integration, environmental adaptability, and real-time optimization to solve the problems in the existing technology. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-objective concrete mix design optimization method based on reinforcement learning, and the specific technical solution is as follows: A reinforcement learning-based multi-objective mix design optimization method for concrete includes the following steps: Step S100: Concrete dataset construction and data preprocessing to obtain the original concrete dataset including a complete representation of the input samples; Step S200: Based on elastic network regression, candidate genes are screened in the original concrete dataset obtained in step S100 to obtain the optimized concrete dataset; Step S300: Select the prediction sub-model of initial concrete performance, and train the prediction sub-model of initial concrete performance based on the concrete optimization dataset obtained in step S200 to obtain the multi-objective mix proportion prediction model of initial concrete. Step S400: Optimize the initial concrete multi-objective mix proportion prediction model based on reinforcement learning, generate a concrete mix proportion strategy that satisfies engineering constraints and achieves balance under multiple objectives, and obtain the final concrete multi-objective mix proportion prediction model. Step S500: Based on the final concrete multi-objective mix design prediction model obtained in step S400, make predictions and output the concrete multi-objective mix design.

[0008] Preferably, step S100 includes the following steps: Step S101: Construct a multi-source, multi-scale concrete material gene dataset, including: ① basic concrete characteristic genes; ② coarse and fine aggregate structure genes; ③ concrete microstructure genes; ④ typical concrete performance genes. Step S102: Use a statistical method based on standard deviation to identify and process outliers in the numerical features of the concrete material gene dataset; Step S103: Normalize all numerical features in the concrete material gene dataset using the maximum-minimum normalization method; Step S104: Perform one-hot encoding on the categorized features in the concrete material gene dataset; concatenate all the encoded one-hot vectors into the original feature vector to form the original concrete dataset containing a complete representation of the input samples.

[0009] Preferably, the basic characteristic genes of concrete include cement, water, water-cement ratio / water-cement ratio, admixtures, fly ash, slag, ultrafine slag, additives, silica fume, limestone, and oxides; the oxides include at least one of CaO, SiO2, Al2O3, MgO, and Fe2O3; The coarse and fine aggregate structure includes coarse aggregate particle size, natural sand and artificial sand; coarse aggregate particle size includes at least one of crushed stone of 5-10mm, 5-12.5mm, 5-16mm, 5-20mm, 5-30mm, 10-20mm and 20-30mm. The microstructure of concrete includes porosity, diffusion coefficient, pore structure, water-reducing agent, and air-entraining agent. Typical performance characteristics of concrete include impermeability, compressive strength, and carbon resistance.

[0010] Preferably, step S200 includes the following steps: Step S201: Based on the elastic network regression model combining L1 and L2 regularization, the following objective function is defined: ; in: It is the feature matrix of the original concrete dataset. This represents the number of concrete samples in the concrete material gene dataset. The number of characteristic genes in the concrete material gene dataset; The three target performance indicators for concrete are electrical flux, 28-day compressive strength, and carbonation depth, corresponding to its impermeability, compressive strength, and carbonation resistance. This is the regression coefficient vector, representing the linear contribution of each feature to the performance index; It is the intercept term; It is a regularization strength hyperparameter that controls the magnitude of the overall penalty term; This is the regularization ratio coefficient, which controls the mixing ratio of L1 regularization and L2 regularization; This indicates a minimize operation; The operation represents the L1 norm of the vector; The operation represents the squared L2 norm of the vector; Step S202: Train the original concrete dataset obtained in step S100 to obtain a trained elastic network regression model; screen candidate genes based on the trained elastic network regression model to obtain a concrete optimization dataset.

[0011] Preferably, candidate genes include CaO, SiO2, Al2O3, MgO, Fe2O3, cement, admixtures, water, water-cement ratio, water-reducing agent, air-entraining agent, natural sand, artificial sand, 5-10mm crushed stone, 10-20mm crushed stone, 20-30mm crushed stone, porosity, and diffusion coefficient.

[0012] Preferably, step S300 includes the following steps: Step S301: Establish prediction sub-models for the initial concrete performance using the random forest regression method, defining the following multi-objective function. : ; ; in: Input the proportions. , including filtered Numerical features and One categorical variable; These represent the predicted outputs for carbonization depth, 28-day compressive strength, and electrical flux, respectively. Indicates that for the first A random forest regression model is constructed based on several target performance parameters. Indicates the first The regression tree in the first Fitting function on each target The total number of trees; Step S302: Based on the concrete optimization dataset obtained in step S200, train the prediction sub-model of the initial concrete performance to obtain the initial concrete multi-objective mix proportion prediction model.

[0013] Preferably, step S400 includes the following steps: Step S401: Formalize the concrete mix design problem into a Markov decision process; Step S402: Design the policy network and value function network; Step S403: Construct the policy loss function; obtain the value function loss; obtain the final multi-objective loss function; Step S404: Generate a concrete mix design strategy that satisfies engineering constraints and achieves balance under multiple objectives, and obtain the final concrete multi-objective mix design prediction model.

[0014] Preferably, the policy loss function in step S403 as follows: ; in: It is a time step Expectations; The strategy ratio function; The dominant function; The shear threshold; This operation is used to limit the policy step size; Value function loss as follows: ; in: Value function network; For multi-objective weighted loss functions; The final multi-objective loss function as follows: .

[0015] The application of the technical solution of the present invention has the following beneficial effects: The present invention provides a reinforcement learning-based method for multi-objective concrete mix design optimization, comprising: constructing and preprocessing a concrete dataset to obtain an original concrete dataset containing complete representations of input samples; screening candidate genes in the original concrete dataset using elastic network regression to obtain an optimized concrete dataset; selecting an initial concrete performance prediction sub-model and training it on the optimized concrete dataset to obtain an initial multi-objective concrete mix design prediction model; optimizing the initial multi-objective concrete mix design prediction model using reinforcement learning to generate a concrete mix design strategy that satisfies engineering constraints and achieves balance under multiple objectives, thus obtaining a final multi-objective concrete mix design prediction model; and performing prediction based on the final multi-objective concrete mix design prediction model to output the multi-objective concrete mix design. Combining the characteristics of concrete as a multiphase composite material, a structured material gene dataset (including a complete representation of the original concrete dataset of the input samples) is constructed, containing raw material composition, microstructural features (such as porosity and fly ash particle size distribution), and typical performance indicators. This establishes a data modeling foundation that integrates material composition, microstructure, and typical performance. In the original high-dimensional features of concrete, a sparse modeling mechanism using elastic networks is employed to eliminate low-contribution variables and select key genes. Simultaneously, a high-precision prediction model (i.e., an initial multi-objective mix proportion prediction model for concrete) is constructed to predict multi-objective performance, reflecting the nonlinear mapping and coupling principle between multiple performance indicators. The feature contribution values ​​from the initial multi-objective mix proportion prediction model are used to construct a concrete material knowledge graph. Based on these contribution values, a strategy adjustment is applied (e.g., increasing sampling density for high-contribution variables and reducing action space for low-contribution variables), resulting in the final multi-objective mix proportion prediction model for concrete. This achieves a performance-driven optimization closed loop integrating data, model, and strategy.

[0016] In addition, the present invention also discloses a computer device, including a memory and a processor; The memory is used to store computer programs that can run on the processor; The processor is used to implement the steps of the reinforcement learning-based multi-objective concrete mix optimization method described above when executing the computer program.

[0017] In addition, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described reinforcement learning-based multi-objective concrete mix design optimization method.

[0018] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the steps of the reinforcement learning-based multi-objective concrete mix design optimization method in a preferred embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the concrete material gene dataset in this invention; Figure 3 This is a schematic diagram of a multi-objective mix design optimization method for concrete based on reinforcement learning. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example: This embodiment provides a reinforcement learning-based multi-objective mix design optimization method for concrete, such as... Figure 1 and Figure 3 As shown, the specific steps include: collecting raw concrete data, performing outlier removal, feature normalization and one-hot encoding, and constructing the raw dataset. D Elastic network regression is performed on high-dimensional features to obtain sparse feature coefficients and remove redundant / low-correlation variables, forming an optimized dataset that retains only key material genes; a multi-objective prediction model is constructed to predict carbonization depth, compressive strength, and chloride ion penetration resistance, which are then uniformly encapsulated as performance feedback functions and a material knowledge graph is constructed; a policy network and a value function network are designed and trained, and a multi-objective weighted reward is combined to learn a collaborative optimal policy; the trained policy is used to generate a high-quality set of proportions that meet the constraints from any initial state, and the performance is evaluated and screened through the prediction model.

[0023] The concrete multi-objective mix design optimization method provided in this embodiment includes the following steps: Step S100: Concrete dataset construction and data preprocessing to obtain an original concrete dataset including complete representations of input samples; Step S200: Candidate gene screening of the original concrete dataset obtained in Step S100 based on elastic network regression to obtain an optimized concrete dataset; Step S300: Selecting an initial concrete performance prediction sub-model, and training the initial concrete performance prediction sub-model based on the optimized concrete dataset obtained in Step S200 to obtain an initial concrete multi-objective mix design prediction model; Step S400: Optimizing the initial concrete multi-objective mix design prediction model based on reinforcement learning to generate a concrete mix design strategy that satisfies engineering constraints and achieves balance under multiple objectives, to obtain a final concrete multi-objective mix design prediction model; Step S500: Predicting based on the final concrete multi-objective mix design prediction model obtained in Step S400 and outputting the concrete multi-objective mix design.

[0024] In this preferred embodiment, before the mix design optimization modeling, a concrete dataset with engineering relevance and holographic feature expression capabilities was constructed, and systematic data preprocessing was performed. Step S100 includes the following steps: Step S101: Construct a multi-source, multi-scale concrete material genome dataset. Considering that concrete is a complex composite material with high-dimensional and multi-objective coupling, its performance is influenced by a variety of factors, exhibiting the following significant characteristics: High-dimensional mix proportion space: Concrete mix proportions involve multiple variables, resulting in a complex composition space with significant inter-variable influences; Multi-scale structural characteristics: Performance is not only affected by composition but also depends on the microstructure, requiring modeling and characterization using structural characterization techniques (such as pore scanning imaging); Performance coupling and conflict: Significant nonlinear relationships and synergistic / conflicting relationships often exist between the three typical performance indicators of compressive strength, impermeability, and carbonation resistance, making comprehensive optimization difficult using traditional single-objective methods. Based on the material genome concept, the key components and structural features of concrete are abstracted into structured "genes," as detailed in [link to details]. Figure 2Specifically, it includes: ① basic characteristic genes of concrete; ② coarse and fine aggregate structure genes; ③ microstructure genes of concrete; ④ typical performance genes of concrete. Further preferably, the basic characteristic genes of concrete include cement, water, water-cement ratio / water-cement ratio, admixtures, fly ash, slag, ultrafine slag, additives, silica fume, limestone, and oxides; oxides include at least one of CaO, SiO2, Al2O3, MgO, and Fe2O3; the coarse and fine aggregate structure genes include coarse aggregate particle size, natural sand, and manufactured sand; the coarse aggregate particle size includes at least one of crushed stone of 5-10mm, 5-12.5mm, 5-16mm, 5-20mm, 5-30mm, 10-20mm, and 20-30mm; the microstructure genes of concrete include porosity, diffusion coefficient, pore structure, water-reducing agent, and air-entraining agent; the typical performance genes of concrete include impermeability, compressive strength, and carbon resistance. The final result includes... A concrete sample, The original concrete dataset with 1 feature gene D Each record corresponds to an actual concrete mix design and its corresponding performance indicators: ; in: Indicates the first The raw materials of the concrete sample are composed of numerical characteristics. and categorical features composition; These are typical properties of concrete.

[0025] Step S102: Outlier identification and processing are performed on the numerical features in the concrete material gene dataset using a statistical method based on standard deviation. Specifically, a statistical method based on standard deviation is used to analyze and process outliers in the numerical features. Perform outlier identification and handling, such as for the first... Numerical features Calculate its sample mean and standard deviation Its outlier upper boundary and lower boundary for: ; ; in: This is the tolerance factor, typically set to 2, corresponding to approximately 95% coverage of the normal distribution. By removing outliers from the data based on the upper and lower bounds, we ensure a reasonable data distribution and prevent interference from extreme points during model training, thereby improving the model's fitting stability and generalization ability.

[0026] Step S103: Normalize all numerical features in the concrete material gene dataset using the max-min normalization method, i.e., dimension normalization. To eliminate the influence of different dimensions on model training, the max-min normalization method is used to normalize all numerical features to the [0, 1] interval: ; in: It is The first concrete sample Numerical features; for The normalized value; , Respectively The maximum and minimum values ​​of numerical features are normalized, which facilitates model convergence and improves training efficiency.

[0027] Step S104: Perform one-hot encoding on the categorized features in the concrete material gene dataset; concatenate all encoded one-hot vectors into the original feature vectors to form the original concrete dataset containing a complete representation of the input samples. Specifically: For categorical features, convert them into structured 0 / 1 vector representations. For the th Classification features have One possible value Then the one-hot encoding form is: ; in: It is The first concrete sample Each categorical feature; This only indicates that the category position corresponding to the current sample is 1, and the rest are 0. All the encoded one-hot vectors will be concatenated into the original feature vector to form the final input sample. The complete representation of .

[0028] Because concrete datasets contain many high-dimensional features, this invention constructs a candidate gene screening mechanism based on Elastic Net Regression to further identify key features contributing to the multi-objective performance of concrete. This method possesses sparse modeling capabilities, effectively controlling overfitting and eliminating redundant or low-relevance features while preserving performance prediction capabilities, thus providing high-quality input variables for subsequent modeling and optimization. In this embodiment, the preferred step S200 includes the following steps: Step S201: Based on the elastic network regression model combining L1 and L2 regularization, define the objective function: ; in: It is the feature matrix of the original concrete dataset. This represents the number of concrete samples in the concrete material gene dataset. The number of characteristic genes in the concrete material gene dataset; The three target performance indicators are concrete carbonation depth, compressive strength, and chloride ion penetration resistance; This is the regression coefficient vector, representing the linear contribution of each feature to the performance index; It is the intercept term; It is a regularization strength hyperparameter that controls the magnitude of the overall penalty term; This is the regularization ratio coefficient, which controls the mixing ratio of L1 regularization and L2 regularization. This indicates a minimize operation; The operation represents the L1 norm of a vector, which is the sum of the absolute values ​​of its elements; The operation represents the square of the L2 norm of a vector, that is, the sum of the squares of each element.

[0029] The resilient network regression model combines the advantages of Lasso (L1 regularization) and Ridge (L2 regularization), making it suitable for engineering scenarios with high variable dimensionality and collinearity. This example is based on a preprocessed concrete dataset. The ratio of the training, test, and validation sets for training the elastic network regression model was set to 4:3:3. This model utilizes the raw material characteristics of concrete samples. As input, the sparsity screening of concrete features is achieved through a regularization mechanism.

[0030] Step S202: Based on the original concrete dataset obtained in step S100, train the model to obtain a trained elastic network regression model; then, based on the trained elastic network regression model, screen candidate genes to obtain a concrete optimization dataset. Specifically, after the elastic network regression model is trained, the final output is a set of sparse feature coefficient vectors, the mathematical expression of which is: ; For each feature in the concrete dataset, the regression coefficients This represents its linear regression coefficient on the target performance. If... This indicates that the feature has no significant impact on concrete performance indicators, and is therefore judged as a redundant or weakly correlated feature, and removed from the dataset; if This indicates that the feature has a significant impact on concrete performance and is defined as a "candidate gene" (i.e., a key input feature). (This refers to the original concrete dataset.) After applying the above filtering logic, retain all those that meet the criteria. The key variables (i.e., candidate genes) were identified, and redundant or weakly correlated variables were removed to obtain the final concrete optimization dataset. ,as follows: ; ; ; in: Indicates the first The key input features retained by each sample through the elastic network screening include Numerical features and Each categorical feature; The target performance indicators of the samples are represented by their carbonization depths. 28-day compressive strength Electric flux .

[0031] The specific experiment used an elastic network to screen candidate genes for concrete component characteristics, including: CaO, SiO2, Al2O3, MgO, Fe2O3, cement, admixtures, water, water-cement ratio, water-reducing agent, air-entraining agent, natural sand, manufactured sand, 5-10mm crushed stone, 10-20mm crushed stone, 20-30mm crushed stone, porosity, and diffusion coefficient genes. Therefore, these components were used as candidate genes (input variables) for subsequent experimental modeling.

[0032] In this preferred embodiment, step S300 involves modeling a multi-objective prediction model driven by coupled performance and constructing a concrete knowledge graph. To accurately characterize the multi-performance coupling characteristics and inter-objective conflicts of concrete materials during mix optimization, this embodiment constructs a multi-objective performance prediction model with concrete material perception capabilities. Combined with the nonlinear correlation characteristics between concrete structure and performance, it can effectively establish the mapping relationship between mix parameters and key performance indicators, providing high-quality environmental feedback for reinforcement learning strategy optimization. Based on the constructed concrete optimization dataset... The ratio of the training set, test set, and validation set for training the model was set to 4:3:3. The input features were selected concrete candidate features, representing not only material composition parameters but also the influence path of microstructure on macroscopic performance. The specific steps included were as follows: Step S301: Establish prediction sub-models for the initial concrete performance using the random forest regression method, defining the following multi-objective function. : ; ; in: Input the proportions. , including filtered Numerical features and One categorical variable; These represent the predicted outputs for carbonization depth, 28-day compressive strength, and electrical flux, respectively. In order to target the A random forest regression model is constructed based on several target performance parameters. For the first The regression tree in the first Fitting function on each target This represents the total number of trees.

[0033] During training, mean squared error is used as the objective function for optimization.

[0034] To reflect the performance of concrete in terms of mechanical and durability properties, the output... It includes three key indicators: ① Carbonation depth (mm) characterizes the concrete's resistance to carbonation; the optimization objective is to minimize it. 28-day compressive strength (MPa), characterizing load-bearing capacity, is converted to a negative value to conform to the minimization direction; ③ : Electric flux (C), characterizing resistance to chloride ion penetration, with the optimization objective being to minimize it.

[0035] Step S302: Based on the concrete optimization dataset obtained in step S200, train the prediction sub-model of the initial concrete performance to obtain the initial concrete multi-objective mix proportion prediction model. As a performance evaluation interface in the reinforcement learning model, it supports the policy network to quickly simulate and provide feedback on the target performance of concrete after each round of mix proportion generation, forming a closed-loop optimization mechanism.

[0036] Simultaneously, the dataset is optimized based on the contribution of concrete composition features output by the prediction model to the target performance. For example, water, cement, oxides (CaO, SiO2, Al2O3, MgO, Fe2O3), admixtures, and porosity contribute significantly to the target performance, while other features contribute relatively less. Specifically, the data density is increased for the mix proportion regions represented by the high-contribution features, and a concrete material knowledge graph is constructed. High-contribution features are selected as key nodes in the graph, and the relationships and weights of the graph edges are defined by the direction and intensity of the variables' influence on performance. A basic triplet of (material composition / microstructure, relationship, performance index) is constructed, ultimately yielding the concrete material knowledge graph, which is used to guide data and model optimization. For other low-contribution input features, their adjustment frequency is reduced in subsequent strategy optimizations based on their physical correlation with concrete performance, thereby improving efficiency and avoiding attention to irrelevant variables.

[0037] In this preferred embodiment, to achieve synergistic optimization of concrete mix design among multiple performance objectives, the present invention proposes a multi-objective optimization strategy based on reinforcement learning, constructing an intelligent decision-making model to learn how to achieve performance balance optimization in a high-dimensional continuous space. Step S400 is the multi-objective optimization strategy based on reinforcement learning, including the following steps: Step S401: Formalize the concrete mix design problem into a Markov decision process; Step S402: Design the policy network and value function network; Step S403: Construct the policy loss function; obtain the value function loss; obtain the final multi-objective loss function; Step S404: Generate a concrete mix design strategy that satisfies engineering constraints and achieves balance under multiple objectives, and obtain the final concrete multi-objective mix design prediction model.

[0038] The specific measures in this embodiment are as follows: The concrete mix design optimization problem is essentially a continuous decision-making problem under multiple objectives, multiple variables, and high constraints, which can be formalized as a Markov decision process. ,as follows: state space : This indicates the current concrete mix design.

[0039] Action space In a continuous action space, the action vector This represents the adjustment amount for the numerical characteristics of the current concrete; while for the categorical characteristics... Unlike other materials, the type of material used in actual engineering projects is usually fixed in the short term once it is determined.

[0040] State transition function Since the concrete mix proportions are deterministic, the state transition function simplifies to: By using the clip operation to restrict each variable to the physical / standard feasible domain, the RL search process is guaranteed to strictly follow the engineering feasibility of concrete materials. This represents the minimum value of a numerical feature. ; This represents the maximum value of the numerical feature. .

[0041] reward function Based on a pre-trained multi-objective prediction model Based on the prediction results of the target performance, a multi-objective weighted loss function is constructed. In other words, the better the performance, the greater the reward, where: weighting coefficient It reflects the constraints on the performance preferences of concrete under different engineering scenarios.

[0042] Design strategy networks and value function networks.

[0043] Policy Network The policy network outputs an action distribution in a continuous action space, which is set to a multidimensional Gaussian distribution, with each dimension being independent: ; in: It follows a multidimensional normal distribution, and the policy network outputs the mean action. and variance The material properties that match the continuous adjustability of concrete mix proportions and the differences in uncertainty of different variables.

[0044] Subsequent actions can be clipped into the physically feasible domain, ensuring that the generated proportions meet engineering specifications.

[0045] Value function network The agent is in a state Next, follow the strategy Expected cumulative return on discounts discount factor Used to regulate the importance of target performance, suitable for concrete materials with a lifespan much longer than the training step length; This indicates the instant reward for the first step starting from the current time.

[0046] Step S403 is as follows: Constructing the policy loss function as follows: ; in: The strategy ratio function measures the relative preference between the old and new strategies for the same action; the dominance function... Reflecting on taking action Compared to the relative goodness or badness of average behavior; shear threshold and The operation is used to limit the policy step size to avoid extreme step size updates in the concrete performance space where there are significant conflicts between objectives.

[0047] The training objective of a value function network is to minimize the predicted value. With the instantaneous value of the reward function The error between them, so the value function loss as follows: .

[0048] The final multi-objective loss function as follows: .

[0049] Model training can be optimized using methods such as cross-validation or Bayesian optimization. The loss function is jointly trained and updated on each mini-batch of samples, using the Adam optimizer to ensure training stability and convergence speed.

[0050] The above is a complete training process for a multi-objective optimization model based on reinforcement learning.

[0051] (1) Initialize the policy network Value function network and experience buffer pool Set reinforcement learning hyperparameters, such as learning rate, discount factor, entropy coefficient, and clipping range. wait.

[0052] (2) From the current state Sampling action Execute actions Get the next state The system uses a multi-objective prediction model to calculate three performance metrics and calculates the immediate reward based on a weighted reward function. Simultaneously store samples to the buffer pool And update the current status.

[0053] (3) For each target Maintenance latest Predicted value of step According to variance Normalize it into a new target weight It automatically focuses on material performance targets with large fluctuations (such as durability indicators, which are often more unstable in the early stages) and achieves dynamic adaptive control.

[0054] (4) Calculate the dominance function by randomly drawing a small batch of samples from the experience pool. Update strategy loss Simultaneously, minimize the mean squared error between the predicted value and the instantaneous reward, and update the value function loss. .

[0055] (5) Construct the joint loss function The Adam optimizer is used to perform backpropagation and parameter updates on the policy network and value function network, respectively. (6) Repeat steps (2) to (5) until the strategy converges, and finally obtain a concrete mix design strategy that can directly generate concrete that satisfies engineering constraints and achieves equilibrium under multiple objectives. .

[0056] By applying the scheme of this embodiment, after completing the training of the reinforcement learning model, the agent can generate a mix design that meets the requirements of multi-performance coordination from any initial state within the practical feasible domain of concrete materials.

[0057] Specifically, let the initial state be Call the policy network to generate actions Update the status to the new concrete mix design. And through a multi-objective prediction model The three key performance parameters are evaluated, and the overall performance of the solution can be calculated.

[0058] A total of 5,774 data points were collected for three typical performance metrics. After preprocessing such as missing value imputation, outlier removal, feature normalization, and one-hot encoding, 1,315 data points remained, which were then used to form the original dataset with material semantics.

[0059] Subsequently, an elastic network was used to sparsely screen the concrete composition characteristics, obtaining candidate genes highly correlated with compressive strength, impermeability, and chloride ion permeability: CaO, SiO2, Al2O3, MgO, Fe2O3, cement, admixtures, water, water-cement ratio (w / b), water-reducing agent dosage, air-entraining agent, natural sand, artificial sand, crushed stone 5-10mm, crushed stone 10-20mm, crushed stone 20-30mm, porosity, and diffusion coefficient. Then, based on these, random forest regression models for three typical properties were trained as performance feedback functions in the reinforcement learning optimization stage.

[0060] Objective function (minimization framework, measuring three properties by carbonization depth, 28-day compressive strength, and electrical flux): Carbonization depth (mm), the smaller the better; 28-day compressive strength (MPa), the higher the better; Electrical flux (C), the smaller the better (i.e., the stronger the resistance to chloride ion penetration). Table 1 shows a comparison between the schemes (RL1 and RL2) in this embodiment and existing technologies. Table 1. Statistics of three performance indicators of the ant colony algorithm in this embodiment and in the prior art.

[0061] Under the same data and constraints, the formulation schemes (RL1, RL2) obtained based on the method of this invention show significant improvements in carbonization depth, 28-day compressive strength, and electrical flux compared to commonly used ant colony optimization schemes. The formulation schemes are shown in Table 2. Table 2 Concrete formulas obtained by ant colony algorithm in this embodiment and in the prior art

[0062] Based on the above, the reinforcement learning-based multi-objective collaborative mix design optimization method for concrete materials in this embodiment achieves improvements in the following aspects compared to traditional methods: 1. Method for constructing a concrete material gene dataset Combining the characteristics of concrete as a multiphase composite material, a structured material gene dataset is constructed, which includes raw material composition, microstructural features (such as porosity, fly ash particle size distribution, etc.) and typical performance indicators. This dataset provides a data modeling foundation that integrates material composition, microstructure, and typical performance, enabling the optimization model to have material understanding capabilities and improving prediction accuracy and physical interpretability.

[0063] 2. Collaborative construction of candidate gene screening mechanism and prediction model In the original high-dimensional features of concrete, a sparse modeling mechanism using elastic networks is employed to eliminate low-contribution variables and screen key genes. Simultaneously, a high-precision prediction model is constructed to predict multi-objective performance, reflecting the nonlinear mapping and coupling principle between multiple performance indicators. The mix design optimization problem is formalized as a Markov process, and a reinforcement learning algorithm is introduced to train the policy network, enabling the model to possess dynamic objective balancing and high-dimensional continuous action control capabilities. This makes it suitable for collaborative control under multiple performance conflicts, significantly outperforming the fixed-objective weighting scheme of traditional algorithms.

[0064] 3. Feature contribution-driven construction and adaptive adjustment mechanism of materials knowledge graph The feature contribution values ​​from the performance prediction model are used to construct a concrete material knowledge graph. These contribution values ​​then influence policy adjustments (e.g., increasing sampling density for high-contribution variables and reducing action space for low-contribution variables), achieving a performance-driven optimization loop integrating data, model, and policy. By introducing feature contribution evaluation and material knowledge graph construction, a feedback loop from data to model to policy is achieved. During training, key features can be dynamically focused on, improving data utilization efficiency.

[0065] This invention also discloses a computer device, including a memory and a processor; The memory is used to store computer programs that can run on the processor; The processor is used to execute the computer program to implement the steps of the above-mentioned intelligent question-answering method for legal knowledge and the multi-objective mix design optimization method for concrete based on reinforcement learning.

[0066] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0067] The computer device may be a mobile phone, desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0068] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0069] The memory can be used to store the computer program and / or modules. The processor implements the computer program by running or executing the computer program and / or modules stored in the memory, and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0070] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0071] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described reinforcement learning-based multi-objective concrete mix optimization method.

[0072] This embodiment of the method processes text and images separately, uses multimodal feature processing, and combines visual features to extract medical relationships. Compared with the prior art, this embodiment of the method compensates for the lack of entity information by combining external knowledge, and integrates multiple medical images in a hierarchical manner to reduce the influence of unrelated information and improve the accuracy of relationship extraction.

[0073] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-objective mix design optimization method for concrete based on reinforcement learning, characterized in that, Includes the following steps: Step S100: Concrete dataset construction and data preprocessing to obtain the original concrete dataset including a complete representation of the input samples; Step S200: Based on elastic network regression, candidate genes are screened in the original concrete dataset obtained in step S100 to obtain the optimized concrete dataset; Step S300: Select the prediction sub-model of initial concrete performance, and train the prediction sub-model of initial concrete performance based on the concrete optimization dataset obtained in step S200 to obtain the multi-objective mix proportion prediction model of initial concrete. Step S400: Optimize the initial concrete multi-objective mix proportion prediction model based on reinforcement learning, generate a concrete mix proportion strategy that satisfies engineering constraints and achieves balance under multiple objectives, and obtain the final concrete multi-objective mix proportion prediction model. Step S500: Based on the final concrete multi-objective mix design prediction model obtained in step S400, make predictions and output the concrete multi-objective mix design.

2. The reinforcement learning-based multi-objective mix design optimization method for concrete as described in claim 1, characterized in that, Step S100 includes the following steps: Step S101: Construct a multi-source, multi-scale concrete material gene dataset, including: ① basic concrete characteristic genes; ② coarse and fine aggregate structure genes; ③ concrete microstructure genes; ④ typical concrete performance genes. Step S102: Use a statistical method based on standard deviation to identify and process outliers in the numerical features of the concrete material gene dataset; Step S103: Normalize all numerical features in the concrete material gene dataset using the maximum-minimum normalization method; Step S104: Perform one-hot encoding on the categorized features in the concrete material gene dataset; concatenate all the encoded one-hot vectors into the original feature vector to form the original concrete dataset containing a complete representation of the input samples.

3. The reinforcement learning-based multi-objective mix design optimization method for concrete according to claim 2, characterized in that, The basic characteristics of concrete include cement, water, water-cement ratio / water-binder ratio, admixtures, fly ash, slag, ultrafine slag, additives, silica fume, limestone, and oxides; oxides include at least one of CaO, SiO2, Al2O3, MgO, and Fe2O3; The coarse and fine aggregate structure includes coarse aggregate particle size, natural sand and artificial sand; coarse aggregate particle size includes at least one of crushed stone of 5-10mm, 5-12.5mm, 5-16mm, 5-20mm, 5-30mm, 10-20mm and 20-30mm. The microstructure of concrete includes porosity, diffusion coefficient, pore structure, water-reducing agent, and air-entraining agent. Typical performance characteristics of concrete include impermeability, compressive strength, and carbon resistance.

4. The reinforcement learning-based multi-objective mix design optimization method for concrete according to claim 2, characterized in that, Step S200 includes the following steps: Step S201: Based on the elastic network regression model combining L1 and L2 regularization, the following objective function is defined: ； in: It is the feature matrix of the original concrete dataset. This represents the number of concrete samples in the concrete material gene dataset. The number of characteristic genes in the concrete material gene dataset; The three target performance indicators for concrete are electrical flux, 28-day compressive strength, and carbonation depth, corresponding to its impermeability, compressive strength, and carbonation resistance. This is the regression coefficient vector, representing the linear contribution of each feature to the performance index; It is the intercept term; It is a regularization strength hyperparameter that controls the magnitude of the overall penalty term; This is the regularization ratio coefficient, which controls the mixing ratio of L1 regularization and L2 regularization; This indicates a minimize operation; The operation represents the L1 norm of the vector; The operation represents the squared L2 norm of the vector; Step S202: Train the original concrete dataset obtained in step S100 to obtain a trained elastic network regression model; screen candidate genes based on the trained elastic network regression model to obtain a concrete optimization dataset.

5. The reinforcement learning-based multi-objective mix design optimization method for concrete according to claim 4, characterized in that, Candidate genes include CaO, SiO2, Al2O3, MgO, Fe2O3, cement, admixtures, water, water-cement ratio, water-reducing agent, air-entraining agent, natural sand, artificial sand, 5-10mm crushed stone, 10-20mm crushed stone, 20-30mm crushed stone, porosity, and diffusion coefficient.

6. The reinforcement learning-based multi-objective mix design optimization method for concrete according to claim 4, characterized in that, Step S300 includes the following steps: Step S301: Establish prediction sub-models for the initial concrete performance using the random forest regression method, defining the following multi-objective function. : ； ； in: Input the proportions. , including filtered Numerical features and One categorical variable; These represent the predicted outputs for carbonization depth, 28-day compressive strength, and electrical flux, respectively. Indicates that for the first A random forest regression model is constructed based on several target performance parameters. Indicates the first The regression tree in the first Fitting function on each target The total number of trees; Step S302: Based on the concrete optimization dataset obtained in step S200, train the prediction sub-model of the initial concrete performance to obtain the initial concrete multi-objective mix proportion prediction model.

7. The reinforcement learning-based multi-objective mix design optimization method for concrete according to claim 6, characterized in that, Step S400 includes the following steps: Step S401: Formalize the concrete mix design problem into a Markov decision process; Step S402: Design the policy network and value function network; Step S403: Construct the policy loss function; obtain the value function loss; obtain the final multi-objective loss function; Step S404: Generate a concrete mix design strategy that satisfies engineering constraints and achieves balance under multiple objectives, and obtain the final concrete multi-objective mix design prediction model.

8. The reinforcement learning-based multi-objective mix design optimization method for concrete according to claim 7, characterized in that, Policy loss function in step S403 as follows: ； in: It is a time step Expectations; The strategy ratio function; The dominant function; The shear threshold; This operation is used to limit the policy step size; Value function loss as follows: ； in: Value function network; For multi-objective weighted loss functions; The final multi-objective loss function as follows: 。 9. A computer device, characterized in that, Including memory and processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the reinforcement learning-based multi-objective mix design optimization method for concrete as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the reinforcement learning-based concrete multi-objective mix design optimization method as described in any one of claims 1 to 8.

Images