Intelligent model construction system supporting rule chain

Abstract

The invention relates to automatic modeling language (AutoML) construction, in particular to an intelligent model construction system supporting a rule chain, which comprises a user demand and data receiving module, a machine learning model generation module and an appointed warehouse uploading module, the user demand and data receiving module is used for receiving a demand required by a user and data in a process that the user wants to put into machine learning model training, and extracting the user demand and the data; the machine learning model generation module generates a corresponding trained machine learning model and a corresponding API interface through a code generator; and the uploading to an appointed warehouse module packages the machine learning model generated by the machine learning model generation module and an API interface, and uploads the machine learning model and the API interface to a Docker warehouse appointed by a user outside the system for use by the user. According to the method, various codes with high adaptability and high reliability can be generated, and the efficiency and reliability of generating the machine learning model by the machine learning model generation module are improved.

Description

Technical Field

[0001] The present invention relates to automated model (AutoML) construction, which is an intelligent model construction system supporting rule chains. Background Art

[0002] The typical process of AutoML methods involves four basic stages: data preprocessing, feature engineering, model generation, and model validation. The first stage involves data preprocessing, (a) data collection methods to generate appropriate datasets to train ML models, (b) data visualization techniques used to help data analysts understand the internal data structure, (c) data cleaning mechanisms to remove noise present in the available datasets and avoid creating impaired models, and (d) data augmentation procedures used to expand the original dataset and enhance the robustness of the model by avoiding overfitting problems. The second stage involves extracting features from the raw data using feature engineering mechanisms to make the design of ML algorithms more convenient and effective in describing the data. It is mainly aimed at: (a) feature selection algorithms used to reject features in the original feature set that appear to be redundant or they degrade the performance of the model, and (b) feature extraction methods that perform dimensionality reduction using specialized functional data transformations to change the original data features. The third stage involves model generation, which is considered the core of the AutoML method. Model generation is divided into two processes, namely structure identification and architecture optimization. Structure identification involves determining the machine learning method that best solves the problem at hand. The typical structures commonly used are neural networks or deep neural networks. The next step is to optimize the architecture and set of hyperparameters (such as learning rate, number of layers, etc.) and come up with the best ML model. The fourth stage involves the evaluation of the model performance, which can be done based on well-defined performance metrics.

[0003] Rule chaining is a rule-based decision model that consists of a series of rules, each of which contains a condition and an action. When a condition is met, the rule triggers the corresponding action and passes control to the next rule. This continues until the termination condition is reached or all rules are executed. Since rule chaining can improve code reusability, simplify code logic, increase flexibility, and facilitate expansion, it is suitable for automatic generation in combination with automation tools.

[0004] Problems with existing technologies

[0005] AutoML is an automated model building method, and its goal is to build a machine learning model that meets the requirements, aiming to improve the performance and efficiency of the model. AutoML can be configured and adjusted by users to automate the process. However, AutoML is not visible in the process of generating models, and the generated model may be a black box model that is difficult to explain, and the entire generation process is not visible. The intelligent model building system that supports rule chains can monitor the process of model generation by setting various rules in the rule chain, and if an error occurs in the generation process, the cause of the error can be discovered in time. In addition, AutoML has limited customization capabilities. For some complex projects, users need to highly customize it, and the automated process of AutoML may not meet specific needs. The intelligent model building system that supports rule chains can better meet user needs because it uses the powerful generation capabilities of large language models and its powerful contextual reasoning capabilities, as well as its high sensitivity to few-shot prompts. The AutoML system adopts an end-to-end optimization strategy, aiming to find the best model and configuration through global search. This approach requires tight integration between different steps to ensure optimal overall performance. The intelligent model building system that supports rule chains separates each module and adopts message passing between modules to improve flexibility and scalability. In addition, since the details within the module do not affect the transmission between modules, maintainability is also greatly improved. Summary of the invention

[0006] The technical problem to be solved by the present invention is: How to realize the automation of machine learning intelligent model deployment

[0007] Automatic code generation and deployment can significantly reduce the time it takes to develop a model and put it into production. In traditional manual processes, developers need to communicate, test, and debug repeatedly, and manage every step of model development. Automated processes can reduce human intervention, reducing the time and manpower required for the development and deployment of new models.

[0008] Technical solution of the present invention:

[0009] An intelligent model building system supporting rule chaining includes: a user demand and data receiving module, a machine learning model generation module and an upload to a designated warehouse module.

[0010] The rule chain is a set of decision rules used to control the processing path of each module. Through the rule chain, the system can flexibly configure the execution path according to user needs, data quality, and model effects to achieve intelligent and dynamic processing;

[0011] Specific:

[0012] The user demand and data receiving module is used to receive the user's requirements and the data that the user wants to put into the machine learning model training process, extract the user requirements and data based on the machine learning model selection rules, and transmit them to the machine learning model generation module through the advanced message passing mechanism;

[0013] The machine learning model selection rules are used to constrain the results after user demand classification so that the classified results are not in the preset machine learning model category.

[0014] The machine learning model generation module generates the corresponding trained machine learning model and the corresponding API interface through the code generator. Specifically, the machine learning model generation module receives requirements and data, and automatically selects and generates a suitable machine learning model based on the machine learning model training rules and the API generation rules required by the user. The generated machine learning model will first be stored in the local server, and the generated model information will be passed to the upload to the designated warehouse module through the message passing mechanism;

[0015] The Upload to Designated Warehouse module packages the machine learning model and API interface generated by the machine learning model generation module based on the Upload to Designated Warehouse module rules, and uploads them to the Docker warehouse designated by the user outside the system for the user to use.

[0016] Beneficial Effects

[0017] The present invention uses a reinforcement learning code generator training method, so that the code generator can generate diverse, adaptable and reliable codes, thereby improving the efficiency and reliability of the machine learning model generation module in generating the machine learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention provides an intelligent model building system supporting rule chaining;

[0019] Figure 2 Data processing flow chart of the present invention;

[0020] Figure 3 Schematic diagram of the training process of the machine learning model generation module code generator of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described with reference to the accompanying drawings.

[0022] An intelligent model building system that supports rule chaining, such as Figure 1 As shown, it includes: user demand and data receiving module, machine learning model generation module and upload to designated warehouse module.

[0023] The rule chain is a set of decision rules used to control the processing path of each module. Through the rule chain, the system can flexibly configure the execution path according to specific conditions (such as user needs, data quality, model effect, etc.) to achieve intelligent and dynamic processing.

[0024] Specific:

[0025] The user demand and data receiving module is used to receive the user's requirements and the data that the user wants to invest in the machine learning model training process. Based on the machine learning model selection rules, it extracts user requirements and data and transmits them to the machine learning model generation module through an advanced messaging mechanism.

[0026] The machine learning model selection rules are used to constrain the results after user demand classification so that the classified results are not in the preset machine learning model category.

[0027] The machine learning model generation module generates the corresponding trained machine learning model and the corresponding API interface through the code generator. Specifically, the machine learning model generation module receives requirements and data, and automatically selects and generates a suitable machine learning model based on the machine learning model training rules and the API generation rules required by the user. The generated machine learning model will first be stored in the local server, and the generated model information will be passed to the upload to the designated warehouse module through the message passing mechanism.

[0028] Furthermore, the machine learning model training rules are used to generate training for the machine learning model code, which involves the division of training data, test data, and validation data, and terminating the training by observing changes in loss to prevent overfitting.

[0029] The user-required API generation rules are used to determine whether the generated API complies with general code specifications, such as using a string detection method to determine whether the API path is correct.

[0030] The Upload to Designated Warehouse module packages the machine learning model and API interface generated by the machine learning model generation module based on the Upload to Designated Warehouse module rules, and uploads them to the Docker warehouse designated by the user outside the system for the user to use.

[0031] After receiving the message, the upload to designated warehouse module transfers the locally stored machine learning model file to the designated Docker warehouse safely and efficiently. By introducing the advanced messaging mechanism Kafka, the throughput of messaging is increased, which is suitable for the transmission of large models and codes, and greatly improves the collaboration efficiency between modules. Using the Docker warehouse as the deployment target enhances the flexibility and reliability of model management and deployment.

[0032] Furthermore, the upload to designated warehouse module rules include: local file upload container rules and push to designated warehouse rules.

[0033] The local file upload container rule confirms the size of the file and performs SHA-256 integrity check on the file integrity. The push to specified warehouse rule pushes the packaged machine learning model and corresponding API according to the docker warehouse specified by the user.

[0034] Furthermore, the user demand and data receiving module is used for the interaction between the system and the user, capturing and analyzing the user's needs, and collecting and processing various data provided by the user. It includes: pre-trained language model LM and single-layer fully connected network FC.

[0035] in,

[0036] The pre-trained language model LM is an existing technology, which is used to convert user requirements from text form to digital form (matrix). The pre-trained language model can use open source models such as Bert.

[0037] The single-layer fully connected network FC consists of a layer of hidden neurons, and its output is a vector. Each value in the vector represents the probability of this task being classified into the corresponding preset task category. This user demand will be classified into the preset task category with the highest probability.

[0038] The fully connected network FC and the pre-trained language model LM are used for classification tasks, and their function is to classify user needs using the digital output of the pre-trained language model and determine the type of machine learning model. The fully connected network FC and the pre-trained language model are jointly trained using an open source dataset.

[0039] Processing flow such as Figure 2 As shown:

[0040] User requirements are input into the user requirements and data receiving module, which uses a combination of a pre-trained language model LM and a single-layer fully connected network FC to classify the machine learning models required by the user, such as target detection models, natural language processing models, etc. The types of machine learning models, such as target detection models, emotion recognition models, text classification models, image classification models, etc., are passed to the machine learning model generation module through a message passing mechanism to guide the code generator to generate basic machine learning models.

[0041] The machine learning model generation module uses a code generator to automatically generate a basic model that adapts to user needs, using the Python language and based on the pytorch library. It is trained and verified based on existing public data sets or data provided by users in the user needs and data receiving modules, and uses network search methods to adjust the model's hyperparameters (manually determined parameters) to obtain the best performance.

[0042] Figure 3 A schematic diagram of the training process of the machine learning model generation module, wherein the code generator includes a policy network (PN) and a large language model LLM. The policy network is composed of a pre-trained language model LM and a single-layer neural network SN, wherein the pre-trained language model LM is pre-trained and its parameters are frozen.

[0043] The agent is a policy network PN, which is used for code generation. The most suitable sample code is selected according to the requirements; during the training process, the actions generated by the training agent when given training examples are rewarded, that is, the accuracy of the generated code.

[0044] like Figure 3 The left part represents the work done by the agent, which is to select the appropriate code sample; Figure 3 The right part represents the environment in which the agent interacts. The agent updates its parameters by interacting with the environment. The environment refers to a series of processes for code generation.

[0045] Training phase (reinforcement learning method is used):

[0046] Step 1: The agent selects a class of code samples related to user needs from the code samples existing in the local data warehouse of the server through the semantic relevance between the code name and the user needs, as follows:

[0047] The code file name and description are encoded to obtain a vector, the user requirements are encoded to obtain a vector, and the cosine similarity (semantic relevance) of the two vectors is calculated.

[0048] According to the cosine similarity, the first n code files with the highest similarity to the user requirements are selected to form a code sample.

[0049] The code samples are processed by the policy network PN, and are screened again according to the relevance to the user's needs, so as to select several code samples that are more suitable for the user's needs.

[0050] Through iterative training, the policy network PN can gradually screen out code samples that better meet user needs so that the needs can be met in actual applications.

[0051] Furthermore, the local data warehouse on the server contains a large amount of open source code for machine learning models, API writing examples, and public data sets for model training.

[0052] Step 2: After screening, select code samples that meet the needs of the client user as samples, input the code samples and the customer needs into the large language model LLM as prompt words, and generate code through the large language model LLM output.

[0053] Step 3: Reward the policy network PN by the code errors in the generated code and the deviation from the conformity with the user requirements, so as to update the parameter θ of the policy network. The update of θ represents the optimization of the code sample selection strategy.

[0054] In the reinforcement learning training process, given a user requirement, the policy network PN finds K code samples e from the candidate pool of machine learning basic models. i ={e i 1 , e i 2 ,···,e i K}, and generate code through LLM Evaluating the generated code With the real code a i The reward is calculated based on the difference in the effects between the two.

[0055] Specifically,

[0056] The policy network PN selection code is as follows:

[0057]

[0058] Among them, p i represents the training sample, i.e., the user demand inputted in the training process is simulated; π represents the action taken under this training sample and the preliminarily screened code sample, i.e., the code sample that is more suitable for the user demand is selected; θ is the parameter in the policy network; each e i k It is an independent and simulated code sample used in the training of the candidate pool of machine learning base models. The code sample in the context is selected according to the policy network; E cand This is the first set of candidate code samples that are screened out based on the semantic relevance between code names and user needs. The subsequent policy network will then screen code samples from this set that are more in line with user needs.

[0059] The generated code as follows:

[0060]

[0061] During the training phase, the real code a i Corresponding to user needs i , obtained from the open source code on the Github website.

[0062] The reward calculation is as follows:

[0063]

[0064] is calculated on the current training example p i The generated code is The reward value obtained, r i Represents the reward for the agent, and R is the reward function.

[0065] Reward i Through the code generated by user requirements With the real code a i The reward is calculated based on the difference in the effects between the two. The specific calculation method is as follows:

[0066]

[0067] Among them, EVAL() is the evaluation function, which is implemented as follows:

[0068] Select n test cases and enter the generated code respectively and the real code a i , calculate the generated code And the real code a i The ratio of the number of identical outputs to the number of all outputs. If the ratio exceeds the threshold λ, it indicates that the generated code is similar to the real code, and the EVAL() function returns 1. If it does not exceed the threshold λ, the EVAL() function returns 0.

[0069] Furthermore, since the expected reward cannot be calculated by a direct mathematical formula, the present invention uses the Monte Carlo sampling method to estimate an accurate result by random sampling. Specifically, the rewards given by the reward function R for each candidate code sample selected by the policy network in this batch of training data are averaged and summed. This method can help approach the actual reward value without introducing deviations. That is:

[0070]

[0071] in, It represents the expected reward of the candidate code sample under the training sample. N is the number of samples selected in each batch generated by the training set. The purpose of this formula is to obtain the most appropriate reward so that the code result obtained in the next training is better than the previous one.

[0072] The goal of training optimization is to maximize the expected reward of generating answers under the policy network.

[0073] The present invention selects the policy gradient method, optimizes the reward according to the parameters of the policy network, and updates the parameters θ of the policy network PN.

[0074] The policy gradient algorithm in reinforcement learning is used to iteratively update the optimal parameters, which enable the sample code selected by the policy network PN in various situations to maximize the correctness of the code generated by the large language model LLM.

[0075]

[0076] The goal expressed on the left side of the formula is to minimize That is, the loss of the function. For this purpose, the parameter θ needs to be adjusted so that the generated e i The right side of the formula uses the policy gradient method to estimate the gradient of the parameters. Using the basic theorem of policy gradient, the expected gradient is obtained by calculating the logarithmic gradient of the policy. It is difficult to solve, so an approximate method is used to estimate the true gradient. The goal of the policy gradient is to maximize the accumulated reward by adjusting the parameters θ of the policy network. The update process is

[0077] Where α is the learning rate, that is, the rate at which the parameters are updated.

[0078] After the above reinforcement learning training process, the code generator can not only take the code generation of each file in the data warehouse as a target, but also take each segment in the code as a generation target. The generated code will have a higher availability and a more complete code structure, which can better meet the requirements of automated machine learning model generation.

[0079] The training strategy for API generation by the code generator can adopt the same training strategy as the above-mentioned code generator for machine learning model generation.

[0080] Application phase:

[0081] During code generation, the policy network trained during training is used, following the approach taken during training.

[0082] If there is user-provided data, that is, there are user-provided training examples, multiple K-fold cross-validation is used to split the user-provided data into a training set and a validation set. The training set is used for training, and the validation set is used to verify the effect of the machine learning model after training. At the same time, the code generator will generate the corresponding API template according to user needs, and package it with the trained model into a local file. The machine learning model that has been evaluated by evaluation indicators such as accuracy and F1 value and passed cross-validation, and the API interface generated by the code generator will be stored together in the specified local file. The machine learning generation module accepts user needs and needs from the data receiving module, uses a single-layer fully connected network FC and a pre-trained language model LM, and uses a method similar to the method of screening out sample codes that meet user needs to identify specific task types. If there is user-provided data, the approximate range of parameters required for the model will be predicted based on the scale of the data provided by the user, the number of data features, and the type of features. (It can also combine the operator's deep understanding of machine learning models and model selection rules to accurately select and generate machine learning models suitable for executing user-specified downstream tasks.) Subsequently, the model is efficiently trained under the premise of strictly following the machine learning model training rules using the public data set in the data warehouse or the user data passed in by the previous module. If a logical error is detected during the generation process, the system will immediately stop model building and return detailed error information to ensure the accuracy and reliability of the entire process. This highly automated and intelligent model generation and training mechanism can greatly improve the flexibility and adaptability of the system.

[0083] The upload repository module is used to upload packaged local files to the specified repository. This module supports automatic recognition and processing of multiple file formats, and can also intelligently generate optimized Docker image configurations based on file content. After the image generation is completed, the system will automatically trigger the push mechanism. The command writing only needs to modify the last push address. The system will recognize this command and push the image to the specified container repository. During the push process, the pushed image file will be integrity checked and security scanned through the hash value to ensure the security and availability of the model. This process significantly simplifies the steps for users to deploy machine learning models, improves deployment efficiency and the reliability of model management, and enables users to quickly and efficiently apply models to actual tasks.

[0084] Modular design decomposes a complex system into multiple independent modules, and uses a message passing mechanism between modules. Changes within a module will not affect other modules without changing the message passing mechanism between modules. This design makes the system more flexible, and a module can be easily replaced or upgraded without affecting the entire system. At the same time, modular design ensures that changes within a module will not affect the outside of the module, and different modules can be developed in parallel, thus shortening the development cycle. In addition, modular design also simplifies the testing process. Each module can be tested separately to ensure that its functions are correct before being integrated into the overall system. It can also ensure seamless connection between modules and the reliability of information transmission, and improve the flexibility of the overall system. (Through this modular design, the accuracy and efficiency of user needs and data extraction can be improved, and the seamless connection between modules and the reliability of information transmission can be ensured, and the flexibility of the overall system can be improved.)

[0085] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.

Claims

1. An intelligent model building system supporting rule chaining, characterized in that: include: User demand and data receiving module, machine learning model generation module and upload to designated warehouse module; The rule chain is a set of decision rules used to control the processing path of each module. Through the rule chain, the system can flexibly configure the execution path according to user needs, data quality, and model effect to achieve intelligent and dynamic processing. Specific: The user demand and data receiving module is used to receive the user's requirements and the data that the user wants to put into the machine learning model training process, extract the user requirements and data based on the machine learning model selection rules, and transmit them to the machine learning model generation module through the advanced message passing mechanism; The machine learning model generation module generates the corresponding trained machine learning model and the corresponding API interface through the code generator; specifically, the machine learning model generation module receives requirements and data, and automatically selects and generates a suitable machine learning model based on the machine learning model training rules and the API generation rules required by the user; the generated machine learning model is first stored in the local server, and the generated model information is passed to the upload to the designated warehouse module through the message passing mechanism; The Upload to Designated Warehouse module packages the machine learning model and API interface generated by the machine learning model generation module based on the Upload to Designated Warehouse module rules, and uploads them to the Docker warehouse designated by the user outside the system for the user to use.

2. The intelligent model building system supporting rule chaining as claimed in claim 1, characterized in that: The machine learning model selection rule is used to constrain the results after user demand classification, so that the classified results are not in the preset machine learning model category; The machine learning model training rule is used to generate training for the machine learning model code, involving the division of training data, test data, and validation data, and terminating the training by observing changes in loss to prevent overfitting; The user-required API generation rules are used to determine whether the generated API complies with general code specifications, such as using a string detection method to determine whether the API path is correct; The upload to designated warehouse module rules include: local file upload container rules and push to designated warehouse rules; The local file upload container rule confirms the size of the file and performs a SHA-256 integrity check on the file integrity; The push to specified warehouse rule pushes the packaged machine learning model and corresponding API according to the docker warehouse specified by the user.

3. The intelligent model building system supporting rule chaining as claimed in claim 1, characterized in that: The user demand and data receiving module is used for the interaction between the system and the user, capturing and analyzing the user's needs, and collecting and processing various data provided by the user; including: a pre-trained language model LM and a single-layer fully connected network FC; in, The pre-trained language model LM is used to convert user requirements from text form to matrix digital form; The single-layer fully connected network FC consists of a layer of hidden neurons. Its output is a vector. Each value in the vector represents the probability of this task being classified into the corresponding preset task category. The user demand is classified into the preset task category with the highest probability. The fully connected network FC and the pre-trained language model LM are used for classification tasks, and their function is to classify user needs using the digitized output of the pre-trained language model and determine the type of machine learning model; the fully connected network FC and the pre-trained language model are jointly trained through open source datasets.

4. The intelligent model building system supporting rule chaining as claimed in claim 1, characterized in that: The code generator includes a policy network PN and a large language model LLM; the policy network is composed of a pre-trained language model LM and a single-layer neural network SN, and the parameters of the pre-trained language model LM are frozen after pre-training; The policy network PN is an intelligent agent used for code generation, that is, selecting the most suitable sample code according to the requirements; During the training process, the agent is trained to generate actions given training examples, i.e., the accuracy of the generated code, by being rewarded.

5. The intelligent model building system supporting rule chaining as claimed in claim 4, characterized in that: The code generator training steps are as follows: Step 1: The agent selects a class of code samples related to user needs from the code samples existing in the local data warehouse of the server through the semantic relevance between the code name and the user needs, as follows: Encode the code file name and description to get a vector, encode the user requirements to get a vector, and calculate the cosine similarity of the two vectors, that is, the semantic relevance; According to the cosine similarity, the first n code files with the highest similarity to the user requirements are selected to form a code sample; The code samples are processed by the policy network PN, and are screened again according to the relevance to the user's needs, so as to select several code samples that are more suitable for the user's needs; Through iterative training, the policy network PN gradually selects code samples that better meet user needs so that the needs can be met in actual applications; The local data warehouse of the server stores a large amount of open source code of machine learning models, API writing examples, and public data sets for model training; Step 2: After screening, select code samples that meet the needs of the client user as samples, input the code samples and the customer needs into the large language model LLM as prompt words, and generate code through the large language model LLM output; Step 3: Reward the policy network PN by the code errors in the generated code and the deviation from the conformity with the user requirements, so as to update the parameter θ of the policy network. The update of θ represents the optimization of the code sample selection strategy.

6. The intelligent model building system supporting rule chaining as claimed in claim 5, characterized in that: In the reinforcement learning training process, given a user requirement, the policy network PN finds K code samples e from the candidate pool of machine learning basic models. i ={e i 1 , e i 2 ,···,e i K }, and generate code a through LLM i , evaluate the generated code With the real code a i The reward is calculated based on the difference in the effects between the two. Specifically, The policy network PN selection code is as follows: Among them, p i represents the training sample, i.e., the user demand inputted in the training process is simulated; π represents the action taken under this training sample and the preliminarily screened code sample, i.e., the code sample that is more suitable for the user demand is selected; θ is the parameter in the policy network; each e i k It is an independent and simulated code sample used in the training of the candidate pool of machine learning base models. The code sample in the context is selected according to the policy network; E cand This is the first set of candidate code samples that are screened based on the semantic relevance between code names and user requirements. The subsequent policy network will then screen code samples that are more in line with user requirements from this set. The generated code as follows: During the training phase, the real code a i Corresponding to user needs i , obtained from the open source code on Github; The reward calculation is as follows: is calculated on the current training example p i The generated code is The reward value obtained, r i Represents the reward for the agent, and R is the reward function; Reward i Through the code generated by user requirements With the real code a i The reward is calculated based on the difference in the effects between the two. The specific calculation method is as follows: Among them, EVAL() is the evaluation function.

7. The intelligent model building system supporting rule chaining as claimed in claim 6, characterized in that: The evaluation function EVAL() is implemented as follows: Select n test cases and enter the generated code respectively and the real code a i , calculate the generated code And the real code a i The ratio of the number of identical outputs to the number of all outputs. If the ratio exceeds the threshold λ, it indicates that the generated code is similar to the real code, and the EVAL() function returns 1. If it does not exceed the threshold λ, the EVAL() function returns 0.

8. The intelligent model building system supporting rule chaining as claimed in claim 6, characterized in that: Use Monte Carlo sampling to estimate the expected reward by random sampling; For each candidate code sample selected by the policy network in this batch of training data, the rewards given by the reward function R are averaged and summed to obtain a reward value close to the actual value, as follows: in, represents the expected reward of the candidate code sample under the training sample, and N is the number of samples selected from each batch generated by the training set.

9. The intelligent model building system supporting rule chaining as claimed in claim 8, characterized in that: Use the policy gradient method to update the parameters θ of the policy network PN; as follows: The goal expressed on the left side of the formula is to minimize That is, the loss of the function. For this purpose, the parameter θ needs to be adjusted so that the generated e i It has a higher value on the reward function R; the right side of the formula uses the policy gradient method to estimate the gradient of the parameters; using the basic theorem of policy gradient, the expected gradient is obtained by calculating the logarithmic gradient of the policy; Because of real expectations It is difficult to solve, and an approximate method is used to estimate the true gradient; the goal of the policy gradient is to maximize the accumulated reward by adjusting the parameters θ of the policy network; The update process is Where α is the learning rate, that is, the rate at which the parameters are updated.

Images