An intelligent model building system supporting rule chaining

Abstract

This invention relates to Automated Model Building (AutoML), specifically an intelligent model building system supporting rule chains. The system includes: a user requirement and data receiving module, a machine learning model generation module, and an upload-to-designated-repository module. The user requirement and data receiving module receives user requirements and the data the user wants to input into the machine learning model training process, extracting the user requirements and data. The machine learning model generation module generates the corresponding trained machine learning model and its corresponding API interface using a code generator. The upload-to-designated-repository module packages the machine learning model and API interface generated by the machine learning model generation module and uploads them to a user-specified Docker repository outside the system for user use. This invention can generate diverse, adaptable, and highly reliable code, improving the efficiency and reliability of the machine learning model generation module in generating machine learning models.

Description

Technical Field

[0001] This invention relates to automated model (AutoML) construction, specifically an intelligent model construction system that supports rule chains. Background Art

[0002] The typical AutoML process involves four basic stages: data preprocessing, feature engineering, model generation, and model validation. The first stage involves data preprocessing, including (a) data collection methods to generate appropriate datasets for training the ML model, (b) data visualization techniques to help data analysts understand the internal data structure, (c) data cleaning mechanisms to remove noise from the available dataset and avoid creating corrupted models, and (d) data augmentation procedures to expand the original dataset and enhance the model's robustness by avoiding overfitting. The second stage involves using feature engineering mechanisms to extract features from the raw data, making the design of ML algorithms more convenient and effective in describing the data. This mainly focuses on: (a) feature selection algorithms to reject features in the original feature set that appear redundant or degrade model performance, and (b) dimensionality reduction methods using specialized feature transformations to alter the original data features. The third stage involves model generation, considered the core of the AutoML process. Model generation is divided into two processes: structure identification and architecture optimization. Structure identification involves determining the machine learning approach that best solves the current problem. Commonly used typical structures are neural networks or deep neural networks. The next step is to optimize the architecture and hyperparameter set (e.g., learning rate, number of layers) and propose the best ML model. The fourth stage involves evaluating the model's performance, which can be done based on well-defined performance metrics.

[0003] A rule chain is a rule-based decision-making model consisting of a series of rules, each containing a condition and an action. When a condition is met, the rule triggers the corresponding action and passes control to the next rule. This process continues until a termination condition is met or all rules have been executed. Because rule chains improve code reusability, simplify code logic, increase flexibility, and are easily extensible, they are suitable for integration with automation tools for automated generation.

[0004] Problems with existing technology

[0005] AutoML is an automated model building method aimed at constructing machine learning models that meet requirements, thereby improving model performance and efficiency. AutoML allows users to configure and adjust the automated process. However, the model generation process is not visible, potentially resulting in a black-box model that is difficult to interpret, and the entire generation process remains unseen. Intelligent model building systems supporting rule chains can monitor the model generation process by setting various rules within the rule chain, and can promptly identify errors if they occur. Furthermore, AutoML has limited customization capabilities. For complex projects requiring high levels of user customization, AutoML's automated process may not meet specific needs. Intelligent model building systems supporting rule chains, leveraging the powerful generation capabilities of large language models, their strong contextual reasoning abilities, and high sensitivity to few-shot hints, can better meet user requirements. AutoML systems employ an end-to-end optimization strategy, aiming to find the optimal model and configuration through global search. This approach requires tight integration between different steps to ensure optimal overall performance. The intelligent model building system that supports rule chains separates each module and uses inter-module message passing, which improves flexibility and scalability. Furthermore, since the details inside the modules do not affect the transmission between modules, maintainability is also greatly improved. Summary of the Invention

[0006] The technical problem this invention aims to solve is: how to automate the deployment of machine learning intelligent models.

[0007] Automated code generation and deployment can significantly reduce the time from model development to production deployment. In traditional manual processes, developers need to repeatedly communicate, test, and debug, and management is required at every stage of model development. Automated processes can reduce human intervention, thus reducing the time and manpower consumed in the development and deployment phases of new models.

[0008] Technical solution of the present invention:

[0009] A smart model building system supporting rule chains includes: a user requirements and data receiving module, a machine learning model generation module, and an upload to a designated repository module.

[0010] The rule chain is a set of decision rules used to control the processing paths of each module. Through the rule chain, the system can flexibly configure execution paths according to user needs, data quality, and model performance, achieving intelligent and dynamic processing.

[0011] Specifically:

[0012] The user requirements and data receiving module is used to receive the user's requirements and the data that the user wants to input into the machine learning model training process. Based on the machine learning model selection rules, it extracts the user requirements and data and transmits them to the machine learning model generation module through an advanced messaging mechanism.

[0013] The machine learning model selection rule is used to constrain the results after user demand classification, so that the classified results are not in the preset machine learning model categories.

[0014] The machine learning model generation module generates trained machine learning models and corresponding API interfaces through a code generator. Specifically, the module receives requirements and data, and automatically selects and generates a suitable machine learning model based on the machine learning model training rules and the user's required API generation rules. The generated machine learning model is first stored on a local server, and then the generated model information is passed to the module that uploads it to the designated repository via a message passing mechanism.

[0015] The Upload to Specified Repository module, based on the rules of the Upload to Specified Repository module, packages the machine learning model and API interface generated by the machine learning model generation module and uploads them to the user-specified Docker repository outside the system for user use.

[0016] Beneficial effects

[0017] This invention uses a reinforcement learning-based code generator training method, which enables the code generator to generate diverse, adaptable, and reliable code, thereby improving the efficiency and reliability of the machine learning model generation module in generating machine learning models. Attached Figure Description

[0018] Figure 1 This invention provides an intelligent model construction system that supports rule chains;

[0019] Figure 2 Data processing flowchart of this invention;

[0020] Figure 3 A schematic diagram of the training process of the machine learning model generation module code generator of this invention. Detailed Implementation

[0021] The invention will be further described with reference to the accompanying drawings.

[0022] A smart model building system that supports rule chains, such as Figure 1 As shown, it includes: a user requirements and data receiving module, a machine learning model generation module, and an upload to a designated repository module.

[0023] The rule chain is a set of decision rules used to control the processing paths of each module. Through the rule chain, the system can flexibly configure execution paths according to specific conditions (such as user needs, data quality, model performance, etc.) to achieve intelligent and dynamic processing.

[0024] Specifically:

[0025] The user requirements and data receiving module receives user needs and data that users want to input into the machine learning model training process. Based on the machine learning model selection rules, it extracts user needs and data and transmits them to the machine learning model generation module through an advanced messaging mechanism.

[0026] The machine learning model selection rule is used to constrain the results after user demand classification, so that the classified results are not in the preset machine learning model categories.

[0027] The machine learning model generation module generates trained machine learning models and corresponding API interfaces through a code generator. Specifically, the module receives requirements and data, and automatically selects and generates a suitable machine learning model based on the machine learning model training rules and the user's required API generation rules. The generated machine learning model is first stored on a local server, and then the generated model information is passed to the module that uploads it to the designated repository via a message passing mechanism.

[0028] Furthermore, the machine learning model training rules are used to generate machine learning model code for training, which involves dividing training data, test data, and validation data, and terminating training by observing changes in loss to prevent overfitting.

[0029] The API generation rules required by the user are used to determine whether the generated API conforms to general coding standards, such as using string detection to determine whether the API path is correct.

[0030] The Upload to Specified Repository module, based on the rules of the Upload to Specified Repository module, packages the machine learning model and API interface generated by the machine learning model generation module and uploads them to the user-specified Docker repository outside the system for user use.

[0031] Upon receiving the message, the module that uploads to the designated repository securely and efficiently transfers the locally stored machine learning model files to the designated Docker repository. By introducing the advanced messaging mechanism Kafka, message throughput is increased, making it suitable for transferring large models and code, and significantly improving the collaboration efficiency between modules. Using the Docker repository as the deployment target enhances the flexibility and reliability of model management and deployment.

[0032] Furthermore, the rules for uploading to the designated repository module include: local file upload container rules and push to the designated repository rules.

[0033] The local file upload container rules confirm the file size and perform SHA-256 integrity verification on the file. The push to a specified repository rule pushes the packaged machine learning model and corresponding APIs according to the user-specified Docker repository.

[0034] Furthermore, the user requirements and data receiving module is used for system-user interaction, capturing and parsing user needs, and collecting and processing various data provided by users. This includes a pre-trained language model (LM) and a single-layer fully connected network (FC).

[0035] in,

[0036] Pre-trained language models (LMs) are an existing technology used to convert user requests from textual form into numerical form (matrices). Pre-trained language models can utilize open-source models such as BERT.

[0037] A single-layer fully connected network (FC) consists of a single layer of hidden neurons. Its output is a vector, where each value represents the probability that the task is classified into a corresponding preset task category. The user's request will be classified into the preset task category with the highest probability.

[0038] The fully connected network (FC) and the pre-trained language model (LM) are used for classification tasks. Their function is to classify user needs using the digitized output of the pre-trained language model, thus determining the type of machine learning model. The fully connected network (FC) and the pre-trained language model are jointly trained using an open-source dataset.

[0039] Processing flow as follows Figure 2 As shown:

[0040] The user input module receives user requirements and data, and uses a combination of a pre-trained language model (LM) and a single-layer fully connected network (FC) to classify the machine learning model required by the user, such as an object detection model or a natural language processing model. The type of machine learning model, such as object detection, sentiment recognition, text classification, or image classification, is then passed to the machine learning model generation module via a message passing mechanism to guide the code generator in generating the basic machine learning model.

[0041] The machine learning model generation module uses a code generator to automatically generate a basic model adapted to user needs, using the Python language and based on the PyTorch library. It trains and validates the model based on existing public datasets or data provided by the user in the user needs and data receiving modules, and uses web search methods to adjust the model's hyperparameters (human-determined parameters) to achieve optimal performance.

[0042] Figure 3 This diagram illustrates the training process of the machine learning model generation module. The code generator includes a policy network (PN) and a large language model (LLM). The policy network consists of a pre-trained language model (LM) and a single-layer neural network (SN). The pre-trained language model (LM) is pre-trained and its parameters are frozen.

[0043] The agent is a policy network (PN) used for code generation. It selects the most suitable example code based on the requirements; during training, rewards are used to train the agent to generate actions given the training examples, thus determining the accuracy of the generated code.

[0044] like Figure 3 The left side represents the work performed by the agent, namely, selecting appropriate code examples; for example... Figure 3 The right side represents the environment in which the agent interacts, updating parameters through this interaction. The environment refers to the series of processes involved in code generation.

[0045] Training phase (using reinforcement learning methods):

[0046] Step 1: The agent selects a type of code example that is relevant to the user's needs from the code examples existing in the local data warehouse on the server, based on the semantic relevance between the code name and the user's requirements. Specifically:

[0047] The code file name and description are encoded into vectors, and the user requirements are encoded into vectors. The cosine similarity (semantic relevance) of the two vectors is calculated.

[0048] The top n code files with the highest similarity to user requirements are selected based on cosine similarity to form code examples.

[0049] The code samples are processed through the policy network PN, and then filtered again based on their relevance to user needs, selecting several code samples that are more suitable for user needs.

[0050] Through iterative training, the policy network PN can gradually filter out code examples that better meet user needs, so that the requirements can be met in actual applications.

[0051] Furthermore, the server's local data warehouse contains a large amount of open-source code for machine learning models, API writing examples, and public datasets for training models.

[0052] Step 2: After screening, select code examples that meet the needs of client users as samples. Input the code examples and client requirements into the Large Language Model (LLM) as prompt words, and generate code through the LLM output.

[0053] Step 3: Reward the policy network PN based on the code errors in the generated code and the deviation from the user's requirements, thereby updating the policy network's parameters θ. The update of θ represents the optimization of the selected code example policy.

[0054] During reinforcement learning training, given a user requirement, the policy network PN selects K code examples e from the candidate pool of the machine learning base model. i ={e i 1 e i 2 ,···,e i K}, and generate code via LLM. Evaluation of the generated code With real code a i The reward is calculated based on the difference in the effects produced by the two.

[0055] Specifically,

[0056] The policy network PN selection code is as follows:

[0057]

[0058] Where, p i π represents the training example, which simulates the user's input needs during training; π represents the action taken based on this training example and the initially selected code examples, i.e., selecting the code example that best suits the user's needs; θ is the parameter in the policy network; each e i k These are independent, simulated code examples used in training within the candidate pool of the machine learning base model; code examples in the context are selected based on the policy network. cand This is the first set of candidate code examples selected based on the semantic relevance of both the code name and the user's needs. The subsequent policy network will then further filter code examples from this set to better match the user's needs.

[0059] The generated code as follows:

[0060]

[0061] During the training phase, the real code a i Corresponding user needs p i It was obtained from open-source code on the Github website.

[0062] The reward calculation method is as follows:

[0063]

[0064] It is to calculate the current training example p i The generated code is obtained below The reward value obtained, r i R represents the reward for the agent, and R is the reward function.

[0065] Reward r i Code generated from user requirements With real code a i The reward is calculated based on the difference in the effects produced by the two methods. The specific calculation method is as follows:

[0066]

[0067] Here, EVAL() is the evaluation function, and its specific implementation is as follows:

[0068] Select n test cases and input the generated code for each. and the real code a i Calculate and generate code And the actual code a i The EVAL() function returns 1 if the ratio of the number of identical outputs to the total number of outputs exceeds a threshold λ, indicating that the generated code is similar to the real code. Otherwise, the EVAL() function returns 0.

[0069] Furthermore, since the expected reward cannot be calculated using a direct mathematical formula, this invention employs Monte Carlo sampling to estimate an accurate result through random sampling. Specifically, it averages the rewards given by the reward function R for each candidate code example selected by the policy network in this batch of training data. This method helps to approximate the true reward value without introducing bias. That is:

[0070]

[0071] in, This represents the expected reward for candidate code examples given the training examples. N is the number of examples selected in each batch generated from the training set. The purpose of this formula is to obtain the most suitable reward so that the code results obtained in the next training session are better than those in the previous session.

[0072] The goal of training optimization is to maximize the expected reward for generating answers under the policy network.

[0073] This invention selects the policy gradient method to optimize the reward based on the parameters of the policy network and update the parameters θ of the policy network PN.

[0074] The optimal parameters are obtained by iteratively updating the policy gradient algorithm in reinforcement learning. The optimal parameters enable the example code selected by the policy network PN to maximize the correctness of the code generated by the large language model LLM in all cases.

[0075]

[0076] The left side of the formula represents the objective of minimizing That is, the loss of the function, for which the parameter θ needs to be adjusted so that the generated e i A higher value is found in the reward function R. The right-hand side of the equation uses the policy gradient method to estimate the gradient of the parameters. Using the fundamental theorem of policy gradient, the gradient of the expectation is obtained by calculating the logarithmic gradient of the policy. Due to the true expectation... Solving this problem is difficult, so an approximation is used to estimate the true gradient. The goal of the policy gradient is to maximize the accumulated reward by adjusting the parameters θ of the policy network. The update process is as follows:

[0077] Where α is the learning rate, that is, the rate at which the parameters are updated.

[0078] Through the above reinforcement learning training process, the code generator can not only take the generation of code for each file in the data warehouse as the target, but also take each segment of code as the generation target. The generated code will have higher availability and more complete code structure, which can better meet the requirements of automated generation of machine learning models.

[0079] The training strategy for API generation by the code generator can adopt the same training strategy as that for machine learning model generation by the code generator described above.

[0080] Application phase:

[0081] During code generation, the policy network trained during the training process is used, following the methods adopted during training.

[0082] If user-provided data exists, meaning user-provided training examples exist, multi-K-fold cross-validation is used to split the user-provided data into a training set and a validation set. The training set is used for training, and the validation set is used to verify the performance of the machine learning model after training. Simultaneously, the code generator generates corresponding API templates based on user requirements and packages them along with the trained model into a local file. The machine learning model, evaluated by metrics such as accuracy and F1 score and passing cross-validation, along with the API interface generated by the code generator, will be stored together in a designated local file. This machine learning generation module accepts user requirements and requirements from the data receiving module. Using a single-layer fully connected network (FC) and a pre-trained language model (LM), it identifies the specific task type in a manner similar to selecting sample code that meets user requirements. If user-provided data exists, it predicts the approximate range of parameters required for the model based on the scale of the user-provided data, the number of data features, and the type of features. (It can also combine the operator's deep understanding of machine learning models and model selection rules to accurately select and generate machine learning models suitable for performing user-specified downstream tasks.) Subsequently, using public datasets in the data warehouse or user data passed in from the previous module, the model is trained efficiently while strictly adhering to machine learning model training rules. If a logical error is detected during the generation process, the system will immediately stop model building and return detailed error information to ensure the accuracy and reliability of the entire process. This highly automated and intelligent model generation and training mechanism can significantly improve the system's flexibility and adaptability.

[0083] The upload repository module is used to upload packaged local files to a specified repository. This module supports automatic recognition and processing of various file formats and can intelligently generate optimized Docker image configurations based on file content. After image generation is complete, the system automatically triggers a push mechanism. The command only requires modification of the final push address; the system will recognize this command and push the image to the specified container repository. During the push process, the system performs integrity verification and security scanning on the pushed image file using hash values ​​to ensure the model's security and availability. This process significantly simplifies the steps for users to deploy machine learning models, improves deployment efficiency and model management reliability, and enables users to quickly and efficiently apply models to real-world tasks.

[0084] Modular design breaks down complex systems into multiple independent modules. These modules communicate using a message-passing mechanism, ensuring that changes within a module do not affect other modules without altering this mechanism. This design makes the system more flexible, allowing for easy replacement or upgrades of individual modules without impacting the entire system. Furthermore, modular design prevents changes within a module from affecting external modules, enabling parallel development of different modules and shortening the development cycle. In addition, modular design simplifies the testing process, allowing each module to be tested individually to ensure functionality before integration into the overall system. This further ensures seamless integration and reliable information transmission between modules, enhancing the overall system's flexibility. (Through this modular design, the accuracy and efficiency of user requirements and data extraction can be improved, and seamless integration and reliable information transmission between modules can be ensured, enhancing the overall system's flexibility.)

[0085] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. A smart model construction system supporting rule chains, characterized in that, include: The module includes a user requirement and data receiving module, a machine learning model generation module, and an upload to a designated repository module. The rule chain is a set of decision rules used to control the processing path of each module. Through the rule chain, the system can flexibly configure the execution path according to user needs, data quality, and model performance, so as to achieve intelligent and dynamic processing. Specifically: The user requirements and data receiving module is used to receive the user's requirements and the data that the user wants to input into the machine learning model training process. Based on the machine learning model selection rules, it extracts the user requirements and data and transmits them to the machine learning model generation module through an advanced messaging mechanism. The machine learning model selection rule is used to constrain the results after user demand classification, so that the classified results are not in the preset machine learning model categories; The result of the user demand classification refers to classifying user demands and determining the type of machine learning model; The machine learning model generation module generates the corresponding trained machine learning model and its corresponding API interface through a code generator. Specifically, the machine learning model generation module receives requirements and data, and automatically selects and generates a suitable machine learning model based on the machine learning model training rules and the API generation rules required by the user. The generated machine learning model is first stored on the local server, and the generated model information is passed to the upload to the designated repository module through a message passing mechanism. The Upload to Specified Repository module, based on the rules of the Upload to Specified Repository module, packages the machine learning model and API interface generated by the machine learning model generation module and uploads them to the user-specified Docker repository outside the system for user use.

2. The intelligent model construction system supporting rule chains as described in claim 1, characterized in that, The machine learning model training rules are used to generate machine learning model code for training. They involve dividing training data, test data, and validation data, and terminating training by observing changes in loss to prevent overfitting. The API generation rules required by the user are used to determine whether the generated API conforms to general coding standards. The rules for uploading to the specified repository module include: local file upload container rules and push to the specified repository rules; The local file upload container rules confirm the file size and perform SHA-256 integrity verification on the file integrity. The push to the specified repository rule pushes the packaged machine learning model and corresponding APIs according to the Docker repository specified by the user.

3. The intelligent model construction system supporting rule chains as described in claim 1, characterized in that, The user needs and data receiving module is used for interaction between the system and the user, capturing and parsing the user's needs, and collecting and processing various data provided by the user; including: a pre-trained language model LM and a single-layer fully connected network FC; in, The pre-trained language model LM is used to convert user requests from text to matrix numbers. A single-layer fully connected network (FC) consists of a single layer of hidden neurons. Its output is a vector, where each value represents the probability that the task is classified into a corresponding preset task category. User needs are classified into the preset task category with the highest probability. The fully connected network (FC) and the pre-trained language model (LM) are used for classification tasks. Their function is to use the digitized output of the pre-trained language model to classify user needs and determine the type of machine learning model. The fully connected network (FC) and the pre-trained language model are jointly trained using an open-source dataset.

4. The intelligent model construction system supporting rule chains as described in claim 1, characterized in that, The code generator includes a policy network PN and a large language model LLM; the policy network consists of a pre-trained language model LM and a single-layer neural network SN, and the parameters of the pre-trained language model LM are frozen after pre-training. The policy network PN is an intelligent agent used for code generation, that is, selecting the most suitable example code according to the requirements. During training, rewards are given to train the agent to generate actions based on training examples, i.e., the accuracy of the generated code.

5. The intelligent model construction system supporting rule chains as described in claim 4, characterized in that, The code generator training steps are as follows: Step 1: The agent selects a type of code example that is relevant to the user's needs from the code examples existing in the local data warehouse on the server, based on the semantic relevance between the code name and the user's requirements. Specifically: The code file name and description are encoded into vectors, and the user requirements are encoded into vectors. The cosine similarity between the two vectors is calculated, which is the semantic relevance. Based on cosine similarity, the top n code files with the highest similarity to user requirements are selected to form code examples; The code samples are processed through the policy network PN, and then filtered again based on their relevance to user needs, selecting several code samples that are more suitable for user needs. Through iterative training, the policy network PN gradually selects code examples that better meet user needs, so that the requirements can be met in actual applications. The server's local data warehouse contains a large number of open-source machine learning model codes, API writing examples, and public datasets for training models. Step 2: After screening, select code examples that meet the needs of client users as samples, input the code examples and client needs together into the Large Language Model (LLM) as prompt words, and generate code through the output of the Large Language Model (LLM). Step 3: Reward the policy network PN based on the code errors in the generated code and the deviation from the user's requirements, thereby updating the policy network's parameters θ. The update of θ represents the optimization of the selected code example policy.

6. The intelligent model construction system supporting rule chains as described in claim 5, characterized in that, During reinforcement learning training, given a user requirement, the policy network PN selects K code examples e from the candidate pool of the machine learning base model. i ={e i 1 e i 2 ,···,e i K }, and generate code via LLM. i Evaluation of the generated code i With real code a i The reward is calculated based on the difference in the effects produced by the two. Specifically, The policy network PN selection code is as follows: ， Where, p i This represents training examples, which are user requests that are simulated during the training process. This represents the action taken based on the training examples and the initially selected code examples, i.e., selecting the code examples that are more suitable for the user's needs; θ is the parameter in the policy network; each e i k These are independent code examples used in training within the candidate pool of basic machine learning models; code examples in the context are selected based on the policy network. This is the first set of candidate code examples selected based on the semantic relevance between the code name and user needs. The subsequent policy network will further select code examples from this set that better match user needs. The generated code i as follows: â i =LLM (ie i ,p i ) During the training phase, the real code a i Corresponding user needs p i The source code was obtained from the open-source website Github. The reward calculation method is as follows: r i ,=R(â i |p i ) R(â i |p i ) is the calculation of the current training example p i The generated code obtained below i The reward value obtained, r i R represents the reward for the agent, where R is the reward function; Reward r i Code generated from user requirements i With real code a i The reward is calculated based on the difference in the effects produced by the two methods; the specific calculation method is as follows: Here, EVAL() is the evaluation function.

7. The intelligent model construction system supporting rule chains as described in claim 6, characterized in that, The evaluation function EVAL() is implemented as follows: Select n test cases and input the generated code respectively. i and the real code a i Calculate and generate code i And the actual code a i The EVAL() function returns 1 if the ratio of the number of identical outputs to the total number of outputs exceeds a threshold λ, indicating that the generated code is similar to the real code. Otherwise, the EVAL() function returns 0.

8. The intelligent model construction system supporting rule chains as described in claim 6, characterized in that, The Monte Carlo sampling method is used to estimate the expected reward by random sampling; For each candidate code example selected by the policy network in this batch of training data, the reward given by the reward function R is averaged and summed to obtain a reward value close to the true value, as follows: in, This represents the expected reward for candidate code examples under the training examples, where N is the number of examples selected in each batch generated from the training set.

9. The intelligent model construction system supporting rule chains as described in claim 8, characterized in that, Update the parameters θ of the policy network PN using the policy gradient method, as follows: The left side of the formula represents the objective of minimizing This refers to the loss of the function, for which the parameter θ needs to be adjusted to ensure that the generated e i It has a higher value in the reward function R; the right side of the equation uses the policy gradient method to estimate the gradient of the parameters; using the fundamental theorem of policy gradient, the expected gradient is obtained by calculating the logarithmic gradient of the policy. Due to real expectations It is difficult to solve, so an approximation is used to estimate the true gradient; the goal of the policy gradient is to maximize the accumulated reward by adjusting the parameters θ of the policy network. The update process is as follows ; where α is the learning rate, i.e. the rate at which parameters are updated.

Images