Intelligent agent optimization method, device and system based on reinforcement learning, and storage medium

Abstract

The invention discloses an agent optimization method, device and system based on reinforcement learning, and a storage medium. The method comprises the steps of obtaining an initial task instruction input by a user; performing instruction deepening and expansion on the initial task instruction; optimizing and adjusting a task instruction and a data collection strategy by using a reinforcement learning algorithm; the task instruction is converted into a specific execution plan, and data collection, data cleaning and analysis, API calling, database retrieval and exception handling tasks are executed; evaluating whether the data collected by the intelligent agent is enough to support the analysis requirement of the current task; and generating an industry analysis result based on optimization data obtained by instruction deepening and expansion, and by adopting the technical scheme of the invention, the intelligent agent can adaptively generate, adjust and optimize the task instruction in a complex task environment, so that the intelligence, flexibility and efficiency of task processing are improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an intelligent agent optimization method and device, system, and storage medium based on reinforcement learning. Background Art

[0002] Existing intelligent agent systems mostly rely on predefined rules or simple learning algorithms to process tasks, and lack the ability to respond flexibly in complex environments. Usually, the task instructions of these systems are preset and cannot be effectively adjusted according to dynamic task requirements and environmental changes. Especially in complex multi-task scenarios, it is difficult for existing technologies to provide flexible task decomposition and instruction optimization mechanisms. In addition, although large language models have demonstrated strong capabilities in language generation and task understanding, existing technologies lack an effective method to combine them with reinforcement learning so that intelligent agents can adaptively optimize task execution paths and further improve the system's task completion efficiency and accuracy.

[0003] With the development of artificial intelligence and automation technology, agents have been widely used in many fields, such as autonomous driving, intelligent customer service, industry data analysis, etc. However, existing agent technology still has many limitations, especially when facing complex, multi-dimensional and dynamically changing tasks, the task processing capabilities and flexibility of existing agent systems are insufficient, as shown below:

[0004] 1. Static task instructions and execution paths

[0005] Most of the existing intelligent agent task execution systems rely on predefined task instructions and execution paths. These instructions are usually written by developers during the system design phase, and the instructions are generated based on fixed rules or limited conditional judgments. This approach makes it difficult for the intelligent agent to adapt to the dynamic changes in the task environment. Once the actual task exceeds the design expectations, the intelligent agent is often unable to respond flexibly. For example, in industry analysis tasks, if the system's task instructions are preset, the intelligent agent may find it difficult to dynamically adjust the analysis path or collect new data when facing new market changes or technological advances.

[0006] 2. Lack of automation in instruction adjustment and optimization

[0007] Existing intelligent agent systems lack effective adaptive adjustment mechanisms during task execution, and often require manual intervention to optimize task instructions and execution paths. For example, when an intelligent agent finds that there is insufficient data in an industry analysis task, the intelligent agent in existing technologies is usually unable to expand the task autonomously, and may need to manually redesign instructions, increase the time span, or expand the analysis dimension. This manual adjustment method is not only labor-intensive, but also inefficient in responding to rapidly changing task requirements.

[0008] 3. Limitations of using large language models

[0009] Although large language models have demonstrated powerful capabilities in language understanding and generation, existing intelligent agent systems have not effectively combined the advantages of large language models to automatically generate task instructions, which lacks deep language understanding capabilities when dealing with complex industry analysis, technology trend prediction and other tasks. For example, existing industry analysis agents may be able to process structured financial data or market data, but often have difficulty in generating complex task instructions (such as comprehensive analysis of technology development, market competition and user behavior). This limitation makes existing agents show low intelligence in task instruction generation and adaptive optimization.

[0010] 4. Insufficient application of reinforcement learning in task management

[0011] Although existing intelligent agent systems have begun to use reinforcement learning for task optimization, they are mostly used for path planning and simple task scheduling in static scenarios. The feedback mechanism of reinforcement learning is usually used to optimize the execution efficiency of a single task, while it has weak support for adaptive optimization of multi-task scenarios and complex tasks. When faced with a multi-dimensional, multi-task complex environment such as industry analysis, existing reinforcement learning methods often find it difficult to fully utilize real-time feedback to adjust the task execution path. For example, in a task of analyzing technology trends and market demand, the intelligent agent may only focus on completing the preset industry analysis tasks, while ignoring the dynamic changes in technology trends, resulting in incomplete final analysis results.

[0012] 5. Insufficient correlation between data feedback and task progress

[0013] During the task execution process, existing intelligent agent systems usually lack the ability to process data feedback in real time during the task execution process. Data feedback and task progress status are usually separated, and the intelligent agent cannot optimize the next task instruction based on real-time feedback. For example, in industry data analysis tasks, the intelligent agent may continue to execute the preset task when there is insufficient data, resulting in deviations in the final results, and unable to adjust the analysis path in time, expand data acquisition, or generate new analysis dimensions.

[0014] 6. Complexity and diversification of tasks make management more difficult

[0015] Existing intelligent agent systems usually show limitations in task complexity and task expansion, especially in dynamic task environments, where intelligent agents cannot effectively manage the task complexity process. For example, in industry analysis tasks, intelligent agents may only perform simple market data collection tasks, but cannot expand the task to multi-dimensional data analysis by adaptively adjusting analysis instructions, such as comprehensive analysis of technology trends, user behavior, and financial data. This inflexible system is difficult to cope with actual complex task scenarios, and the efficiency and quality of task execution are also low.

[0016] In summary, existing intelligent agent systems have shown obvious deficiencies in dealing with complex tasks and changing task requirements, especially in task generation, task adjustment and optimization, task complexity and diversified management. Existing technologies cannot fully utilize the language understanding capabilities of large language models and the feedback mechanism of reinforcement learning to achieve adaptive task optimization, which limits the intelligent performance and task execution efficiency of intelligent agents in complex tasks. Summary of the invention

[0017] The technical problem to be solved by the present invention is to provide an intelligent agent optimization method and device, system, and storage medium based on reinforcement learning, so that the intelligent agent can adaptively generate, adjust, and optimize task instructions in a complex task environment, thereby improving the intelligence, flexibility, and efficiency of task processing.

[0018] To achieve the above object, the present invention adopts the following technical solution:

[0019] An intelligent task optimization method comprises the following steps:

[0020] Step S1, obtaining an initial task instruction input by a user;

[0021] Step S2, deepening and expanding the initial task instructions;

[0022] Step S3: using a reinforcement learning algorithm to optimize and adjust task instructions and data collection strategies;

[0023] Step S4: convert the task instructions into a specific execution plan, and execute tasks including data collection, data cleaning and analysis, API calls, database retrieval, and exception handling;

[0024] Step S5: Evaluate whether the data collected by the agent is sufficient to support the analysis requirements of the current task;

[0025] Step S6: Generate industry analysis results based on the optimization data obtained through instruction deepening and expansion.

[0026] Preferably, in step S3, the reinforcement learning algorithm includes: deep Q network DQN, policy gradient method, Actor-Critic; the intelligent agent perceives the current task state by continuously interacting with the environment, selects the optimal strategy to execute task instructions, obtains rewards based on the execution results, and optimizes subsequent task instructions and data collection strategies.

[0027] Preferably, in step S5, the fine-tuned large language model is used to perform a comprehensive evaluation on the aggregated text data of different dimensions to determine its adequacy in a certain task and generate corresponding descriptive tags.

[0028] The present invention also provides an intelligent task optimization device based on reinforcement learning, comprising:

[0029] A user interaction interface module, used to obtain initial task instructions input by the user;

[0030] The task instruction deepening and expansion module is used to deepen and expand the initial task instructions;

[0031] A reinforcement learning-driven agent engine for optimizing and adjusting task instructions and data collection strategies using reinforcement learning algorithms;

[0032] Intelligent task execution and data processing engine, which is used to convert task instructions into specific execution plans and perform tasks including data collection, data cleaning and analysis, API calls, database retrieval and exception handling;

[0033] The data sufficiency assessment module is used to evaluate whether the data collected by the agent is sufficient to support the analysis requirements of the current task;

[0034] The analysis result output module is used to generate industry analysis results based on the optimization data obtained through instruction deepening and expansion.

[0035] Preferably, the reinforcement learning algorithm includes: deep Q network DQN, policy gradient method, Actor-Critic; the reinforcement learning-driven intelligent agent engine enables the intelligent agent to perceive the current task status through continuous interaction with the environment, select the optimal strategy to execute task instructions, obtain rewards based on the execution results, and optimize subsequent task instructions and data collection strategies.

[0036] Preferably, the data sufficiency assessment module uses a fine-tuned large language model to perform a comprehensive assessment on aggregated text data of different dimensions, determines its sufficiency in a certain task, and generates corresponding descriptive labels.

[0037] The present invention also provides an intelligent task optimization system based on reinforcement learning, comprising: a memory and a processor, wherein the memory stores a computer program run by the processor, and the computer program executes an intelligent task optimization method based on reinforcement learning when run by the processor.

[0038] The present invention also provides a storage medium, on which a computer program is stored, and the computer program executes an intelligent task optimization method based on reinforcement learning when running.

[0039] The present invention realizes intelligent task execution and optimization through task generation, instruction deepening and expansion, multi-step reasoning, feedback optimization and reinforcement learning adaptive learning mechanism, and adaptively adjusts the task execution and instruction optimization of the intelligent agent to realize automatic processing and efficient optimization of multi-dimensional complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 A schematic diagram of the principle of an intelligent task optimization method based on reinforcement learning according to an embodiment of the present invention;

[0042] Figure 2 The present invention is a flowchart of an intelligent task optimization method based on reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Embodiment 1:

[0046] The English abbreviations involved in the embodiments of the present invention are described as follows:

[0047] DQN (Deep Q-Network)

[0048] DQN is an algorithm that combines deep learning and reinforcement learning for decision-making problems in high-dimensional state space. It approximates the Q-value function through a neural network and estimates the expected return of performing an action in a specific state. This technology is widely used in task optimization and adaptive learning, and is an important method in the field of reinforcement learning.

[0049] RL (Reinforcement Learning)

[0050] RL refers to a machine learning method that achieves optimization goals by gradually adjusting strategies based on feedback information (rewards or penalties) through interaction with the environment. In the present invention, reinforcement learning is used for task execution optimization and strategy adjustment, helping the agent to adaptively adjust task instructions based on environmental feedback.

[0051] LLM (Large Language Model)

[0052] LLM refers to a deep learning model trained by a large number of corpora, which has powerful natural language processing capabilities and can understand and generate language. The present invention uses LLM to parse the task instructions input by the user and use it for data adequacy assessment.

[0053] BERT (Bidirectional Encoder Representations from Transformers)

[0054] BERT is a pre-trained model based on the Transformer architecture, which is widely used in natural language processing tasks. It can obtain the semantic information of words from the context of the text.

[0055] Sentence-BERT (SBERT)

[0056] Sentence-BERT is an improved version of the BERT model, which is specifically used to convert sentences into vector representations for sentence similarity calculation and text retrieval. Compared with traditional BERT, SBERT can generate sentence-level embedding representations more quickly and efficiently. In the system, SBERT is used to convert task instructions input by users into vectors to facilitate subsequent semantic understanding and task processing.

[0057] API (Application Programming Interface)

[0058] API is a tool and protocol that allows applications to communicate with each other. In the present invention, API is used to dynamically obtain data required for tasks from external data sources to ensure the acquisition of real-time information required for analysis.

[0059] Q value (Quality Value)

[0060] The Q value is a value used in reinforcement learning to represent the expected cumulative reward for performing an action in a specific state. In the DQN model of the present invention, the Q value is used to help the agent select the optimal task instruction.

[0061] Epsilon-greedy strategy

[0062] This is a common exploration strategy in reinforcement learning. When selecting an action, there is a certain probability that an action is randomly selected to promote exploration. In other cases, the action with the largest current Q value is selected for execution. In the present invention, the epsilon-greedy strategy is used to optimize the task instruction selection.

[0063] ReLU (Rectified Linear Unit)

[0064] ReLU is a commonly used activation function, which is widely used in neural networks. Its output is a larger value between the input value and 0, which can effectively alleviate the gradient vanishing problem and accelerate the training process of deep neural networks. In the present invention, ReLU is used to activate hidden layer neurons in the DQN network to help the model learn high-dimensional semantic features.

[0065] Actor-Critic

[0066] This is an algorithmic structure in reinforcement learning. The Actor is responsible for selecting actions, and the Critic is responsible for evaluating the quality of the current strategy and providing feedback to the Actor. In the intelligent agent system of the present invention, the Actor-Critic is used to optimize task instructions and adjust data collection strategies.

[0067] γ (Gamma, discount factor)

[0068] γ is a discount factor in reinforcement learning, which indicates the importance of future rewards. Its value is between 0 and 1. The closer γ is to 1, the more the model focuses on long-term rewards; the closer γ is to 0, the more the model focuses on immediate rewards. In the DQN model of the present invention, γ is used to calculate the Q value.

[0069] Experience Replay

[0070] This is a common technique in reinforcement learning to break the data correlation problem. The agent will store each interaction (state, action, reward and next state) in the experience pool, and randomly sample data from the experience pool for training during training to enhance the stability of the model. The experience replay mechanism is used in the DQN training process of the present invention.

[0071] like Figure 1 , 2 As shown, the embodiment of the present invention provides an intelligent task optimization method based on reinforcement learning, which solves the technical problems of single instruction generation, low data collection efficiency, and low quality of analysis results in industry analysis tasks; through the reinforcement learning algorithm, the task instructions and data collection strategies are continuously optimized, and finally a high-quality industry analysis report is generated. It includes the following steps:

[0072] Step S1, obtaining an initial task instruction input by a user through a user interaction interface module;

[0073] Step S2: deepening and expanding the initial task instruction through the task instruction deepening and expansion module;

[0074] Step S3, the intelligent agent engine driven by reinforcement learning optimizes and adjusts the task instructions and data collection strategies by using the reinforcement learning algorithm;

[0075] Step S4: The intelligent task execution and data processing engine converts the task instructions into a specific execution plan, and executes tasks including data collection, data cleaning and analysis, API calls, database retrieval, and exception handling;

[0076] Step S5: Evaluate whether the data collected by the agent is sufficient to support the analysis requirements of the current task through the data sufficiency evaluation module;

[0077] Step S6: Generate industry analysis results through the analysis result output module based on the optimization data obtained by instruction deepening and expansion.

[0078] As an implementation method of the embodiment of the present invention, the working process of the user interaction interface module is as follows:

[0079] As the entry point for the system to interact with the user, it is responsible for receiving the initial task instructions input by the user. For example, if the user inputs: "Please analyze the market trend of the new energy vehicle industry.", the system obtains the initial simple instructions from the user input or environmental perception as the starting point of the system.

[0080] Implementation:

[0081] Front-end: Building user interfaces using HTML, CSS, and JavaScript.

[0082] Backend: Use a web framework such as Flask or Django to handle user requests and responses, or integrate with other user systems.

[0083] As an implementation method of an embodiment of the present invention, the working process of the task instruction deepening and expansion module is as follows:

[0084] The data analysis dimension is used for instruction deepening strategy and instruction expansion strategy. When deepening the instruction, the system conducts a deeper analysis within the current dimension and selects a more specific option; when expanding the instruction, the system jumps from the current dimension to other dimensions and expands the scope of analysis. The actions in subsequent reinforcement learning will correspond to the specific strategy type (deepening or expanding) and dimension options one by one, so that the intelligent agent can accurately choose which specific option of which dimension to deepen, or which new dimension to expand to when making decisions.

[0085] Use common analysis dimensions and constraints in industry analysis as the basis for building directive deepening and directive expansion strategies:

[0086] Time dimension: past year, past five years, next three years, quarterly analysis, annual analysis.

[0087] Geographical dimension: global market, regional market (such as Asia, Europe), national market, provincial market, and city market.

[0088] Market dimensions: market size, market share, market growth rate, and market trends.

[0089] Competitive dimensions: competitive landscape, analysis of major competitors, threat of new entrants, and threat of substitutes.

[0090] Consumer dimension: consumer behavior, consumer preferences, demand analysis, and user portraits.

[0091] Technology dimension: technology development trends, impact of innovative technologies, and technology maturity.

[0092] Policy and regulatory dimension: government policies, industry supervision, and the impact of laws and regulations.

[0093] Supply chain dimension: supply chain analysis, raw material cost, supplier analysis.

[0094] Financial dimensions: revenue, profit, cost structure, return on investment.

[0095] Risk dimensions: industry risk, market risk, financial risk, and operational risk.

[0096] Instruction optimization strategies include:

[0097] Instruction deepening strategy: In instruction deepening, the agent conducts a deeper analysis within the current dimension, adds constraints, and makes the task instructions more specific and complex. Specific methods include adding constraints, refining the analysis dimension, introducing time span, etc. For example, the current task instruction is "analyze the market trend of a certain industry", which belongs to the market dimension. Using the instruction deepening strategy, the instruction is deepened to "analyze the market trend of a certain industry and deeply analyze the quarterly changes in the market growth rate."

[0098] Instruction extension strategy: In instruction extension, the agent jumps from the current dimension to a new dimension, adds a new analysis angle, and expands the task instruction. For example, the current task instruction is "analyze the market trend of a certain industry". This instruction belongs to the market dimension. Using the instruction extension strategy, it is extended from "market dimension" to "technology dimension". The extended task instruction is: "Analyze the impact of the technology development trend of a certain industry".

[0099] As an implementation method of an embodiment of the present invention, the working process of the intelligent agent engine driven by reinforcement learning is:

[0100] The intelligent agent engine driven by reinforcement learning is the core module of the embodiment of the present invention. By using reinforcement learning algorithms (such as deep Q network (DQN), policy gradient method, Actor-Critic, etc.), the optimization and adjustment of task instructions and data collection strategies are realized to ensure that the generated analysis results meet user needs and have high quality. The intelligent agent continuously interacts with the environment, perceives the current task state, selects the optimal strategy to execute the task instructions, obtains rewards based on the execution results, and optimizes subsequent task instructions and data collection strategies.

[0101] The embodiment of the present invention uses a self-developed deep Q network (DQN) with an improved reward function (Reward Function) based on a large language model as the core reinforcement learning algorithm to optimize the task execution strategy. Through the reward function evaluated based on the large language model, the intelligent agent is guided to optimize in the direction of improving the task completion rate and efficiency.

[0102] Introduction to Deep Q-Network (DQN) and related reinforcement learning concepts:

[0103] Deep Q-Network (DQN) is an algorithm that combines deep learning and reinforcement learning to solve decision-making problems with high-dimensional state spaces. DQN estimates the expected return (Q value) of performing an action in a given state by using a neural network to approximate the Q-value function. The core idea is to use a neural network to learn the Bellman equation for Q-value updates:

[0104] Q(s t ,a t)=r t +γmaxQ(s t+1 ,a′)

[0105] Among them, s t Indicates the current state, a t is the current action, r t is the reward and γ is the discount factor.

[0106] In the embodiment of the present invention, the core of reinforcement learning is how the intelligent agent gradually optimizes the task instructions and execution plan by interacting with the environment. Therefore, it is necessary to first define the three key elements of reinforcement learning - action set, state set and reward function, which will form the decision-making basis of the intelligent agent. The following is a specific reinforcement learning driven intelligent agent engine construction method:

[0107] Action Collection

[0108] Action encoding rules:

[0109] Strategy type (1 digit): D (deepen), E (extend)

[0110] Dimensions (1 bit): T (time), R (region), M (market), C (competition), S (consumer), H (technology), P (policy), S (supply chain), F (finance), K (risk)

[0111] Specific option number (2 digits): The specific option number under this dimension

[0112] Calculation of the number of actions:

[0113] Number of instruction deepening strategy actions: number of dimensions × number of specific options for each dimension

[0114] Assume that each dimension has an average of 5 specific options

[0115] Number of actions to deepen the strategy = 10 × 5 = 50

[0116] Number of instruction extension strategy actions: number of dimensions (the current dimension is not included in the specific execution, but for the convenience of matrix calculation, the current dimension is included in the matrix representation)

[0117] Number of expansion strategy actions = 10 (a total of 10 dimensions, the actual calculation process will exclude the current dimension and select from the remaining 9 dimensions)

[0118] Total number of actions: number of actions to deepen the strategy + number of actions to expand the strategy

[0119] Total number of moves = 50 + 10 = 60

[0120] Action Matrix Design:

[0121] The action matrix can be represented as a vector of 1 row and 60 columns, with each column corresponding to an action. t , the DQN model predicts the Q values ​​of 60 actions and selects the action with the largest Q value according to the epsilon-greedy strategy.

[0122] Instruction deepening strategy actions (partial representation, refer to the schematic):

[0123] D_T_01: Deepen the time dimension, specific option 01 (past year)

[0124] D_M_03: Deepen the market dimension, specific option 03 (market growth rate)

[0125] Expansion strategy actions (partial representation, reference diagram):

[0126] E_R: Expanding to the regional dimension

[0127] E_C: Expanding to the competitive dimension

[0128] Action selection formula description

[0129] Definition of action set A:

[0130]

[0131] Each action t Representation:

[0132] a t ∈A,t=1,2,…,60

[0133] Action selection:

[0134] In each state s t Next, the agent selects an action a through DQN t :

[0135]

[0136] Among them, Q(s t ,a) is the state s t The Q value of the next action a.

[0137] State collection and state representation

[0138] In the field of industry analysis, the state set of reinforcement learning usually does not have a clear discrete nature. To meet this challenge, the embodiment of the present invention designs a set of 10 labels based on data sufficiency, such as "severely insufficient data, unable to conduct effective analysis". These 10 descriptive labels for evaluating data sufficiency are universal in the data analysis industry, covering evaluations from severe data insufficiency to extremely sufficient data. They not only effectively simplify the representation of the state space, but also provide the system with a quantitative task data quality assessment standard, enabling the intelligent agent to better optimize task instructions and decisions based on data conditions. The specific 10 descriptive labels are as follows:

[0139] The data is seriously insufficient and further collection is recommended.

[0140] The data are extremely limited and may lead to significant bias in the analysis results.

[0141] The data are insufficient and the reliability of the analysis results is low. Supplementary data are needed.

[0142] Due to the small amount of data, the analysis results may not be comprehensive enough, affecting the accuracy.

[0143] The data are fair and the analysis is feasible, but the quality of the results needs to be improved.

[0144] The data are generally sufficient to support preliminary analysis, but there is still room for improvement.

[0145] The data are sufficient to allow for a relatively reliable analysis, but can be improved.

[0146] The data are sufficient to support reliable analysis.

[0147] The data are sufficient to support high-quality analysis.

[0148] The data are sufficient to support comprehensive and reliable high-quality analysis.

[0149] The role of these labels is not limited to state representation, but also serves as the prediction result of the core evaluation of the subsequent data sufficiency assessment module. The evaluation module uses the fine-tuned large language model to analyze the collected data and generate these fixed descriptive labels (this part will be specifically introduced in the data sufficiency assessment module). These labels are then converted into state vectors through a text embedding model (Sentence-BERT is used in this embodiment of the present invention) as state input in the reinforcement learning process.

[0150] In addition, these labels are directly linked to the reward function in reinforcement learning. Each label represents a different level of data quality and corresponds to a specific reward value. The reward value is used to guide the decision-making process of the agent, helping the agent to select the optimal task instructions and execution plan, and optimizing the overall effect of the industry analysis task.

[0151] Construction of state vector:

[0152] Descriptive label vectorization (768 dimensions):

[0153] Based on the above 10 descriptive tags, the Sentence-BERT model is used to convert the descriptive tags in text form into a 768-dimensional vector representation to capture its semantic information.

[0154] Action failure / abnormal flag (1 dimension):

[0155] Depending on whether the previous action failed or an exception occurred, the value is 0 (success) or 1 (failure / abnormal). Help the agent identify the results of action execution and adjust the strategy.

[0156] State vector (769 dimensions in total):

[0157] Combination method: Concatenate the descriptive label vector with the action failure / abnormal flag to form a complete state vector.

[0158] Formula: s = [v desc ,f error ]

[0159] Among them, v desc is the descriptive label vector, f error Action failure / abnormal flag.

[0160] Reward function design

[0161] In order to optimize the decision-making behavior of the agent, the system designs a reward function based on the data sufficiency evaluation results. Each descriptive label corresponds to a specific reward value, ranging from -5 to 5. The agent obtains the corresponding reward based on the label of the current state, thereby affecting the choice of subsequent actions. The specific reward function is designed as follows:

[0162] The data is seriously insufficient, further collection is recommended. (Reward value: -5)

[0163] The data is extremely limited, which may lead to significant deviations in the analysis results. (Reward value: -4)

[0164] Insufficient data, the reliability of the analysis results is low, and additional data is needed. (Reward value: -3)

[0165] The data is small, and the analysis results may not be comprehensive enough, affecting the accuracy. (Reward value: -2)

[0166] The data is average, the analysis can be performed, but the quality of the results needs to be improved. (Reward value: -1)

[0167] The data is basically sufficient to support preliminary analysis, but there is still room for improvement. (Reward value: 0)

[0168] The data is sufficient to conduct a reliable analysis, but it can be improved. (Reward value: 1)

[0169] Sufficient data to support reliable analysis. (Reward value: 2)

[0170] The data is sufficient to support high-quality analysis. (Reward value: 3)

[0171] The data is sufficient to support comprehensive and reliable high-quality analysis. (Award value: 5)

[0172] This reward function ensures that the reinforcement learning agent continuously adjusts its strategy and optimizes towards more efficient task instructions and execution plans by rewarding or penalizing the task performance in each state.

[0173] Consider the action failure / abnormal flag:

[0174] If the action fails / the exception flag is f error =1, a penalty term γ is further subtracted from the reward value.

[0175] Final reward value calculation:

[0176] R t =r t -γ×f error

[0177] Where: r t is the reward value corresponding to the descriptive label (-5 to 5), γ is the penalty coefficient for action failure, which takes a positive value (for example, 2), and f error Action failure / abnormal flag, 0 or 1.

[0178] In the reward function design, the present invention introduces a fine-tuned large language model to realize the generation of reward signals, solving the key problem of reinforcement learning application in the field of industry analysis. The data state of industry analysis is highly complex and dynamic, and it is difficult to directly construct a standard reward function. However, through the fine-tuned large model, the complex data sufficiency evaluation can be quantified into a specific reward signal. This design not only improves the accuracy and applicability of the reward function, but also breaks through the application bottleneck of reinforcement learning in industry analysis scenarios, and significantly improves the decision-making efficiency and task optimization capabilities of the intelligent agent. The method and application of fine-tuning the large model will be specifically introduced in the data sufficiency evaluation module.

[0179] Deep Q Network (DQN) Architecture and Training Process

[0180] The present invention uses a deep reinforcement learning algorithm - a deep Q network to be responsible for the decision-making of the agent. By estimating the Q value of each task instruction action in different states, the agent selects the optimal action that can maximize the reward. The following is the detailed architecture design of DQN.

[0181] Network structure design

[0182] Input layer: Input is a state vector (769 dimensions)

[0183] Descriptive tag vectorization (768 dimensions): The descriptive tags in text form are vectorized through the Sentence-BERT model to form a 768-dimensional semantic vector.

[0184] Action failure / abnormal flag (1 dimension): whether the previous action failed, the value is 0 or 1. Output:

[0185] Hidden Layer:

[0186] Hidden layer 1: 256 neurons, activation function is ReLU. It is used to extract high-dimensional features of the state vector and capture the deep relationship of semantic information.

[0187] Hidden layer 2: 128 neurons, activation function is ReLU. Its function is to further extract features and simplify the representation of high-dimensional semantic features.

[0188] Output layer: Output is Q value vector

[0189] Number of neurons: 60 (corresponding to 60 task instruction selection actions).

[0190] Function: Output the Q value prediction for each action, which represents the expected cumulative reward for executing each task instruction in the current state.

[0191] Training process

[0192] Experience replay mechanism: In order to break the time correlation and improve the stability of the model, the experience replay mechanism is used to store each interaction in the experience pool and perform random sampling during training.

[0193] Target network: The target network Q' is used to stabilize the Q value estimation. The parameters of the main network are copied to the target network every certain number of steps to prevent instability caused by frequent updates. The target network is updated every 1000 steps.

[0194] The training process of the deep Q network

[0195] 1. State acquisition: The agent obtains the current state s from the environment perception module t

[0196] 2. Action selection: Select the task instruction action a through the epsilon-greedy strategy t

[0197] Randomly select actions with probability ∈ (exploration)

[0198] The action (utilization) with the largest Q value is selected with probability 1-∈.

[0199] 3. Execute action: The agent executes the selected task instructions.

[0200] 4. Obtain rewards: Obtain rewards R from the data quality assessment module based on the task execution results t

[0201] 5. Experience storage: store the quadruple (s) of each interaction t ,a t ,R t ,s t+1 ) is stored in the experience pool.

[0202] 6. Training update: Randomly sample batches of samples from the experience pool and update the DQN model parameters by minimizing the loss function.

[0203] 7. Target network update: Update the target network Q' every certain number of steps.

[0204] As an implementation method of an embodiment of the present invention, the working process of the intelligent task execution and data processing engine is as follows:

[0205] The intelligent task execution and data processing engine is responsible for converting the task instructions received by the agent module into specific execution plans, and performs tasks including data collection, data cleaning and analysis, API calls, database retrieval, and exception handling. This module combines the exception handling mechanism to ensure efficient execution of tasks.

[0206] The main functions of this module include:

[0207] Task instruction parsing and execution: Based on the strategy instructions of the intelligent agent module, a specific execution plan is generated through the large language model, and a detailed execution plan is generated based on the dimension of the task and the strategy type (deepening or expanding).

[0208] Automatically match API and data retrieval: Automatically match the appropriate data source API or database according to different task instruction dimensions (such as market, technology, finance, etc.). Real-time call external API to obtain dynamic data, or retrieve internal database to obtain stored data.

[0209] Data processing and analysis: Clean, deduplicate and process outliers of the collected data, aggregate information in multiple dimensions, integrate text data from different data sources, and provide the text information required for subsequent tasks.

[0210] Exception handling mechanism: When the data source fails to be obtained or the task execution is interrupted, the exception handling mechanism is triggered. Through the feedback of the reinforcement learning module, the current state vector and reward function are updated, so that the intelligent agent can learn and optimize the execution strategy of subsequent tasks.

[0211] Using large language models to complete instruction parsing and execution

[0212] The task execution plan generation module uses a large language model (such as GPT-4) to convert high-level task instructions into actionable and specific plans. The functions of the large language model include:

[0213] Task instruction parsing: The task instructions received from the agent module are parsed into specific execution content by the large language model. The model generates a detailed execution plan based on the dimension of the task and the strategy type (deepening or expanding).

[0214] Example:

[0215] Enter the task command: D_T_01 (deepen the time dimension and analyze the past year)

[0216] Large model analysis output: Generate data collection requirements (market data for the past year) and automatically match appropriate analysis tools (such as regression analysis models).

[0217] Generate actionable steps: The large language model generates a series of specific execution steps based on the parsed instructions, including the type and source of data collection, the selection of analysis methods, etc.

[0218] Example:

[0219] Collect market sales data, user behavior data, and policy documents.

[0220] Perform time series analysis models to predict market trends.

[0221] Data acquisition and storage strategy

[0222] This module needs to dynamically acquire and process a large amount of external data, so it adopts a strategy that combines advance collection with real-time call, and selects the appropriate data acquisition method according to task requirements. Some commonly used data can be collected and stored in advance, while other dynamic data needs to be called in real time.

[0223] Data sources and retrieval methods

[0224] Select appropriate data sources and acquisition methods according to the task instruction dimensions (such as market, technology, finance, etc.), as shown in Table 1:

[0225] Table 1

[0226]

[0227]

[0228] Data acquisition method and storage strategy

[0229] Historical data, collected and stored in advance: The market, technology, financial and other data collected in advance by the system will be stored in the database for quick retrieval in subsequent tasks. For example, historical data on market trends, paper data, patent data, etc. These historical data are updated regularly to reduce the pressure and cost of real-time API calls.

[0230] Dynamic data, real-time through API calls: Data that changes rapidly, such as market data and financial data, is called in real time through API calls to ensure the timeliness and accuracy of the data.

[0231] Data processing and aggregation

[0232] The core purpose of the data processing and aggregation module is to collect and process relevant text data from multiple data sources, perform data cleaning and information aggregation for subsequent analysis and reasoning.

[0233] Data cleaning: After data acquisition is completed, the data processing module cleans the text data, including:

[0234] Data deduplication: Delete duplicate content to ensure the uniqueness of data.

[0235] Outlier processing: Identify and process invalid or missing data to ensure data integrity and consistency.

[0236] Data aggregation: After data cleaning, the system aggregates data from multiple data sources to ensure that all relevant text information can be integrated according to different dimensions. The purpose of information aggregation is to combine data from different sources to provide a comprehensive perspective.

[0237] Exception handling mechanism

[0238] The exception handling mechanism is designed to ensure that when a data source fails to be obtained or task execution is interrupted, the system can respond in time, and learn and adaptively adjust through the reinforcement learning module. When data acquisition exceptions occur during task execution (such as API call failure, data source response timeout, etc.), the system can quickly handle the exception. Through the reinforcement learning module, the exception handling will update the current state vector and feedback the abnormal situation to the system, so that the agent can learn how to avoid similar problems in subsequent tasks.

[0239] As an implementation method of an embodiment of the present invention, the working process of the data sufficiency evaluation module is as follows:

[0240] The core function of this module is to evaluate whether the data collected by the agent is sufficient to support the analysis requirements of the current task. This module uses the semantic analysis capabilities of the fine-tuned large language model to aggregate and comprehensively evaluate text data of different dimensions, judge its adequacy in a certain task, and generate corresponding descriptive labels. This evaluation result will also be fed back to the reinforcement learning module to provide environmental feedback to the agent, helping it to judge whether the current data is sufficient to support high-quality analysis and decide whether it is necessary to further optimize task instructions or data collection strategies.

[0241] Fine-tune the large language model to create a large model for data sufficiency assessment: The main goal of fine-tuning the large model is to enable it to better understand data in specific fields, especially the ability to evaluate text in industry analysis scenarios. The fine-tuning process is based on a pre-trained large language model (such as GPT-3.5-turbo, GPT-4 or Zhipu GLM-4), and adjusts it through annotated datasets to make the model more focused on data sufficiency assessment tasks. Fine-tuning the large model includes data collection, data aggregation, dataset annotation, fine-tuning training and model verification steps:

[0242] Fine-tuning data collection: The data required for fine-tuning mainly comes from the following types of text data

[0243] Industry reports: Contains market size, technology trends, competition analysis, policy impact, etc., and comes from market research companies (such as Statista, IDC) and industry databases.

[0244] Patent data: such as technical patent abstracts and description documents, from patent databases (such as Google Patents, WIPO).

[0245] Scientific literature: scientific literature and paper data from databases such as IEEE, arXiv, etc.

[0246] Financial reports: Financial performance, investment and financing data of companies from financial databases (such as PitchBook and Crunchbase).

[0247] Policy and regulatory data: relevant policy documents and regulatory interpretations from government websites and policy databases.

[0248] Data aggregation: Aggregate data from multiple dimensions such as market, technology, finance, and policy to form multiple industry analysis text fragments with different data sufficiency levels, so as to build a complete fine-tuning dataset. The following is the strategy for data aggregation:

[0249] The data is divided into multiple segments. Some text segments have relatively complete information, covering data from multiple dimensions and supporting comprehensive industry analysis. Other text segments have insufficient data and may lack support from certain dimensions (such as policies or technologies), resulting in incomplete analysis segments. In this way, the model can encounter various data segments of different sufficiency during the fine-tuning process, thereby learning how to effectively score and generate labels for texts of different data qualities.

[0250] Annotated dataset:

[0251] Text data sufficiency annotation: Through the process of data aggregation, a variety of different data fragments are generated, some of which are complete and some are insufficient.

[0252] Each piece of text data will be labeled according to its adequacy in industry analysis (including coverage, granularity, and relevance), and divided into 10 levels (such as "the data is seriously insufficient, and further collection is recommended", "the data is sufficient to support reliable analysis", etc.), indicating whether the data is sufficient to support task analysis.

[0253] Fine-tuning training and effect verification

[0254] Based on the annotated dataset, the GPT-3.5-turbo pre-trained model is fine-tuned to focus on data sufficiency assessment in industry analysis tasks, allowing the model to generate corresponding descriptive labels based on the input text to evaluate the sufficiency of the data.

[0255] A 20% validation dataset is used to test the performance of the model to ensure that the model can accurately assess data adequacy in different industry analysis scenarios.

[0256] The fine-tuned large language model is named “data sufficiency assessment large model”.

[0257] Data adequacy assessment and result feedback

[0258] Use the fine-tuned large language model (data sufficiency assessment large model) to evaluate the data output by the intelligent task execution and data processing engine to generate descriptive labels.

[0259] The data sufficiency assessment results are passed to the reinforcement learning-driven intelligent agent engine to provide the agent with environmental feedback to help it determine whether the current data is sufficient to support high-quality analysis. The agent continuously adjusts its strategy based on the feedback from the data sufficiency assessment to achieve the best analysis results.

[0260] As an implementation of an embodiment of the present invention, the analysis result output module is specifically:

[0261] The main function of the analysis result output module is to generate high-quality industry analysis results based on the optimized data obtained by the agent through previous instruction deepening and expansion. When the data sufficiency assessment module determines that the current data has reached the highest level (that is, "the data is very sufficient to support comprehensive and reliable high-quality analysis"), it means that the collected data has been optimized through the optimal decision path and has completeness and coherence. At this time, the system calls a large language model (such as GPT-4, Zhipu GLM-4, etc.) to analyze and reason about these data to generate the final industry analysis report. Since each step of the data collection strategy is optimized by the agent module through the deepening or expansion of reinforcement learning instructions, the data collection process forms a "thinking chain" attribute. This means that each data decision is based on the results of the previous optimization step, ensuring the logic and depth of the analysis process.

[0262] Input data:

[0263] Data source: Data obtained through reinforcement learning-driven intelligent agent modules using instructions to deepen or expand strategy optimization, covering multiple dimensions such as market, technology, finance, and policy.

[0264] Data characteristics: The data integrity is extremely high, covering multiple key dimensions required for industry analysis. Each step of data collection decision is based on the result of the reinforcement learning decision optimization of the previous step, and has obvious thinking chain attributes. The data is preprocessed and cleaned to ensure that it can be directly used for reasoning and generating analysis results of large language models.

[0265] Industry analysis generation

[0266] Use large language models (such as GPT-4) to perform reasoning and analysis based on the optimized data collected by the agent through reinforcement learning to generate complete industry reports.

[0267] Thinking chain reasoning: Since the reinforcement learning of the intelligent agent module continuously deepens or expands the optimal instructions during the task execution, each batch of data collected by the system is based on the result of the previous decision. This thinking chain data collection path allows the large language model to perform step-by-step reasoning based on a coherent data flow, ensuring clear analysis logic and in-depth analysis.

[0268] Output

[0269] Structured industry analysis reports include comprehensive analysis of market status, competition situation, technology trends, policy impact, etc.

[0270] By combining reinforcement learning algorithms and large language models, this invention not only optimizes the task execution efficiency and analysis accuracy in the field of industry analysis, but also gives the system the ability of "thinking chain", enabling it to have the deep analysis function of cross-dimensional and dynamic adjustment of task instructions. It has universality and can be extended to other fields, such as enterprise analysis and industry analysis. The specific technical effects are as follows:

[0271] 1. Significantly improved task execution efficiency

[0272] The intelligent agent module driven by reinforcement learning can dynamically optimize task instructions and adjust execution paths based on feedback during task execution, reducing manual intervention and improving task processing efficiency. Especially for complex industry analysis tasks, it automatically processes multi-dimensional data analysis steps, significantly reducing the manual intervention time required for traditional analysis.

[0273] 2. Improved accuracy and depth of analysis results

[0274] By introducing a data sufficiency assessment module and using the assessment results as the basis for the reward function of reinforcement learning, the quality of the collected data can be assessed in real time, and task instructions can be adaptively adjusted to ensure that the data based on the analysis has sufficient coverage and depth. This mechanism ensures the accuracy and high quality of the system's analysis results in complex, multi-dimensional task environments.

[0275] 3. Enhanced flexibility in task processing

[0276] Through the task instruction deepening and expansion module, it is possible to flexibly switch analysis dimensions and jump between multiple dimensions to meet the complex analysis needs of users. This function gives the system the ability to conduct in-depth analysis from a single dimension (such as market or technology) to cross-dimensional analysis (such as expanding from market to finance and technology), which is particularly suitable for multi-field tasks such as industry analysis, market trend forecasting, and enterprise data analysis.

[0277] 4. Empower industry analysis with “thinking chain” capabilities

[0278] Through multiple rounds of feedback from reinforcement learning, each level of task decision-making is closely related to the execution of the previous step, forming a "thinking chain". This allows industry analysis to not only stay on the surface, but also make the analysis logic more rigorous through deep thinking chain reasoning, and the coherence and depth of the analysis report are also significantly improved. This "thinking chain" capability can be extended to other fields, such as market forecasting, enterprise decision support, etc., to further improve the level of intelligence in complex task processing.

[0279] Embodiment 2:

[0280] The embodiment of the present invention further provides an intelligent task optimization device based on reinforcement learning, comprising:

[0281] A user interaction interface module, used to obtain initial task instructions input by the user;

[0282] The task instruction deepening and expansion module is used to deepen and expand the initial task instructions;

[0283] A reinforcement learning-driven agent engine for optimizing and adjusting task instructions and data collection strategies using reinforcement learning algorithms;

[0284] Intelligent task execution and data processing engine, which is used to convert task instructions into specific execution plans and perform tasks including data collection, data cleaning and analysis, API calls, database retrieval and exception handling;

[0285] The data sufficiency assessment module is used to evaluate whether the data collected by the agent is sufficient to support the analysis requirements of the current task;

[0286] The analysis result output module is used to generate industry analysis results based on the optimization data obtained through instruction deepening and expansion.

[0287] As an implementation method of an embodiment of the present invention, a reinforcement learning algorithm includes: a deep Q network DQN, a policy gradient method, and an Actor-Critic; the reinforcement learning-driven intelligent agent engine enables the intelligent agent to perceive the current task state by continuously interacting with the environment, select the optimal strategy to execute task instructions, obtain rewards based on the execution results, and optimize subsequent task instructions and data collection strategies.

[0288] As an implementation method of an embodiment of the present invention, the data sufficiency assessment module uses a fine-tuned large language model to perform a comprehensive assessment on aggregated text data of different dimensions, determines its sufficiency in a certain task, and generates corresponding descriptive labels.

[0289] Embodiment 3:

[0290] An embodiment of the present invention also provides an intelligent task optimization system based on reinforcement learning, comprising: a memory and a processor, wherein the memory stores a computer program run by the processor, and the computer program executes an intelligent task optimization method based on reinforcement learning when run by the processor.

[0291] Embodiment 4:

[0292] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes an intelligent task optimization method based on reinforcement learning when running.

[0293] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An intelligent task optimization method based on reinforcement learning, characterized in that: The following steps are involved: Step S1, obtaining an initial task instruction input by a user; Step S2, deepening and expanding the initial task instructions; Step S3: using a reinforcement learning algorithm to optimize and adjust task instructions and data collection strategies; Step S4: Convert the task instructions into a specific execution plan, and execute tasks including data collection, data cleaning and analysis, API calls, database retrieval, and exception handling; Step S5: Evaluate whether the data collected by the agent is sufficient to support the analysis requirements of the current task; Step S6: Generate industry analysis results based on the optimization data obtained through instruction deepening and expansion.

2. The intelligent task optimization method based on reinforcement learning according to claim 1, characterized in that: In step S3, the reinforcement learning algorithm includes: deep Q network DQN, policy gradient method, Actor-Critic; the intelligent agent continuously interacts with the environment, perceives the current task state, selects the optimal strategy to execute task instructions, obtains rewards based on the execution results, and optimizes subsequent task instructions and data collection strategies.

3. The intelligent task optimization method based on reinforcement learning according to claim 2, characterized in that: In step S5, the fine-tuned large language model is used to perform a comprehensive evaluation on the aggregated text data of different dimensions to determine its adequacy in a certain task and generate corresponding descriptive labels.

4. An intelligent task optimization device based on reinforcement learning, characterized in that: include: A user interaction interface module, used to obtain initial task instructions input by the user; The task instruction deepening and expansion module is used to deepen and expand the initial task instructions; A reinforcement learning-driven agent engine for optimizing and adjusting task instructions and data collection strategies using reinforcement learning algorithms; Intelligent task execution and data processing engine, which is used to convert task instructions into specific execution plans and perform tasks including data collection, data cleaning and analysis, API calls, database retrieval and exception handling; The data sufficiency assessment module is used to evaluate whether the data collected by the agent is sufficient to support the analysis requirements of the current task; The analysis result output module is used to generate industry analysis results based on the optimization data obtained through instruction deepening and expansion.

5. The intelligent task optimization device based on reinforcement learning according to claim 4, characterized in that: Reinforcement learning algorithms include: deep Q network DQN, policy gradient method, and Actor-Critic; the reinforcement learning-driven intelligent agent engine enables the intelligent agent to perceive the current task status through continuous interaction with the environment, select the optimal strategy to execute task instructions, obtain rewards based on the execution results, and optimize subsequent task instructions and data collection strategies.

6. The intelligent task optimization device based on reinforcement learning according to claim 5, characterized in that: The data sufficiency assessment module uses a fine-tuned large language model to perform a comprehensive assessment on aggregated text data of different dimensions, determine its sufficiency in a certain task, and generate corresponding descriptive labels.

7. An intelligent task optimization system based on reinforcement learning, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the intelligent task optimization method based on reinforcement learning as described in any one of claims 1 to 3 is executed.

8. A storage medium, characterized in that: The storage medium stores a computer program, which, when running, executes the intelligent task optimization method based on reinforcement learning as described in any one of claims 1 to 3.

Images