Intelligent agent training method and system based on continuous learning

Abstract

The invention belongs to the technical field of agent training, and discloses an agent training method and system based on continuous learning. The method comprises the following steps: defining a meta-learning target and a plurality of preset tasks of an intelligent agent, and constructing an experience playback pool and a data feature extraction model; carrying out feature extraction and clustering on a plurality of historical data; optimizing the initial intelligent agent parameters of the intelligent agent by using a swarm intelligence optimization algorithm; pre-training the initialized intelligent agent, and storing a plurality of historical meta learning experiences into an experience playback pool; performing feature extraction on the plurality of real-time data, and confirming a plurality of new tasks; and adjusting the loss function, and training the pre-trained agent under a plurality of new tasks to obtain a final agent. According to the method, the problems of disastrous forgetting, insufficient generalization ability, mark data dependence, low training efficiency and difficulty in processing a non-static environment in the prior art are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent agent training, and specifically relates to an intelligent agent training method and system based on continuous learning. Background Art

[0002] An intelligent agent is an entity that has a certain degree of autonomy and can achieve a certain goal by perceiving its environment and taking actions. An intelligent agent can be a software program, a hardware device, or a combination of software and hardware. Intelligent agent technology has played an important role in promoting social progress, improving production efficiency, and improving people's lives. Intelligent agent training is an important branch of the field of artificial intelligence. Its main purpose is to enable intelligent agents to acquire certain strategies through learning in the process of interacting with the environment in order to complete specific tasks.

[0003] The existing intelligent agent training technology has the following defects: 1) Catastrophic forgetting: Existing agent training methods often suffer from catastrophic forgetting when learning new tasks continuously, that is, forgetting previously learned experience in the process of learning new tasks. This is because traditional training methods are usually performed on fixed data sets and do not take into account the long-term retention of experience; 2) Insufficient generalization ability: Many agent training methods have limited generalization ability to new tasks or unfamiliar environments, which means that the agent performs well in the training environment, but may encounter performance degradation in actual applications; 3) Dependence on large amounts of labeled data: Existing deep learning models usually require a large amount of labeled data for training. However, in some fields, obtaining a large amount of high-quality labeled data is time-consuming and expensive, which limits the application of intelligent agents in these fields; 4) Inefficient training: The training process of an agent often requires a lot of computing resources and time. For complex tasks, the training process may take weeks or even months, which is impractical in practical applications; 5) Difficulty in handling non-static environments: In the real world, the environment is often dynamically changing. Existing agent training methods often assume that the environment is static, which makes it difficult for agents to adapt to changes in the environment. Summary of the invention

[0004] In order to solve the problems of catastrophic forgetting, insufficient generalization ability, labeled data dependence, low training efficiency and difficulty in handling non-static environments in the prior art, the present invention aims to provide an intelligent agent training method and system based on continuous learning.

[0005] The technical solution adopted by the present invention is: A method for training an intelligent agent based on continuous learning, comprising the following steps: Use the meta-learning algorithm to define the meta-learning objectives and several preset tasks of the agent, use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool, and use the deep learning algorithm to build a data feature extraction model; Collecting a number of historical data, using a data feature extraction model to extract features from the number of historical data, obtaining a number of historical data features, and clustering the number of historical data features with preset tasks as clustering objects, obtaining a number of historical data features with different preset tasks as first clustering centers; Taking minimizing the mean square error as the optimization goal, the swarm intelligence optimization algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent, and the agent is initialized according to the optimal initial agent parameters to obtain the initialized agent; Under several preset tasks, use several corresponding historical data features to pre-train the initialized agent, obtain the pre-trained agent and several historical meta-learning experiences, and store the several historical meta-learning experiences in the experience replay pool; Collecting a number of real-time data, using a data feature extraction model to extract features from the number of real-time data, obtaining a number of real-time data features, and confirming a number of new tasks based on the number of real-time data features and a number of first clustering centers of different preset tasks; The elastic weight connection mechanism of the continuous learning algorithm is used to adjust the loss function to obtain the adjusted loss function. Under several new tasks, based on the adjusted loss function and the experience replay pool, the corresponding several real-time data features are used to train the pre-trained agent to obtain the final agent.

[0006] Furthermore, a meta-learning algorithm is used to define meta-learning objectives and several preset tasks of the agent, an experience replay mechanism of the continuous learning algorithm is used to build an experience replay pool, and a deep learning algorithm is used to build a data feature extraction model, including the following steps: With the goal of enabling the agent to quickly adapt to new tasks, the meta-learning goal of the meta-learning algorithm is defined, and the preset functions of the agent are split to define several preset tasks of the meta-learning algorithm. Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool and define the form of meta-learning experience stored in the experience replay pool; Based on the recognizable feature format of the intelligent agent, a deep learning algorithm is used to build an initial data feature extraction model; A number of training data are collected, and an initial data feature extraction model is trained based on the training data to obtain a final data feature extraction model.

[0007] Further, a number of historical data are collected, and a data feature extraction model is used to extract features from the number of historical data to obtain a number of historical data features, and the number of historical data features are clustered with preset tasks as clustering objects to obtain a number of historical data features with different preset tasks as first clustering centers, including the following steps: Collecting a number of historical data of the intelligent agent in different preset tasks, and preprocessing the number of historical data to obtain a number of preprocessed historical data; Use the data feature extraction model to extract features from some pre-processed historical data to obtain some historical data features; Taking the preset tasks as clustering objects, using the FCM clustering algorithm, clustering several historical data features to obtain several first clustering centers and corresponding first clustering clusters; A corresponding preset task is set for each first cluster center, and the preset task of the first cluster center is diffused to a number of historical data features of the corresponding first cluster clusters, so as to obtain a number of historical data features with different preset tasks as the first cluster centers.

[0008] Furthermore, taking the preset task as the clustering object, using the FCM clustering algorithm, clustering is performed on a number of historical data features to obtain a number of first clustering centers and corresponding first clustering clusters, including the following steps: According to a number of preset tasks, clustering parameters of the FCM clustering algorithm are set, and according to the clustering parameters, the FCM clustering algorithm is used to initialize the clustering centers to obtain a number of initial first clustering centers; the clustering parameters include a fuzzy factor and the total number of first clustering centers; According to the Euclidean distance between each historical data feature and a number of initial first cluster centers, a corresponding historical fuzzy membership is set for each initial first cluster center; According to the historical fuzzy membership and clustering parameters, several initial first clustering centers are updated to obtain several updated first clustering centers; Use the Lagrange multiplier method to calculate the merge function, and obtain the merge function value and the merge function change value. If the merge function value is greater than the function threshold, or the merge function change value is greater than the change value threshold, then continue to update the first cluster center, otherwise, use the current first cluster center as the final first cluster center; A number of historical data features are divided into final first cluster centers with the shortest Euclidean distance, to obtain a number of final first cluster centers and corresponding first cluster clusters; the first cluster clusters include a number of historical data features.

[0009] Furthermore, taking minimizing the mean square error as the optimization goal, the swarm intelligence optimization algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent, and the agent is initialized according to the optimal initial agent parameters to obtain the initialized agent, including the following steps: Taking minimizing the mean square error as the optimization goal, the fitness function of the swarm intelligence optimization algorithm is set, and the individual encoding format of the swarm intelligence optimization algorithm is set according to the initial agent parameters of the agent; Set the algorithm parameters of the swarm intelligence optimization algorithm, and use the swarm intelligence optimization algorithm to iteratively search for the best individual based on the fitness function, individual encoding format and algorithm parameters; The individual encoding vector of the optimal individual is decoded to obtain the optimal initial intelligent agent parameters of the intelligent agent, and the intelligent agent is initialized according to the optimal initial intelligent agent parameters to obtain the initialized intelligent agent.

[0010] Furthermore, under a number of preset tasks, the initialized agent is pre-trained using a number of corresponding historical data features to obtain a pre-trained agent and a number of historical meta-learning experiences, and the number of historical meta-learning experiences is stored in an experience replay pool, including the following steps: Under the current preset task, use the corresponding historical data features to pre-train the initialized agent to obtain some historical meta-learning experience of the current preset task; Traverse all preset tasks, pre-train the initialized agent, and obtain some historical meta-learning experience of the pre-trained agent and all preset tasks; Several historical meta-learning experiences of all preset tasks are stored in the experience replay pool; the historical meta-learning experience contains the corresponding historical data features.

[0011] Further, a plurality of real-time data are collected, and a data feature extraction model is used to extract features from the plurality of real-time data to obtain a plurality of real-time data features, and a plurality of new tasks are confirmed according to the plurality of real-time data features and a plurality of first clustering centers of different preset tasks, including the following steps: Collecting a number of real-time data, and preprocessing the real-time data to obtain a number of preprocessed real-time data; A data feature extraction model is obtained to extract features from a number of preprocessed real-time data to obtain a number of real-time data features; Obtain the Euclidean distance between each real-time data feature and the first cluster center, and obtain the corresponding real-time fuzzy membership according to the Euclidean distance; According to a number of real-time data features and the real-time fuzzy membership of the first cluster centers, a number of first cluster centers are updated to obtain a number of corresponding second cluster centers; Comparing the plurality of second cluster centers with the plurality of first cluster centers to obtain a plurality of third cluster centers other than the preset tasks, and setting a corresponding new task for each third cluster center; According to the Euclidean distance between each real-time data feature and the third cluster centers, the real-time data features are divided into corresponding third cluster centers to obtain the second cluster clusters of the third cluster centers; The new task at the third cluster center is diffused to a number of real-time data features of the corresponding second cluster clusters, so as to obtain a number of real-time data features with different new tasks as the third cluster center.

[0012] Furthermore, the elastic weight connection mechanism of the continuous learning algorithm is used to adjust the loss function to obtain an adjusted loss function, and under several new tasks, based on the adjusted loss function and the experience replay pool, the pre-trained agent is trained using several corresponding real-time data features to obtain the final agent, including the following steps: Obtain the loss function of the pre-trained agent and obtain several key weights of the pre-trained agent under each preset task; Obtaining importance measures of several key weights, generating penalty terms according to the importance measures, and adjusting the loss function according to the penalty terms to obtain an adjusted loss function; Under several new tasks, based on the adjusted loss function and experience replay pool, the pre-trained agent is trained using several corresponding real-time data features to obtain the final agent.

[0013] Furthermore, under several new tasks, based on the adjusted loss function and the experience replay pool, the pre-trained agent is trained using several corresponding real-time data features to obtain the final agent, including the following steps: Under the current new task, use the corresponding real-time data features to train the pre-trained agent, obtain some real-time meta-learning experiences of the new task, and store some real-time meta-learning experiences in the experience replay pool; the real-time meta-learning experience contains the corresponding real-time data features; Randomly extract several historical meta-learning experiences of different preset tasks in the experience replay pool, and mix several historical meta-learning experiences with several real-time meta-learning experiences to obtain several mixed meta-learning experiences and corresponding mixed data features; According to several hybrid meta-learning experiences, use several corresponding hybrid data features to train the pre-trained agent, and use the adjusted loss function to obtain the loss value of the agent training; If the loss value is always less than the loss value threshold, the agent training for the next new task is carried out, and all new tasks are traversed until the agent training for all new tasks is completed to obtain the final agent; If the loss value is greater than or equal to the loss value threshold, several historical meta-learning experiences of different preset tasks are re-extracted from the experience replay pool, and the agent training of the current new task is returned.

[0014] An agent training system based on continuous learning is used to implement an agent training method. The system includes an algorithm initialization unit, a feature clustering unit, an agent initialization unit, an agent pre-training unit, a new task confirmation unit and an agent training unit which are connected in sequence.

[0015] The beneficial effects of the present invention are: The present invention provides an intelligent agent training method and system based on continuous learning. By using the experience replay mechanism of the continuous learning algorithm, the intelligent agent effectively reduces the forgetting of old experience when learning new tasks, improves the persistence and stability of the intelligent agent in multi-task learning, and reduces catastrophic forgetting; through the meta-learning algorithm and the data feature extraction model, the intelligent agent can be better generalized to unseen tasks and environments, improves the adaptability of the intelligent agent in complex and changeable environments, and enhances the generalization ability; the adopted feature extraction and clustering methods can effectively utilize historical data, reduce the dependence on a large amount of new labeled data, reduce the cost of data collection and annotation, and reduce the dependence on a large amount of labeled data; through the use of the swarm intelligence optimization algorithm and the hybrid meta-learning experience, the training process of the intelligent agent is accelerated, the required computing resources and time are reduced, so that the intelligent agent can adapt to new tasks more quickly and improve the training efficiency; it can handle dynamically changing environments, and through the collection of real-time data and task confirmation, the intelligent agent can timely adjust its behavior strategy to adapt to environmental changes and non-static environments; the meta-learning algorithm used can effectively handle complex tasks, and by mixing the meta-learning experience of different tasks, the learning efficiency of the intelligent agent on new tasks is improved.

[0016] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the intelligent agent training method based on continuous learning in the present invention.

[0018] Figure 2 It is a structural block diagram of the intelligent agent training system based on continuous learning in the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments.

[0020] Embodiment 1: like Figure 1 As shown, this embodiment provides an agent training method based on continuous learning, comprising the following steps: S1: Use the meta-learning algorithm to define the meta-learning objectives and several preset tasks of the agent, use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool, and use the deep learning algorithm to build a data feature extraction model, including the following steps: S1-1: With the goal of enabling the agent to quickly adapt to new tasks, define the meta-learning goal of the meta-learning algorithm, split the preset functions of the agent, and define several preset tasks of the meta-learning algorithm; In addition to the ability to quickly adapt to new tasks as a meta-learning goal in this embodiment, the ability to quickly adapt to new tasks: the agent should be able to adapt to changes in task distribution and maintain good performance even when task conditions change; other meta-learning goals are also included, which are set according to specific usage requirements, including: Fast convergence: The agent should be able to learn quickly and converge to an effective solution with very few training examples; High generalization ability: the agent's performance on new tasks should be close to or better than traditional models trained on large amounts of data; Low sample complexity: The agent should be able to learn from a limited number of samples, reducing the reliance on large amounts of data; Robustness: The agent should be robust to noise and outliers, and its performance should not be significantly affected by small fluctuations in the data. Cross-domain learning: The intelligent agent should be able to transfer the experience learned in one domain to another domain to achieve cross-domain learning; The preset tasks of the meta-learning algorithm can be a series of related tasks implemented by the agent when it involves similar data types or problem domains, including data processing, data analysis, data compression and other tasks, or they can be different but related environments or problems with similar structures, including image data classification, sequence data classification, etc. In traditional agent training, the agent parameters usually start from random initialization. However, in meta-learning, the agent learns a set of optimized parameter initializations by training on a series of tasks. These initialization parameters can provide a good starting point on new, unseen tasks. S1-2: Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool and define the form of meta-learning experience stored in the experience replay pool; The experience replay pool is a database that stores previous task learning experiences, including parameter settings, model status, learning strategies, etc. Joining the experience pool allows the agent to store and reuse past experiences to improve learning efficiency; The experience replay pool is used to store the agent's interaction experience in the environment. The meta-learning experience of these experiences is usually stored in the form of (s, a, r, s'), where s is the current state, the current agent parameters of the agent, a is the action taken, including agent parameter adjustment, deletion, addition and other actions, r is the reward obtained, which is used to characterize the influence of the action on the state, and s' is the next state, that is, the state after the agent parameters are adjusted during the training process; Since the data in the experience replay pool is randomly drawn, it helps to break the correlation between consecutive experiences, thereby reducing the variance in agent training. By reusing experience, the agent can learn more from limited experience, which is especially useful in new tasks or when samples are scarce. Experience replay helps stabilize the learning process and reduce fluctuations during training. Regularly check the size of the experience replay pool. If it exceeds the preset capacity, remove some experience according to a certain strategy (such as priority sampling, longest unvisited). S1-3: Based on the recognizable feature format of the agent, use the deep learning algorithm to build an initial data feature extraction model; The deep learning algorithm used is determined by the data type input by the agent. If the input data is a plain text sequence or numerical format, the Long Short-Term Memory (LSTM) network is used as the model algorithm of the data feature extraction model. If the input data is image data, the Convolutional Neural Networks (CNN) is used as the model algorithm of the data feature extraction model. If the input data is multimodal data, the fusion algorithm CNN-LSTM is used as the model algorithm of the data feature extraction model. S1-4: Collect some training data, and train the initial data feature extraction model according to the training data to obtain the final data feature extraction model; S2: Collect some historical data, use the data feature extraction model to extract features from the historical data, obtain some historical data features, and cluster the historical data features with the preset tasks as clustering objects to obtain some historical data features with different preset tasks as the first clustering centers, including the following steps: S2-1: Collecting a number of historical data of the agent in different preset tasks, and preprocessing the historical data to obtain a number of preprocessed historical data; Preprocessing includes data cleaning, format conversion, and normalization. While eliminating duplicate and erroneous data, it also performs standardized conversion, eliminates magnitude differences and noise, and converts the data into a format that can be recognized by subsequent models. S2-2: Use the data feature extraction model to extract features from some preprocessed historical data to obtain some historical data features; S2-3: Taking the preset task as the clustering object, using the Fuzzy C-mean (FCM) clustering algorithm to cluster several historical data features, and obtaining several first clustering centers and corresponding first clustering clusters, including the following steps: S2-3-1: according to a number of preset tasks, set the clustering parameters of the FCM clustering algorithm, and according to the clustering parameters, use the FCM clustering algorithm to initialize the clustering centers to obtain a number of initial first clustering centers; the clustering parameters include a fuzzy factor and the total number of first clustering centers; S2-3-2: according to the Euclidean distance between each historical data feature and a number of initial first cluster centers, a corresponding historical fuzzy membership is set for each initial first cluster center; The formula is:

[0021] In the formula, For the i Historical data characteristics and j The Euclidean distance of the initial first cluster center; For the Historical data characteristics; For the j The initial first cluster center; is the data characteristic indicator; is the indicator of the first cluster center; S2-3-3: updating a number of initial first cluster centers according to the historical fuzzy membership and clustering parameters to obtain a number of updated first cluster centers; The formula is:

[0022] In the formula, For the j Updated first cluster center; is the fuzzy factor; is the data characteristic indicator; is the total number of data; is the indicator of the first cluster center; For the Historical data characteristics; For the i The first cluster center belongs to The updated historical fuzzy membership of the first cluster center;

[0023] In the formula, For the Real-time gaming market fusion feature vector; is the data characteristic indicator; All are indicators of the first cluster center; is the total number of cluster centers; For the i Historical data features to j , k Updated distance of the first cluster center; For the i The first cluster center belongs to The updated historical fuzzy membership of the cluster centers; S5-4: Repeat the above steps to obtain several final cluster centers, divide the real-time game market fusion feature vector into the final cluster center with the closest Euclidean distance, and obtain several cluster clusters containing real-time prediction analysis labels; S2-3-4: Use the Lagrange multiplier method to calculate the merge function, and obtain the merge function value and the merge function change value. If the merge function value is greater than the function threshold, or the merge function change value is greater than the change value threshold, then continue to update the first cluster center, otherwise, use the current first cluster center as the final first cluster center; The formula is:

[0024] In the formula, For the t , t -1 iteration of the combined function value of the Lagrange multiplier method; is the corresponding change value; For the i Characteristic parameters; t is the indicator of the number of iterations; For the i The historical data characteristics belong to The updated historical fuzzy membership of the first cluster center; is the fuzzy factor; is the data indicator; is the total number of data; is the indicator of the first cluster center; is the total number of cluster centers; For the i Historical data features to j Updated Euclidean distance of the first cluster center; S2-3-5: Divide the several historical data features into the final first cluster centers with the shortest Euclidean distance, and obtain several final first cluster centers and corresponding first cluster clusters; the first cluster clusters include the several historical data features; S2-4: setting a corresponding preset task for each first cluster center, spreading the preset task of the first cluster center to a number of historical data features of the corresponding first cluster cluster, and obtaining a number of historical data features with different preset tasks as the first cluster center; S3: Taking minimizing the mean square error as the optimization goal, the Improved Crested Porcupine Optimizer (ICPO) algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent, and the agent is initialized according to the optimal initial agent parameters to obtain the initialized agent, including the following steps: S3-1: Taking minimizing the mean square error as the optimization goal, set the fitness function of the swarm intelligence optimization algorithm, and set the individual encoding format of the swarm intelligence optimization algorithm according to the initial agent parameters of the agent; S3-2: Set the algorithm parameters of the swarm intelligence optimization algorithm, and use the swarm intelligence optimization algorithm to iteratively search for the best result according to the fitness function, individual encoding format and algorithm parameters, and retain the best individual, including the following steps: S3-2-1: Set the algorithm parameters of the ICPO algorithm, which include the ICPO population parameters and the maximum number of iterations of the ICPO algorithm; S3-2-2: According to the fitness function, individual encoding format and ICPO population parameters, the Circle chaotic mapping sequence is used to initialize the ICPO population to obtain an initial ICPO population containing several initial ICPO individuals; The formula is:

[0025] In the formula, is the initial ICPO individual of the Circle chaos map, corresponding to an initial solution; is the randomly generated initial ICPO individual; It is the ICPO individual indicator; S3-2-3: Introduce a cyclic population reduction mechanism to limit the number of individuals in the ICPO population parameters and obtain updated ICPO population parameters for the next iteration; The formula is:

[0026] In the formula, For the The number of individuals in the ICPO population parameter of the iteration; For the The number of individuals in the ICPO population parameter of the iteration; is the minimum number of individuals in the ICPO population parameter; Evaluate arguments for functions; Evaluate loop parameters for a function; Evaluate loop parameters for the maximum function; t is the indicator of the number of iterations; S3-2-4: Calculate the initial fitness value of the initial ICPO individuals in the initial ICPO population according to the fitness function; The formula is:

[0027] In the formula, is the fitness function; is the mean square error function; S3-2-5: According to the initial fitness value and the updated ICPO population parameters, the first defense strategy, the second defense strategy, the third defense strategy and the fourth defense strategy are used to update the initial ICPO population to obtain an updated ICPO population; The formula for the first defense strategy is:

[0028] In the formula, For the updated ICPO individuals within the first defense range; The initial ICPO individual within the first defense range; is a random number based on normal distribution; is a random value in the interval [0,1]; It is the optimal solution within the first defense range; is the vector generated between the true optimal solution within the first defense range and the optimal solution randomly selected from the ICPO population; It is the ICPO individual indicator; is the iteration indicator; The formula for the second defense strategy is:

[0029] In the formula, For the updated ICPO individuals within the second defense range; The initial ICPO individual within the second defense range; is the search upper limit vector of the second defense range; is a random value in the interval [0,1]; Respectively Initial ICPO individuals; All are [1, ]; is the vector generated between the true optimal solution within the second defense range and the optimal solution randomly selected from the ICPO population; The formula for the third defense strategy is:

[0030] In the formula, For the updated ICPO individuals within the third defense range; The initial ICPO individual within the third defense range; is the search upper limit vector of the third defense range; Respectively Initial ICPO individuals; is [1, ]; The odor diffusion factor defined for the fitness function; It is a defense factor; Control parameters for search direction; The formula for the fourth defense strategy is:

[0031] In the formula, For the updated ICPO individuals within the fourth defense range; The initial ICPO individual within the fourth defense range; It is the optimal solution within the fourth defense range; All are random values ​​in the interval [0,1]; It is a defense factor; Control parameters for search direction; is the average force affecting the search direction; is the convergence speed factor;

[0032] In the formula, is the convergence speed factor; tanh(.) is the hyperbolic tangent function; a max , a min are the maximum and minimum values ​​of the convergence factor respectively; λ is the deceleration rate parameter, k is the decreasing cycle parameter, λ =-2 π , k = π ; is the iteration indicator; is the maximum number of iterations; S3-2-6: Use the dynamic reverse learning algorithm to perform dynamic reverse learning on the updated ICPO population to generate a dynamic reverse ICPO population; The formula is:

[0033] In the formula, It is a dynamically reversed ICPO individual; is the decreasing inertia coefficient; are the maximum and minimum values ​​of the vector space respectively; For the updated ICPO individuals; S3-2-7: According to the fitness function, calculate the fitness values ​​of all ICPO individuals in the updated ICPO population and the dynamically reversed ICPO population, take the ICPO individual with the minimum fitness value as the optimal individual, and retain the optimal individual; S3-3: Decode the individual encoding vector of the optimal individual to obtain the optimal initial agent parameters of the agent, and initialize the agent according to the optimal initial agent parameters to obtain the initialized agent; Use swarm intelligence optimization algorithm to optimize the initial agent parameters of the agent to adapt to the new task and keep the agent parameters from overfitting the new task during continuous learning. The initial agent parameters include the initial weights of the neural network, the initial parameters of the policy network, the initial parameters of the value function, and the initial learning rate of the agent. S4: Under a number of preset tasks, using a number of corresponding historical data features, pre-training the initialized agent, obtaining a pre-trained agent and a number of historical meta-learning experiences, and storing the several historical meta-learning experiences in an experience replay pool, including the following steps: S5-1: Under the current preset task, use the corresponding historical data features to pre-train the initialized agent to obtain some historical meta-learning experience of the current preset task; S5-2: Traverse all preset tasks, pre-train the initialized agent, and obtain some historical meta-learning experiences of the pre-trained agent and all preset tasks; Train the agent on a set of predefined tasks to learn common features and policies, using an experience replay pool to enhance the learning process and improve learning efficiency through experience reuse; S5-3: Store several historical meta-learning experiences of all preset tasks into the experience replay pool; the historical meta-learning experiences include corresponding historical data features; Ensure that the experience replay pool contains experience from different tasks to maintain the agent's generalization ability on multiple tasks; S5: collecting a number of real-time data, using a data feature extraction model to extract features from the number of real-time data, obtaining a number of real-time data features, and confirming a number of new tasks according to the number of real-time data features and a number of first clustering centers of different preset tasks, including the following steps: S5-1: collecting a number of real-time data, and preprocessing the real-time data to obtain a number of preprocessed real-time data; S5-2: Obtain a data feature extraction model, perform feature extraction on a number of preprocessed real-time data, and obtain a number of real-time data features; S5-3: Obtain the Euclidean distance between each real-time data feature and the first cluster center, and obtain the corresponding real-time fuzzy membership according to the Euclidean distance; S5-4: updating a number of first cluster centers according to a number of real-time data features and the real-time fuzzy membership of the first cluster centers to obtain a number of corresponding second cluster centers; S5-5: Compare the plurality of second cluster centers with the plurality of first cluster centers to obtain a plurality of third cluster centers other than the preset tasks, and set a corresponding new task for each third cluster center; S5-6: according to the Euclidean distance between each real-time data feature and the third cluster centers, the plurality of real-time data features are divided into corresponding third cluster centers to obtain second cluster clusters of the third cluster centers; S5-7: spreading the new task of the third cluster center to a number of real-time data features of the corresponding second cluster clusters, and obtaining a number of real-time data features with different new tasks as the third cluster center; S6: Using the elastic weight connection mechanism of the continuous learning algorithm, the loss function is adjusted to obtain an adjusted loss function, and under several new tasks, based on the adjusted loss function and the experience replay pool, several corresponding real-time data features are used to train the pre-trained intelligent agent to obtain the final intelligent agent, and the final intelligent agent is obtained, including the following steps: S6-1: Obtain the loss function of the pre-trained agent, and obtain several key weights of the pre-trained agent under each preset task; For each task, identify the key weights that are critical to task performance. This is usually achieved by calculating the gradient of the weights with respect to the task loss. This method can identify the network weights that are critical to the performance of each task and use this information to reduce catastrophic forgetting when training new tasks. This is especially important in continuous learning scenarios. S6-2: Obtain importance measures of several key weights, generate penalty terms according to the importance measures, and adjust the loss function according to the penalty terms to obtain an adjusted loss function; When training a new task, in order to prevent the agent from forgetting the previously learned experience, an additional penalty term can be added to the loss function using the Elastic Weight Consolidation (EWC) method. This penalty term is proportional to the importance measure of the weight and inversely proportional to the amount of change in the weight on the new task. This penalty term ensures that when training on a new task, the weights that are important for the old task do not change too much, thereby reducing the risk of catastrophic forgetting. The larger the importance measure, the more restricted the change of the corresponding weight on the new task. In this way, the agent can retain the experience of the old task while learning the new task; S6-3: Under several new tasks, based on the adjusted loss function and experience replay pool, use the corresponding several real-time data features to train the pre-trained agent to obtain the final agent, including the following steps: S6-3-1: Under the current new task, use the corresponding real-time data features to train the pre-trained agent, obtain some real-time meta-learning experiences of the new task, and store the real-time meta-learning experiences in the experience replay pool; the real-time meta-learning experiences include the corresponding real-time data features; S6-3-2: Randomly extract several historical meta-learning experiences of different preset tasks from the experience replay pool, and mix the several historical meta-learning experiences with several real-time meta-learning experiences to obtain several mixed meta-learning experiences and corresponding mixed data features; S6-3-3: Based on several hybrid meta-learning experiences, use several corresponding hybrid data features to train the pre-trained agent, and use the adjusted loss function to obtain the loss value of the agent training; By combining old experience, the agent can transfer what it has learned from previous tasks to new tasks, which helps improve learning efficiency and reduce the need for training data for new tasks. Combining old experience can help prevent the agent from forgetting what it has learned on old tasks when training on new tasks, a common continuous learning challenge known as "catastrophic forgetting." Old experience can provide a broader view of data distribution, which helps the agent generalize better on new tasks. S6-3-4: If the loss value is always less than the loss value threshold, the agent training for the next new task is performed, and all new tasks are traversed until the agent training for all new tasks is completed to obtain the final agent; S6-3-5: If the loss value is greater than or equal to the loss value threshold, then re-extract several historical meta-learning experiences of different preset tasks from the experience replay pool, and return to the agent training of the current new task; By evaluating the loss value, the training progress and performance of the agent can be monitored. If the loss value is lower than the set threshold, it can be considered that the agent's performance on the new task is acceptable and training for the next task can continue. If the loss value is higher than the threshold, it may be necessary to adjust the training strategy or continue training on the current task.

[0034] Embodiment 2: like Figure 2 As shown, this embodiment provides an agent training system based on continuous learning, which is used to implement an agent training method. The system includes an algorithm initialization unit, a feature clustering unit, an agent initialization unit, an agent pre-training unit, a new task confirmation unit, and an agent training unit connected in sequence; The algorithm initialization unit is used to use the meta-learning algorithm to define the meta-learning objectives and several preset tasks of the intelligent agent, use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool, and use the deep learning algorithm to build a data feature extraction model; A feature clustering unit is used to collect a number of historical data, use a data feature extraction model to extract features from the number of historical data, obtain a number of historical data features, and cluster the number of historical data features with preset tasks as clustering objects to obtain a number of historical data features with different preset tasks as first clustering centers; An agent initialization unit is used to optimize the initial agent parameters of the agent by using a swarm intelligence optimization algorithm with the minimization of mean square error as the optimization goal, obtain the optimal initial agent parameters of the agent, and initialize the agent according to the optimal initial agent parameters to obtain an initialized agent; An agent pre-training unit is used to pre-train the initialized agent under several preset tasks using several corresponding historical data features, obtain the pre-trained agent and several historical meta-learning experiences, and store the several historical meta-learning experiences in the experience replay pool; A new task confirmation unit is used to collect a number of real-time data, use a data feature extraction model to perform feature extraction on the number of real-time data, obtain a number of real-time data features, and confirm a number of new tasks according to the number of real-time data features and a number of first cluster centers of different preset tasks; The intelligent agent training unit is used to use the elastic weight connection mechanism of the continuous learning algorithm to adjust the loss function, obtain the adjusted loss function, and train the pre-trained intelligent agent under several new tasks based on the adjusted loss function and the experience replay pool, using several corresponding real-time data features to obtain the final intelligent agent.

[0035] The present invention provides an intelligent agent training method and system based on continuous learning. By using the experience replay mechanism of the continuous learning algorithm, the intelligent agent effectively reduces the forgetting of old experience when learning new tasks, improves the persistence and stability of the intelligent agent in multi-task learning, and reduces catastrophic forgetting; through the meta-learning algorithm and the data feature extraction model, the intelligent agent can be better generalized to unseen tasks and environments, improves the adaptability of the intelligent agent in complex and changeable environments, and enhances the generalization ability; the adopted feature extraction and clustering methods can effectively utilize historical data, reduce the dependence on a large amount of new labeled data, reduce the cost of data collection and annotation, and reduce the dependence on a large amount of labeled data; through the use of the swarm intelligence optimization algorithm and the hybrid meta-learning experience, the training process of the intelligent agent is accelerated, the required computing resources and time are reduced, so that the intelligent agent can adapt to new tasks more quickly and improve the training efficiency; it can handle dynamically changing environments, and through the collection of real-time data and task confirmation, the intelligent agent can timely adjust its behavior strategy to adapt to environmental changes and non-static environments; the meta-learning algorithm used can effectively handle complex tasks, and by mixing the meta-learning experience of different tasks, the learning efficiency of the intelligent agent on new tasks is improved.

[0036] The present invention is not limited to the above optional implementations, and anyone can derive other various forms of products under the enlightenment of the present invention. The above specific implementations should not be understood as limiting the scope of protection of the present invention. The scope of protection of the present invention should be based on the definition in the claims, and the description can be used to interpret the claims.

Claims

1. A method for training an intelligent agent based on continuous learning, characterized in that: The steps include: Use the meta-learning algorithm to define the meta-learning objectives and several preset tasks of the agent, use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool, and use the deep learning algorithm to build a data feature extraction model; Collecting a number of historical data, using a data feature extraction model to extract features from the number of historical data, obtaining a number of historical data features, and clustering the number of historical data features with preset tasks as clustering objects, obtaining a number of historical data features with different preset tasks as first clustering centers; Taking minimizing the mean square error as the optimization goal, the swarm intelligence optimization algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent, and the agent is initialized according to the optimal initial agent parameters to obtain the initialized agent; Under several preset tasks, use several corresponding historical data features to pre-train the initialized agent, obtain the pre-trained agent and several historical meta-learning experiences, and store the several historical meta-learning experiences in the experience replay pool; Collecting a number of real-time data, using a data feature extraction model to extract features from the number of real-time data, obtaining a number of real-time data features, and confirming a number of new tasks based on the number of real-time data features and a number of first clustering centers of different preset tasks; The elastic weight connection mechanism of the continuous learning algorithm is used to adjust the loss function to obtain the adjusted loss function. Under several new tasks, based on the adjusted loss function and the experience replay pool, the corresponding several real-time data features are used to train the pre-trained agent to obtain the final agent.

2. The method for training an intelligent agent based on continuous learning according to claim 1, characterized in that: Use the meta-learning algorithm to define the meta-learning objectives and several preset tasks of the agent, use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool, and use the deep learning algorithm to build a data feature extraction model, including the following steps: With the goal of enabling the agent to quickly adapt to new tasks, the meta-learning goal of the meta-learning algorithm is defined, and the preset functions of the agent are split to define several preset tasks of the meta-learning algorithm. Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool and define the form of meta-learning experience stored in the experience replay pool; Based on the recognizable feature format of the intelligent agent, a deep learning algorithm is used to build an initial data feature extraction model; A number of training data are collected, and an initial data feature extraction model is trained based on the training data to obtain a final data feature extraction model.

3. The agent training method based on continuous learning according to claim 1, characterized in that: Collecting some historical data, using a data feature extraction model to extract features from the some historical data, obtaining some historical data features, and clustering the some historical data features with preset tasks as clustering objects, obtaining some historical data features with different preset tasks as first clustering centers, including the following steps: Collecting a number of historical data of the intelligent agent in different preset tasks, and preprocessing the number of historical data to obtain a number of preprocessed historical data; Use the data feature extraction model to extract features from some pre-processed historical data to obtain some historical data features; Taking the preset tasks as clustering objects, using the FCM clustering algorithm, clustering several historical data features to obtain several first clustering centers and corresponding first clustering clusters; A corresponding preset task is set for each first cluster center, and the preset task of the first cluster center is diffused to a number of historical data features of the corresponding first cluster clusters, so as to obtain a number of historical data features with different preset tasks as the first cluster centers.

4. The method for training an agent based on continuous learning according to claim 3, characterized in that: Taking the preset task as the clustering object, using the FCM clustering algorithm, clustering several historical data features to obtain several first clustering centers and corresponding first clustering clusters, including the following steps: According to a number of preset tasks, clustering parameters of the FCM clustering algorithm are set, and according to the clustering parameters, the FCM clustering algorithm is used to initialize the clustering centers to obtain a number of initial first clustering centers; the clustering parameters include a fuzzy factor and the total number of first clustering centers; According to the Euclidean distance between each historical data feature and a number of initial first cluster centers, a corresponding historical fuzzy membership is set for each initial first cluster center; According to the historical fuzzy membership and clustering parameters, several initial first clustering centers are updated to obtain several updated first clustering centers; Use the Lagrange multiplier method to calculate the merge function, and obtain the merge function value and the merge function change value. If the merge function value is greater than the function threshold, or the merge function change value is greater than the change value threshold, then continue to update the first cluster center, otherwise, use the current first cluster center as the final first cluster center; A number of historical data features are divided into final first cluster centers with the shortest Euclidean distance, to obtain a number of final first cluster centers and corresponding first cluster clusters; the first cluster clusters include a number of historical data features.

5. The method for training an intelligent agent based on continuous learning according to claim 1, characterized in that: Taking minimizing the mean square error as the optimization goal, the swarm intelligence optimization algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent, and the agent is initialized according to the optimal initial agent parameters to obtain the initialized agent, including the following steps: Taking minimizing the mean square error as the optimization goal, the fitness function of the swarm intelligence optimization algorithm is set, and the individual encoding format of the swarm intelligence optimization algorithm is set according to the initial agent parameters of the agent; Set the algorithm parameters of the swarm intelligence optimization algorithm, and use the swarm intelligence optimization algorithm to iteratively search for the best individual based on the fitness function, individual encoding format and algorithm parameters; The individual encoding vector of the optimal individual is decoded to obtain the optimal initial intelligent agent parameters of the intelligent agent, and the intelligent agent is initialized according to the optimal initial intelligent agent parameters to obtain the initialized intelligent agent.

6. The method for training an intelligent agent based on continuous learning according to claim 4, characterized in that: Under several preset tasks, the initialized agent is pre-trained using several corresponding historical data features to obtain the pre-trained agent and several historical meta-learning experiences, and the several historical meta-learning experiences are stored in the experience replay pool, including the following steps: Under the current preset task, use the corresponding historical data features to pre-train the initialized agent to obtain some historical meta-learning experience of the current preset task; Traverse all preset tasks, pre-train the initialized agent, and obtain some historical meta-learning experience of the pre-trained agent and all preset tasks; Several historical meta-learning experiences of all preset tasks are stored in the experience replay pool; the historical meta-learning experiences include corresponding historical data features.

7. The method for training an intelligent agent based on continuous learning according to claim 6, characterized in that: Collecting a number of real-time data, using a data feature extraction model to extract features from the number of real-time data, obtaining a number of real-time data features, and confirming a number of new tasks according to the number of real-time data features and a number of first clustering centers of different preset tasks, including the following steps: Collecting a number of real-time data, and preprocessing the real-time data to obtain a number of preprocessed real-time data; A data feature extraction model is obtained to extract features from a number of preprocessed real-time data to obtain a number of real-time data features; Obtain the Euclidean distance between each real-time data feature and the first cluster center, and obtain the corresponding real-time fuzzy membership according to the Euclidean distance; According to the real-time fuzzy membership of the first cluster centers and the real-time data features, the first cluster centers are updated to obtain the corresponding second cluster centers; Comparing the plurality of second cluster centers with the plurality of first cluster centers to obtain a plurality of third cluster centers other than the preset tasks, and setting a corresponding new task for each third cluster center; According to the Euclidean distance between each real-time data feature and the third cluster centers, the real-time data features are divided into corresponding third cluster centers to obtain the second cluster clusters of the third cluster centers; The new task at the third cluster center is diffused to a number of real-time data features of the corresponding second cluster clusters, so as to obtain a number of real-time data features with different new tasks as the third cluster center.

8. The method for training an intelligent agent based on continuous learning according to claim 7, characterized in that: The elastic weight connection mechanism of the continuous learning algorithm is used to adjust the loss function to obtain the adjusted loss function, and the pre-trained agent is trained based on the adjusted loss function and the experience replay pool under several new tasks using several corresponding real-time data features to obtain the final agent, including the following steps: Obtain the loss function of the pre-trained agent and obtain several key weights of the pre-trained agent under each preset task; Obtaining importance measures of several key weights, generating penalty terms according to the importance measures, and adjusting the loss function according to the penalty terms to obtain an adjusted loss function; Under several new tasks, based on the adjusted loss function and experience replay pool, the pre-trained agent is trained using several corresponding real-time data features to obtain the final agent.

9. The method for training an intelligent agent based on continuous learning according to claim 8, characterized in that: Under several new tasks, based on the adjusted loss function and experience replay pool, the pre-trained agent is trained using several corresponding real-time data features to obtain the final agent, including the following steps: Under the current new task, use the corresponding real-time data features to train the pre-trained agent, obtain a number of real-time meta-learning experiences of the new task, and store the real-time meta-learning experiences in the experience replay pool; the real-time meta-learning experiences include the corresponding real-time data features; Randomly extract several historical meta-learning experiences of different preset tasks in the experience replay pool, and mix several historical meta-learning experiences with several real-time meta-learning experiences to obtain several mixed meta-learning experiences and corresponding mixed data features; According to several hybrid meta-learning experiences, use several corresponding hybrid data features to train the pre-trained agent, and use the adjusted loss function to obtain the loss value of the agent training; If the loss value is always less than the loss value threshold, the agent training for the next new task is carried out, and all new tasks are traversed until the agent training for all new tasks is completed to obtain the final agent; If the loss value is greater than or equal to the loss value threshold, several historical meta-learning experiences of different preset tasks are re-extracted from the experience replay pool, and the agent training of the current new task is returned.

10. An agent training system based on continuous learning, used to implement the agent training method according to any one of claims 1 to 9, characterized in that: The system comprises an algorithm initialization unit, a feature clustering unit, an agent initialization unit, an agent pre-training unit, a new task confirmation unit and an agent training unit which are connected in sequence.

Images