A method and system for training intelligent agents based on continuous learning

Abstract

The present invention belongs to the field of intelligent agent training technology, and discloses an intelligent agent training method and system based on continuous learning. The method comprises the following steps: defining meta-learning objectives and several preset tasks of the intelligent agent, constructing an experience replay pool and a data feature extraction model; performing feature extraction and clustering on several historical data; optimizing the initial intelligent agent parameters of the intelligent agent using a swarm intelligence optimization algorithm; pre-training the initialized intelligent agent, and storing several historical meta-learning experiences in the experience replay pool; performing feature extraction on several real-time data, and confirming several new tasks; adjusting the loss function, and training the pre-trained intelligent agent under several new tasks to obtain the final intelligent agent. The present invention solves the problems of catastrophic forgetting, insufficient generalization ability, dependence on labeled data, low training efficiency, and difficulty in handling non-static environments in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent agent training, and specifically relates to an intelligent agent training method and system based on continuous learning. Background Art

[0002] An intelligent agent is an entity with a certain degree of autonomy that can perceive its environment and take actions to achieve a goal. An agent can be a software program, a hardware device, or a combination of both. Intelligent agent technology has played a significant role in promoting social progress, improving productivity, and improving people's lives. Agent training is a key branch of artificial intelligence. Its primary goal is to enable agents to acquire strategies through learning while interacting with their environment to complete specific tasks.

[0003] Existing intelligent agent training technology has the following defects:

[0004] 1) Catastrophic forgetting: Existing agent training methods often suffer from catastrophic forgetting when continuously learning new tasks, where previously learned experience is forgotten during the learning process. This is because traditional training methods are typically performed on fixed datasets and do not consider the long-term retention of experience.

[0005] 2) Insufficient generalization: Many agent training methods have limited generalization capabilities to new tasks or unfamiliar environments. This means that agents may perform well in the training environment but may experience performance degradation in real-world applications.

[0006] 3) Dependence on large amounts of labeled data: Existing deep learning models typically require large amounts of labeled data for training. However, in some areas, obtaining large amounts of high-quality labeled data is time-consuming and expensive, which limits the application of intelligent agents in these areas;

[0007] 4) Inefficient training: The training process of intelligent agents often requires a lot of computing resources and time. For complex tasks, the training process may take weeks or even months, which is impractical in real applications.

[0008] 5) Difficulty in handling non-static environments: In the real world, environments often change dynamically. Existing agent training methods often assume that the environment is static, which makes it difficult for agents to adapt to changes in the environment. Summary of the Invention

[0009] In order to solve the problems of catastrophic forgetting, insufficient generalization ability, dependence on labeled data, low training efficiency and difficulty in handling non-static environments in the existing technology, the purpose of the present invention is to provide an intelligent agent training method and system based on continuous learning.

[0010] The technical solution adopted in the present invention is:

[0011] A method for training an intelligent agent based on continuous learning, comprising the following steps:

[0012] Use a meta-learning algorithm to define meta-learning objectives and several preset tasks for the agent. Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool. Use a deep learning algorithm to build a data feature extraction model.

[0013] Collecting a number of historical data, using a data feature extraction model to extract features from the number of historical data to obtain a number of historical data features, and clustering the number of historical data features with preset tasks as clustering objects to obtain a number of historical data features with different preset tasks as first cluster centers;

[0014] Taking minimizing the mean square error as the optimization goal, the swarm intelligence optimization algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent, and the agent is initialized based on the optimal initial agent parameters to obtain the initialized agent;

[0015] Under several preset tasks, the initialized agent is pre-trained using the corresponding historical data features to obtain the pre-trained agent and several historical meta-learning experiences, which are then stored in the experience replay pool.

[0016] Collecting a number of real-time data, performing feature extraction on the number of real-time data using a data feature extraction model to obtain a number of real-time data features, and identifying a number of new tasks based on the number of real-time data features and a number of first cluster centers of different preset tasks;

[0017] The elastic weight connection mechanism of the continuous learning algorithm is used to adjust the loss function to obtain an adjusted loss function. Based on the adjusted loss function and the experience replay pool, the pre-trained intelligent agent is trained using the corresponding real-time data features under several new tasks to obtain the final intelligent agent.

[0018] Furthermore, a meta-learning algorithm is used to define meta-learning objectives and several preset tasks for the agent. The experience replay mechanism of the continuous learning algorithm is used to build an experience replay pool. Finally, a deep learning algorithm is used to build a data feature extraction model. The following steps are included:

[0019] With the goal of enabling the agent to quickly adapt to new tasks, the meta-learning goal of the meta-learning algorithm is defined. The preset functions of the agent are then split and several preset tasks of the meta-learning algorithm are defined.

[0020] Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool and define the form of meta-learning experience stored in the experience replay pool;

[0021] Based on the recognizable feature format of the intelligent agent, a deep learning algorithm is used to build an initial data feature extraction model;

[0022] A number of training data are collected, and an initial data feature extraction model is trained based on the training data to obtain a final data feature extraction model.

[0023] Furthermore, a plurality of historical data are collected, and a data feature extraction model is used to extract features from the plurality of historical data to obtain a plurality of historical data features. The plurality of historical data features are clustered with preset tasks as clustering objects to obtain a plurality of historical data features with different preset tasks as first cluster centers, including the following steps:

[0024] Collecting a number of historical data of the agent in different preset tasks, and preprocessing the historical data to obtain a number of preprocessed historical data;

[0025] Use the data feature extraction model to extract features from some pre-processed historical data to obtain some historical data features;

[0026] Taking the preset tasks as clustering objects, the FCM clustering algorithm is used to cluster several historical data features to obtain several first cluster centers and corresponding first cluster clusters;

[0027] A corresponding preset task is set for each first cluster center, and the preset task of the first cluster center is diffused to several historical data features of the corresponding first cluster clusters, thereby obtaining several historical data features with different preset tasks as first cluster centers.

[0028] Furthermore, taking the preset task as the clustering object, the FCM clustering algorithm is used to cluster the features of the historical data to obtain a plurality of first cluster centers and corresponding first cluster clusters, including the following steps:

[0029] According to a number of preset tasks, the clustering parameters of the FCM clustering algorithm are set, and according to the clustering parameters, the FCM clustering algorithm is used to initialize the cluster centers to obtain a number of initial first cluster centers; the clustering parameters include the fuzzy factor and the total number of first cluster centers;

[0030] According to the Euclidean distance between each historical data feature and a number of initial first cluster centers, a corresponding historical fuzzy membership is set for each initial first cluster center;

[0031] According to the historical fuzzy membership and clustering parameters, several initial first cluster centers are updated to obtain several updated first cluster centers;

[0032] Use the Lagrange multiplier method to calculate the merge function and obtain the merge function value and the merge function change value. If the merge function value is greater than the function threshold, or the merge function change value is greater than the change value threshold, then continue to update the first cluster center. Otherwise, use the current first cluster center as the final first cluster center.

[0033] The plurality of historical data features are divided into final first cluster centers with the shortest Euclidean distance, to obtain a plurality of final first cluster centers and corresponding first cluster clusters; the first cluster clusters include the plurality of historical data features.

[0034] Furthermore, with minimizing the mean square error as the optimization goal, the swarm intelligence optimization algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent, and the agent is initialized based on the optimal initial agent parameters to obtain the initialized agent, including the following steps:

[0035] Taking minimizing the mean square error as the optimization goal, the fitness function of the swarm intelligence optimization algorithm is set, and the individual encoding format of the swarm intelligence optimization algorithm is set according to the initial agent parameters of the agent;

[0036] Set the algorithm parameters of the swarm intelligence optimization algorithm, and use the swarm intelligence optimization algorithm to iteratively search for the best individual based on the fitness function, individual encoding format, and algorithm parameters, and retain the best individual;

[0037] The individual encoding vector of the optimal individual is decoded to obtain the optimal initial intelligent agent parameters of the intelligent agent, and the intelligent agent is initialized according to the optimal initial intelligent agent parameters to obtain an initialized intelligent agent.

[0038] Furthermore, under several preset tasks, the initialized agent is pre-trained using several corresponding historical data features to obtain the pre-trained agent and several historical meta-learning experiences, and the several historical meta-learning experiences are stored in the experience replay pool, including the following steps:

[0039] Under the current preset task, use the corresponding historical data features to pre-train the initialized agent to obtain some historical meta-learning experience of the current preset task;

[0040] Traverse all preset tasks, pre-train the initialized agent, and obtain some historical meta-learning experience of the pre-trained agent and all preset tasks;

[0041] Several historical meta-learning experiences of all preset tasks are stored in the experience replay pool; the historical meta-learning experiences contain the corresponding historical data features.

[0042] Furthermore, a plurality of real-time data are collected, and a data feature extraction model is used to extract features from the plurality of real-time data to obtain a plurality of real-time data features. Based on the plurality of real-time data features and a plurality of first cluster centers of different preset tasks, a plurality of new tasks are identified, including the following steps:

[0043] Collecting a number of real-time data, and preprocessing the real-time data to obtain a number of preprocessed real-time data;

[0044] Obtain and use a data feature extraction model to perform feature extraction on a number of pre-processed real-time data to obtain a number of real-time data features;

[0045] Obtain the Euclidean distance between each real-time data feature and the first cluster center, and obtain the corresponding real-time fuzzy membership based on the Euclidean distance;

[0046] According to a number of real-time data features and the real-time fuzzy membership of the first cluster center, a number of first cluster centers are updated to obtain a number of corresponding second cluster centers;

[0047] Comparing the plurality of second cluster centers with the plurality of first cluster centers to obtain a plurality of third cluster centers in addition to the preset tasks, and setting a corresponding new task for each third cluster center;

[0048] According to the Euclidean distance between each real-time data feature and the third cluster centers, the real-time data features are divided into corresponding third cluster centers to obtain the second cluster clusters of the third cluster centers;

[0049] The new task of the third cluster center is diffused to several real-time data features of the corresponding second cluster clusters, and several real-time data features with different new tasks as the third cluster centers are obtained.

[0050] Furthermore, the elastic weight connection mechanism of the continuous learning algorithm is used to adjust the loss function to obtain an adjusted loss function. Based on the adjusted loss function and the experience replay pool, the pre-trained agent is trained using the corresponding real-time data features under several new tasks to obtain the final agent, including the following steps:

[0051] Obtain the loss function of the pre-trained agent and obtain several key weights of the pre-trained agent under each preset task;

[0052] Obtaining importance measures of several key weights, generating penalty terms based on the importance measures, and adjusting the loss function based on the penalty terms to obtain an adjusted loss function;

[0053] Under several new tasks, based on the adjusted loss function and experience replay pool, the pre-trained intelligent agent is trained using several corresponding real-time data features to obtain the final intelligent agent.

[0054] Furthermore, under several new tasks, based on the adjusted loss function and experience replay pool, the pre-trained agent is trained using the corresponding real-time data features to obtain the final agent, including the following steps:

[0055] Under the current new task, the pre-trained agent is trained using the corresponding real-time data features to obtain real-time meta-learning experiences for the new task, and the real-time meta-learning experiences are stored in the experience replay pool; the real-time meta-learning experiences contain the corresponding real-time data features;

[0056] Randomly extract several historical meta-learning experiences of different preset tasks from the experience replay pool, and mix these historical meta-learning experiences with several real-time meta-learning experiences to obtain several mixed meta-learning experiences and corresponding mixed data features;

[0057] Based on several hybrid meta-learning experiences, use the corresponding several hybrid data features to train the pre-trained agent, and use the adjusted loss function to obtain the loss value of the agent training;

[0058] If the loss value is always less than the loss value threshold, the agent training for the next new task is carried out, and all new tasks are traversed until the agent training for all new tasks is completed to obtain the final agent;

[0059] If the loss value is greater than or equal to the loss value threshold, several historical meta-learning experiences of different preset tasks are re-extracted from the experience replay pool, and the agent training of the current new task is returned.

[0060] An intelligent agent training system based on continuous learning is used to implement an intelligent agent training method. The system includes an algorithm initialization unit, a feature clustering unit, an intelligent agent initialization unit, an intelligent agent pre-training unit, a new task confirmation unit and an intelligent agent training unit which are connected in sequence.

[0061] The beneficial effects of the present invention are:

[0062] The present invention provides a method and system for training an intelligent agent based on continuous learning. By using the experience replay mechanism of the continuous learning algorithm, the method effectively reduces the agent's forgetting of old experience when learning new tasks, improves the agent's persistence and stability in multi-task learning, and reduces catastrophic forgetting. Through the meta-learning algorithm and data feature extraction model, the intelligent agent can better generalize to unseen tasks and environments, improve the agent's adaptability in complex and changing environments, and enhance generalization ability. The feature extraction and clustering methods adopted can effectively utilize historical data, reduce the dependence on large amounts of new labeled data, reduce the cost of data collection and annotation, and reduce the dependence on large amounts of labeled data. By utilizing a swarm intelligence optimization algorithm and hybrid meta-learning experience, the training process of the intelligent agent is accelerated, the required computing resources and time are reduced, and the intelligent agent can adapt to new tasks more quickly, improving training efficiency. The method can handle dynamically changing environments. Through real-time data collection and task confirmation, the intelligent agent can promptly adjust its behavior strategy to adapt to environmental changes and non-static environments. The meta-learning algorithm used can effectively handle complex tasks. By mixing meta-learning experience from different tasks, the learning efficiency of the intelligent agent on new tasks is improved.

[0063] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flowchart of the intelligent agent training method based on continuous learning in the present invention.

[0065] Figure 2 It is a structural block diagram of the intelligent agent training system based on continuous learning in the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0067] Example 1:

[0068] like Figure 1 As shown, this embodiment provides an agent training method based on continuous learning, including the following steps:

[0069] S1: Use a meta-learning algorithm to define the meta-learning goal and several preset tasks for the agent. Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool. Then use a deep learning algorithm to build a data feature extraction model. This includes the following steps:

[0070] S1-1: With the goal of enabling the agent to quickly adapt to new tasks, define the meta-learning goal of the meta-learning algorithm, split the agent's preset functions, and define several preset tasks of the meta-learning algorithm;

[0071] In addition to the ability to quickly adapt to new tasks as a meta-learning goal in this embodiment, the ability to quickly adapt to new tasks means that the agent should be able to adapt to changes in the task distribution and maintain good performance even when task conditions change; other meta-learning goals are also included and set according to specific usage requirements, including:

[0072] Fast convergence: The agent should be able to learn quickly and converge to an effective solution with very few training examples;

[0073] High generalization ability: The agent's performance on new tasks should approach or exceed that of traditional models trained on large amounts of data;

[0074] Low sample complexity: The agent should be able to learn from a limited number of samples, reducing the reliance on large amounts of data;

[0075] Robustness: The agent should be robust to noise and outliers, and its performance should not be significantly affected by small fluctuations in the data.

[0076] Cross-domain learning: The intelligent agent should be able to transfer the experience learned in one domain to another domain, achieving cross-domain learning;

[0077] The preset tasks of the meta-learning algorithm can be a series of related tasks implemented by the agent when involving similar data types or problem domains, including data processing, data analysis, data compression, etc., or they can be different but related environments or problems with similar structures, including image data classification, sequence data classification, etc.

[0078] In traditional agent training, agent parameters typically start from random initialization. However, in meta-learning, the agent learns a set of optimized parameter initializations by training on a series of tasks. These initialization parameters can provide a good starting point for new, unseen tasks.

[0079] S1-2: Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool and define the form of meta-learning experience stored in the experience replay pool;

[0080] The experience replay pool is a database that stores previous task learning experiences, including parameter settings, model status, learning strategies, etc. Joining the experience pool allows the agent to store and reuse past experiences to improve learning efficiency;

[0081] The experience replay pool is used to store the agent's interaction experience in the environment. The meta-learning experience of these experiences is usually stored in the form of (s, a, r, s'), where s is the current state, the current agent parameters of the agent, a is the action taken, including agent parameter adjustment, deletion, addition, etc., r is the reward obtained, which is used to represent the impact of the action on the state, and s' is the next state, that is, the state after the agent parameters are adjusted during training.

[0082] Since the data in the experience replay pool is randomly sampled, this helps break the correlation between consecutive experiences, thereby reducing the variance in agent training. By reusing experience, the agent can learn more from limited experience, which is especially useful in new tasks or when samples are scarce. Experience replay helps stabilize the learning process and reduce fluctuations during training. The size of the experience replay pool is regularly checked. If it exceeds the preset capacity, some experience is removed according to a certain strategy (such as priority sampling and the least recently visited).

[0083] S1-3: Based on the recognizable feature format of the agent, use the deep learning algorithm to build the initial data feature extraction model;

[0084] The deep learning algorithm used is determined by the data type of the agent input. If the input data is a plain text sequence or numeric format, the Long Short-Term Memory (LSTM) network is used as the model algorithm for the data feature extraction model. If the input data is image data, the Convolutional Neural Network (CNN) is used as the model algorithm for the data feature extraction model. If the input data is multimodal data, the fusion algorithm CNN-LSTM is used as the model algorithm for the data feature extraction model.

[0085] S1-4: Collect some training data, and train the initial data feature extraction model based on the training data to obtain the final data feature extraction model;

[0086] S2: Collect some historical data, use the data feature extraction model to extract features from the historical data, obtain some historical data features, and cluster the historical data features with the preset tasks as clustering objects to obtain some historical data features with different preset tasks as the first cluster center, including the following steps:

[0087] S2-1: Collecting a number of historical data of the agent in different preset tasks, and preprocessing the historical data to obtain a number of preprocessed historical data;

[0088] Preprocessing includes data cleaning, format conversion, and normalization. While eliminating duplicate and erroneous data, it also performs standardization conversion, eliminates magnitude differences and noise, and converts the data into a format that can be recognized by subsequent models.

[0089] S2-2: Use the data feature extraction model to extract features from some pre-processed historical data to obtain some historical data features;

[0090] S2-3: Using the preset tasks as clustering objects, the Fuzzy C-mean (FCM) clustering algorithm is used to cluster several historical data features to obtain several first cluster centers and corresponding first cluster clusters, including the following steps:

[0091] S2-3-1: According to a number of preset tasks, set the clustering parameters of the FCM clustering algorithm, and use the FCM clustering algorithm to initialize the cluster centers based on the clustering parameters to obtain a number of initial first cluster centers; the clustering parameters include the fuzzy factor and the total number of first cluster centers;

[0092] S2-3-2: According to the Euclidean distance between each historical data feature and several initial first cluster centers, set the corresponding historical fuzzy membership for each initial first cluster center;

[0093] The formula is:

[0094]

[0095] Where, For the i Historical data characteristics and j The Euclidean distance of the initial first cluster center; For the Historical data characteristics; For the j The initial first cluster center; is the data characteristic indicator; is the indicator of the first cluster center;

[0096] S2-3-3: Update several initial first cluster centers according to the historical fuzzy membership and clustering parameters to obtain several updated first cluster centers;

[0097] The formula is:

[0098]

[0099] Where, For the j Updated first cluster center; is the fuzzy factor; is the data characteristic indicator; is the total number of data; is the indicator of the first cluster center; For the Historical data characteristics; For the i The first cluster center belongs to The updated historical fuzzy membership of the first cluster center;

[0100]

[0101] Where, For the Real-time gaming market convergence feature vector; is the data characteristic indicator; Both are indicators of the first cluster center; is the total number of cluster centers; For the i Historical data features to j 、 k Updated distance to the first cluster center; For the i The first cluster center belongs to The updated historical fuzzy membership of the cluster centers;

[0102] S5-4: Repeat the above steps to obtain several final cluster centers, divide the real-time game market fusion feature vectors into the final cluster centers with the closest Euclidean distance, and obtain several clusters containing real-time prediction analysis labels;

[0103] S2-3-4: Use the Lagrange multiplier method to calculate the merge function and obtain the merge function value and the merge function change value. If the merge function value is greater than the function threshold, or the merge function change value is greater than the change value threshold, then continue to update the first cluster center. Otherwise, use the current first cluster center as the final first cluster center.

[0104] The formula is:

[0105]

[0106] Where, For the t 、 t -1 iteration of the combined function value of the Lagrange multiplier method; is the corresponding change value; For the i characteristic parameters; t is the indicator of the number of iterations; For the i The historical data characteristics belong to The updated historical fuzzy membership of the first cluster center; is the fuzzy factor; is the data indicator; is the total number of data; is the indicator of the first cluster center; is the total number of cluster centers; For the i Historical data features to j Updated Euclidean distance of the first cluster center;

[0107] S2-3-5: Divide the plurality of historical data features into final first cluster centers with the closest Euclidean distance, and obtain a plurality of final first cluster centers and corresponding first cluster clusters; the first cluster clusters include the plurality of historical data features;

[0108] S2-4: Setting a corresponding preset task for each first cluster center, and spreading the preset task of the first cluster center to a number of historical data features of the corresponding first cluster clusters, thereby obtaining a number of historical data features with different preset tasks as first cluster centers;

[0109] S3: With the minimization of mean square error as the optimization goal, the Improved Crested Porcupine Optimizer (ICPO) algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent. The agent is then initialized based on the optimal initial agent parameters to obtain the initialized agent, including the following steps:

[0110] S3-1: Set the fitness function of the swarm intelligence optimization algorithm with the optimization goal of minimizing the mean square error, and set the individual encoding format of the swarm intelligence optimization algorithm based on the initial agent parameters of the agent;

[0111] S3-2: Set the algorithm parameters of the swarm intelligence optimization algorithm, and use the swarm intelligence optimization algorithm to iteratively search for the best individual based on the fitness function, individual encoding format, and algorithm parameters, and retain the best individual. This includes the following steps:

[0112] S3-2-1: Set the algorithm parameters of the ICPO algorithm, including the ICPO population parameters and the maximum number of iterations of the ICPO algorithm;

[0113] S3-2-2: Based on the fitness function, individual encoding format and ICPO population parameters, the Circle chaotic mapping sequence is used to initialize the ICPO population to obtain an initial ICPO population containing several initial ICPO individuals;

[0114] The formula is:

[0115]

[0116] Where, is the initial ICPO individual of the Circle chaos map, corresponding to an initial solution; is the randomly generated initial ICPO individual; It is the individual indicator of ICPO;

[0117] S3-2-3: Introduce a cyclic population reduction mechanism to limit the number of individuals in the ICPO population parameters and obtain the updated ICPO population parameters for the next iteration;

[0118] The formula is:

[0119]

[0120] Where, For the The number of individuals in the ICPO population parameter of the iteration; For the The number of individuals in the ICPO population parameter of the iteration; is the minimum number of individuals in the ICPO population parameter; Evaluate arguments for functions; Evaluate loop parameters for a function; Evaluate loop parameters for the maximum function; t is the indicator of the number of iterations;

[0121] S3-2-4: Calculate the initial fitness value of the initial ICPO individuals in the initial ICPO population according to the fitness function;

[0122] The formula is:

[0123]

[0124] Where, is the fitness function; is the mean square error function;

[0125] S3-2-5: Based on the initial fitness value and the updated ICPO population parameters, the first defense strategy, the second defense strategy, the third defense strategy, and the fourth defense strategy are used to update the initial ICPO population to obtain an updated ICPO population;

[0126] The formula for the first defense strategy is:

[0127]

[0128] Where, For the updated ICPO individuals within the first defense range; The initial ICPO individual within the first defense range; is a random number based on normal distribution; is a random value in the interval [0,1]; It is the optimal solution within the first defense range; is the vector generated between the true optimal solution within the first defense range and the optimal solution randomly selected from the ICPO population; It is the individual indicator of ICPO; is the iteration indicator;

[0129] The formula for the second defense strategy is:

[0130]

[0131] Where, For the updated ICPO individuals within the second defense range; The initial ICPO individual within the second defense range; is the search upper limit vector of the second defense range; is a random value in the interval [0,1]; Respectively Initial ICPO individuals; All are [1, ] two random integers between; is the vector generated between the true optimal solution within the second defense range and the optimal solution randomly selected from the ICPO population;

[0132] The formula for the third defense strategy is:

[0133]

[0134] Where, For the updated ICPO individuals within the third defense range; The initial ICPO individual within the third defense range; is the search upper limit vector of the third defense range; Respectively Initial ICPO individuals; is [1, ] a random integer between ; The odor diffusion factor defined for the fitness function; It is a defense factor; Control parameters for search direction;

[0135] The formula for the fourth defense strategy is:

[0136]

[0137] Where, For the updated ICPO individuals within the fourth defense range; For the initial ICPO individuals within the fourth defense range; It is the optimal solution within the fourth defense range; All are random values ​​in the interval [0,1]; It is a defense factor; Control parameters for search direction; is the average force affecting the search direction; is the convergence speed factor;

[0138]

[0139] Where, is the convergence speed factor; tanh(.) is the hyperbolic tangent function; a max 、 a min are the maximum and minimum values ​​of the convergence factor respectively; λ is the deceleration rate parameter, k is the decrement period parameter, λ =-2 π , k = π ; is the iteration indicator; is the maximum number of iterations;

[0140] S3-2-6: Use the dynamic reverse learning algorithm to perform dynamic reverse learning on the updated ICPO population to generate a dynamic reverse ICPO population;

[0141] The formula is:

[0142]

[0143] Where, It is a dynamically reversed ICPO individual; is the decreasing inertia coefficient; are the maximum and minimum values ​​of the vector space respectively; For the updated ICPO individual;

[0144] S3-2-7: According to the fitness function, calculate the fitness values ​​of all ICPO individuals in the updated ICPO population and the dynamically reversed ICPO population, take the ICPO individual with the minimum fitness value as the optimal individual, and retain the optimal individual;

[0145] S3-3: Decode the individual encoding vector of the optimal individual to obtain the optimal initial agent parameters of the agent, and initialize the agent based on the optimal initial agent parameters to obtain an initialized agent;

[0146] Use swarm intelligence optimization algorithms to optimize the initial agent parameters to adapt to new tasks and prevent the agent's parameters from overfitting to new tasks during continuous learning. The initial agent parameters include the initial weights of the neural network, the initial parameters of the policy network, the initial parameters of the value function, and the initial learning rate.

[0147] S4: Under several preset tasks, use the corresponding historical data features to pre-train the initialized agent, obtain the pre-trained agent and several historical meta-learning experiences, and store the several historical meta-learning experiences in the experience replay pool, including the following steps:

[0148] S5-1: Under the current preset task, use the corresponding historical data features to pre-train the initialized agent to obtain some historical meta-learning experience for the current preset task;

[0149] S5-2: Traverse all preset tasks, pre-train the initialized agent, and obtain some historical meta-learning experience of the pre-trained agent and all preset tasks;

[0150] Train the agent on a set of predefined tasks to learn common features and strategies, using an experience replay pool to enhance the learning process and improve learning efficiency through experience reuse;

[0151] S5-3: Store several historical meta-learning experiences of all preset tasks into the experience replay pool; the historical meta-learning experiences contain the corresponding historical data features;

[0152] Ensure that the experience replay pool contains experience from different tasks to maintain the agent's generalization ability on multiple tasks;

[0153] S5: Collecting a number of real-time data, using a data feature extraction model to extract features from the real-time data to obtain a number of real-time data features, and identifying a number of new tasks based on the real-time data features and a number of first cluster centers of different preset tasks, including the following steps:

[0154] S5-1: collecting a number of real-time data, and preprocessing the real-time data to obtain a number of preprocessed real-time data;

[0155] S5-2: Obtain and use a data feature extraction model to perform feature extraction on a number of pre-processed real-time data to obtain a number of real-time data features;

[0156] S5-3: Obtain the Euclidean distance between each real-time data feature and the first cluster center, and obtain the corresponding real-time fuzzy membership based on the Euclidean distance;

[0157] S5-4: updating the first cluster centers according to the real-time data features and the real-time fuzzy membership of the first cluster centers to obtain corresponding second cluster centers;

[0158] S5-5: Comparing the plurality of second cluster centers with the plurality of first cluster centers to obtain a plurality of third cluster centers other than the preset tasks, and setting a corresponding new task for each third cluster center;

[0159] S5-6: dividing the real-time data features into corresponding third cluster centers according to the Euclidean distance between each real-time data feature and the third cluster centers, and obtaining second clusters of the third cluster centers;

[0160] S5-7: spreading the new task at the third cluster center to the corresponding real-time data features of the second cluster, and obtaining a number of real-time data features with different new tasks as the third cluster center;

[0161] S6: Use the elastic weight connection mechanism of the continuous learning algorithm to adjust the loss function to obtain an adjusted loss function. Then, based on the adjusted loss function and the experience replay pool, use the corresponding real-time data features to train the pre-trained agent under several new tasks to obtain the final agent. The final agent includes the following steps:

[0162] S6-1: Obtain the loss function of the pre-trained agent and obtain several key weights of the pre-trained agent under each preset task;

[0163] For each task, identify the key weights that are critical to task performance. This is usually achieved by calculating the gradient of the weights with respect to the task loss. This approach can identify the network weights that are critical to each task's performance and use this information to reduce catastrophic forgetting when training new tasks. This is particularly important in continuous learning scenarios.

[0164] S6-2: Obtain importance measures of several key weights, generate penalty terms based on the importance measures, and adjust the loss function based on the penalty terms to obtain an adjusted loss function;

[0165] When training a new task, to prevent the agent from forgetting previously learned experience, an additional penalty term can be added to the loss function using the Elastic Weight Consolidation (EWC) method. This penalty term is proportional to the importance measure of the weight and inversely proportional to the amount by which the weight changes on the new task. This penalty term ensures that when training on the new task, the weights that are important for the old task do not change too much, thereby reducing the risk of catastrophic forgetting. The larger the importance measure, the more restricted the change of the corresponding weight on the new task. In this way, the agent can retain experience from the old task while learning the new task;

[0166] S6-3: Based on the adjusted loss function and experience replay pool, the pre-trained agent is trained using the corresponding real-time data features under several new tasks to obtain the final agent, including the following steps:

[0167] S6-3-1: Under the current new task, use the corresponding real-time data features to train the pre-trained agent to obtain some real-time meta-learning experience for the new task, and store the real-time meta-learning experience in the experience replay pool; the real-time meta-learning experience contains the corresponding real-time data features;

[0168] S6-3-2: Randomly extract several historical meta-learning experiences of different preset tasks from the experience replay pool, and mix these historical meta-learning experiences with several real-time meta-learning experiences to obtain several mixed meta-learning experiences and corresponding mixed data features;

[0169] S6-3-3: Based on several hybrid meta-learning experiences, use the corresponding several hybrid data features to train the pre-trained agent, and use the adjusted loss function to obtain the loss value of the agent training;

[0170] By combining old experience, the agent can transfer what it has learned from previous tasks to new tasks, which helps improve learning efficiency and reduces the need for training data for new tasks. Combining old experience can help prevent the agent from forgetting what it has learned on old tasks when training on new tasks, a common continuous learning challenge known as "catastrophic forgetting." Old experience can provide a broader view of the data distribution, which helps the agent generalize better on new tasks.

[0171] S6-3-4: If the loss value is always less than the loss value threshold, the agent training for the next new task is carried out, and all new tasks are traversed until the agent training for all new tasks is completed to obtain the final agent;

[0172] S6-3-5: If the loss value is greater than or equal to the loss value threshold, re-extract several historical meta-learning experiences of different preset tasks from the experience replay pool and return to the current agent training of the new task;

[0173] By evaluating the loss value, the training progress and performance of the agent can be monitored. If the loss value is lower than the set threshold, it can be considered that the agent's performance on the new task is acceptable and training can continue on the next task. If the loss value is higher than the threshold, it may be necessary to adjust the training strategy or continue training on the current task.

[0174] Example 2:

[0175] like Figure 2 As shown, this embodiment provides an agent training system based on continuous learning, which is used to implement an agent training method. The system includes an algorithm initialization unit, a feature clustering unit, an agent initialization unit, an agent pre-training unit, a new task confirmation unit, and an agent training unit connected in sequence;

[0176] The algorithm initialization unit is used to use the meta-learning algorithm to define the meta-learning objectives and several preset tasks of the intelligent agent, use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool, and use the deep learning algorithm to build a data feature extraction model;

[0177] A feature clustering unit is used to collect a number of historical data, use a data feature extraction model to extract features from the number of historical data, obtain a number of historical data features, and cluster the number of historical data features with preset tasks as clustering objects to obtain a number of historical data features with different preset tasks as first cluster centers;

[0178] The agent initialization unit is used to optimize the initial agent parameters of the agent using a swarm intelligence optimization algorithm with the optimization goal of minimizing the mean square error to obtain the optimal initial agent parameters of the agent, and initialize the agent based on the optimal initial agent parameters to obtain an initialized agent;

[0179] The agent pre-training unit is used to pre-train the initialized agent under several preset tasks using corresponding historical data features, obtain the pre-trained agent and several historical meta-learning experiences, and store the several historical meta-learning experiences in the experience replay pool;

[0180] a new task confirmation unit, configured to collect a plurality of real-time data, perform feature extraction on the plurality of real-time data using a data feature extraction model to obtain a plurality of real-time data features, and confirm a plurality of new tasks based on the plurality of real-time data features and a plurality of first cluster centers of different preset tasks;

[0181] The intelligent agent training unit is used to use the elastic weight connection mechanism of the continuous learning algorithm to adjust the loss function to obtain the adjusted loss function. Under several new tasks, based on the adjusted loss function and the experience replay pool, the pre-trained intelligent agent is trained using several corresponding real-time data features to obtain the final intelligent agent.

[0182] The present invention provides a method and system for training an intelligent agent based on continuous learning. By using the experience replay mechanism of the continuous learning algorithm, the method effectively reduces the agent's forgetting of old experience when learning new tasks, improves the agent's persistence and stability in multi-task learning, and reduces catastrophic forgetting. Through the meta-learning algorithm and data feature extraction model, the intelligent agent can better generalize to unseen tasks and environments, improve the agent's adaptability in complex and changing environments, and enhance generalization ability. The feature extraction and clustering methods adopted can effectively utilize historical data, reduce the dependence on large amounts of new labeled data, reduce the cost of data collection and annotation, and reduce the dependence on large amounts of labeled data. By utilizing a swarm intelligence optimization algorithm and hybrid meta-learning experience, the training process of the intelligent agent is accelerated, the required computing resources and time are reduced, and the intelligent agent can adapt to new tasks more quickly, improving training efficiency. The method can handle dynamically changing environments. Through real-time data collection and task confirmation, the intelligent agent can promptly adjust its behavior strategy to adapt to environmental changes and non-static environments. The meta-learning algorithm used can effectively handle complex tasks. By mixing meta-learning experience from different tasks, the learning efficiency of the intelligent agent on new tasks is improved.

[0183] The present invention is not limited to the above optional embodiments. Anyone can derive various other forms of products based on the teachings of the present invention. The above specific embodiments should not be construed as limiting the scope of protection of the present invention. The scope of protection of the present invention shall be based on the scope defined in the claims, and the description can be used to interpret the claims.

Claims

1. A method for training an intelligent agent based on continuous learning, characterized by: The steps include: For image data classification tasks, we use a meta-learning algorithm to define meta-learning objectives and several preset tasks for the agent. We use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool, and use a deep learning algorithm to build a data feature extraction model. Collecting a number of historical data for image data classification, using a data feature extraction model to extract features from the number of historical data to obtain a number of historical data features, and clustering the number of historical data features with preset tasks as clustering objects to obtain a number of historical data features with different preset tasks as first cluster centers; Taking minimizing the mean square error as the optimization goal, the swarm intelligence optimization algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent. Based on the optimal initial agent parameters, the agent is initialized to obtain an initialized agent for image data classification. Under several preset tasks corresponding to the image data classification task, using several corresponding historical data features, pre-training the initialized intelligent agent for image data classification, obtaining the pre-trained intelligent agent for image data classification and several historical meta-learning experiences, and storing the several historical meta-learning experiences in the experience replay pool; Collecting a plurality of real-time data for image data classification, performing feature extraction on the plurality of real-time data using a data feature extraction model to obtain a plurality of real-time data features, and determining a plurality of new tasks for image data classification based on the plurality of real-time data features and a plurality of first cluster centers of different preset tasks; The elastic weight connection mechanism of the continuous learning algorithm is used to adjust the loss function to obtain an adjusted loss function. Based on the adjusted loss function and the experience replay pool, the pre-trained intelligent agent is trained using several corresponding real-time data features under several new tasks of image data classification to obtain the final intelligent agent for image data classification.

2. The method for training an intelligent agent based on continuous learning according to claim 1, characterized in that: Use a meta-learning algorithm to define meta-learning objectives and several preset tasks for the agent. Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool. Then use a deep learning algorithm to build a data feature extraction model. The steps include: With the goal of enabling the agent to quickly adapt to new tasks, the meta-learning goal of the meta-learning algorithm is defined. The preset functions of the agent are then split and several preset tasks of the meta-learning algorithm are defined. Use the experience replay mechanism of the continuous learning algorithm to build an experience replay pool and define the form of meta-learning experience stored in the experience replay pool; Based on the recognizable feature format of the intelligent agent, a deep learning algorithm is used to build an initial data feature extraction model; A number of training data are collected, and an initial data feature extraction model is trained based on the training data to obtain a final data feature extraction model.

3. The method for training an intelligent agent based on continuous learning according to claim 1, characterized in that: Collecting a number of historical data, using a data feature extraction model to extract features from the historical data to obtain a number of historical data features, and clustering the historical data features with preset tasks as clustering objects to obtain a number of historical data features with different preset tasks as first cluster centers, including the following steps: Collecting a number of historical data of the agent in different preset tasks, and preprocessing the historical data to obtain a number of preprocessed historical data; Use the data feature extraction model to extract features from some pre-processed historical data to obtain some historical data features; Taking the preset tasks as clustering objects, the FCM clustering algorithm is used to cluster several historical data features to obtain several first cluster centers and corresponding first cluster clusters; A corresponding preset task is set for each first cluster center, and the preset task of the first cluster center is diffused to several historical data features of the corresponding first cluster clusters, thereby obtaining several historical data features with different preset tasks as first cluster centers.

4. The method for training an intelligent agent based on continuous learning according to claim 3, characterized in that: Taking the preset tasks as clustering objects, the FCM clustering algorithm is used to cluster several historical data features to obtain several first cluster centers and corresponding first cluster clusters, including the following steps: According to a number of preset tasks, clustering parameters of the FCM clustering algorithm are set, and according to the clustering parameters, the FCM clustering algorithm is used to initialize the cluster centers to obtain a number of initial first cluster centers; the clustering parameters include a fuzzy factor and the total number of first cluster centers; According to the Euclidean distance between each historical data feature and a number of initial first cluster centers, a corresponding historical fuzzy membership is set for each initial first cluster center; According to the historical fuzzy membership and clustering parameters, several initial first cluster centers are updated to obtain several updated first cluster centers; Use the Lagrange multiplier method to calculate the merge function and obtain the merge function value and the merge function change value. If the merge function value is greater than the function threshold, or the merge function change value is greater than the change value threshold, then continue to update the first cluster center. Otherwise, use the current first cluster center as the final first cluster center. Several historical data features are divided into final first cluster centers with the shortest Euclidean distance, to obtain several final first cluster centers and corresponding first cluster groups; the first cluster groups include several historical data features.

5. The method for training an intelligent agent based on continuous learning according to claim 1, characterized in that: Taking minimizing the mean square error as the optimization goal, the swarm intelligence optimization algorithm is used to optimize the initial agent parameters of the agent to obtain the optimal initial agent parameters of the agent. The agent is initialized based on the optimal initial agent parameters to obtain the initialized agent, including the following steps: Taking minimizing the mean square error as the optimization goal, the fitness function of the swarm intelligence optimization algorithm is set, and the individual encoding format of the swarm intelligence optimization algorithm is set according to the initial agent parameters of the agent; Set the algorithm parameters of the swarm intelligence optimization algorithm, and use the swarm intelligence optimization algorithm to iteratively search for the best individual based on the fitness function, individual encoding format, and algorithm parameters, and retain the best individual; The individual encoding vector of the optimal individual is decoded to obtain the optimal initial intelligent agent parameters of the intelligent agent, and the intelligent agent is initialized according to the optimal initial intelligent agent parameters to obtain an initialized intelligent agent.

6. The method for training an intelligent agent based on continuous learning according to claim 4, characterized in that: Under several preset tasks, the initialized agent is pre-trained using the corresponding historical data features to obtain the pre-trained agent and several historical meta-learning experiences. The historical meta-learning experiences are then stored in the experience replay pool, including the following steps: Under the current preset task, use the corresponding historical data features to pre-train the initialized agent to obtain some historical meta-learning experience of the current preset task; Traverse all preset tasks, pre-train the initialized agent, and obtain some historical meta-learning experience of the pre-trained agent and all preset tasks; Several historical meta-learning experiences of all preset tasks are stored in the experience replay pool; the historical meta-learning experiences include corresponding historical data features.

7. The method for training an intelligent agent based on continuous learning according to claim 6, characterized in that: Collecting a number of real-time data, using a data feature extraction model to extract features from the real-time data to obtain a number of real-time data features, and identifying a number of new tasks based on the real-time data features and a number of first cluster centers of different preset tasks, including the following steps: Collecting a number of real-time data, and preprocessing the real-time data to obtain a number of preprocessed real-time data; Obtain and use a data feature extraction model to perform feature extraction on a number of pre-processed real-time data to obtain a number of real-time data features; Obtain the Euclidean distance between each real-time data feature and the first cluster center, and obtain the corresponding real-time fuzzy membership based on the Euclidean distance; According to a number of real-time data features and the real-time fuzzy membership of the first cluster center, a number of first cluster centers are updated to obtain a number of corresponding second cluster centers; Comparing the plurality of second cluster centers with the plurality of first cluster centers to obtain a plurality of third cluster centers in addition to the preset tasks, and setting a corresponding new task for each third cluster center; According to the Euclidean distance between each real-time data feature and the third cluster centers, the real-time data features are divided into corresponding third cluster centers to obtain the second cluster clusters of the third cluster centers; The new task of the third cluster center is diffused to several real-time data features of the corresponding second cluster clusters, and several real-time data features with different new tasks as the third cluster centers are obtained.

8. The method for training an intelligent agent based on continuous learning according to claim 7, characterized in that: The elastic weight connection mechanism of the continuous learning algorithm is used to adjust the loss function to obtain an adjusted loss function. Based on the adjusted loss function and the experience replay pool, the pre-trained agent is trained on several new tasks using the corresponding real-time data features to obtain the final agent. The steps include the following: Obtain the loss function of the pre-trained agent and obtain several key weights of the pre-trained agent under each preset task; Obtaining importance measures of several key weights, generating penalty terms based on the importance measures, and adjusting the loss function based on the penalty terms to obtain an adjusted loss function; Under several new tasks, based on the adjusted loss function and experience replay pool, the pre-trained intelligent agent is trained using several corresponding real-time data features to obtain the final intelligent agent.

9. The method for training an intelligent agent based on continuous learning according to claim 8, characterized in that: Under several new tasks, based on the adjusted loss function and experience replay pool, the pre-trained agent is trained using the corresponding real-time data features to obtain the final agent, including the following steps: Under the current new task, the pre-trained agent is trained using the corresponding real-time data features to obtain a number of real-time meta-learning experiences for the new task, and the real-time meta-learning experiences are stored in the experience replay pool; the real-time meta-learning experiences include the corresponding real-time data features; Randomly extract several historical meta-learning experiences of different preset tasks from the experience replay pool, and mix these historical meta-learning experiences with several real-time meta-learning experiences to obtain several mixed meta-learning experiences and corresponding mixed data features; Based on several hybrid meta-learning experiences, use the corresponding several hybrid data features to train the pre-trained agent, and use the adjusted loss function to obtain the loss value of the agent training; If the loss value is always less than the loss value threshold, the agent training for the next new task is carried out, and all new tasks are traversed until the agent training for all new tasks is completed to obtain the final agent; If the loss value is greater than or equal to the loss value threshold, several historical meta-learning experiences of different preset tasks are re-extracted from the experience replay pool, and the agent training of the current new task is returned.

10. An agent training system based on continuous learning, for implementing the agent training method according to any one of claims 1 to 9, characterized in that: The system comprises an algorithm initialization unit, a feature clustering unit, an agent initialization unit, an agent pre-training unit, a new task confirmation unit and an agent training unit which are connected in sequence.

Images