Multi-agent cooperation method and device based on meta reinforcement learning, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms of medical health, financial science and technology and the like, and discloses a multi-agent cooperation method based on meta reinforcement learning, which comprises the following steps: constructing a task feature vector according to acquired multi-source information data; calculating a task difference degree between the new task and the training task according to the task feature vector; constructing a meta-training model through the task difference degree; performing double-loop training on the meta-training model according to the sampling task to obtain an optimal meta-parameter; taking the optimal meta-parameter as an initial parameter of a new task, updating a task difference degree between the new task and the training task according to the initial parameter, determining a fine tuning learning rate according to the updated task difference degree, and performing fine tuning processing on the meta-training model by using the fine tuning learning rate; environment data of preset time are obtained, the environment data are input into the meta-training model after fine tuning processing, the action distribution probability is obtained, specific actions are determined according to the action distribution probability, and the multiple agents are made to execute the specific actions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-agent collaboration method, apparatus, device, and storage medium based on meta-reinforcement learning. Background Technology

[0002] In the field of multi-agent reinforcement learning, although some studies have attempted to combine meta-learning with multi-agent learning, a complete solution has yet to be formed. Some studies only introduce meta-parameters in a single task without constructing a multi-task training set, thus failing to achieve multi-task learning. Other studies, while achieving meta-learning for individual agents, lack collaborative constraint mechanisms, making policy synchronization problems difficult to solve. For example, the MADDPG algorithm uses a centralized critic network to process global information and a distributed actor network to generate individual actions for collaboration, but it does not introduce a meta-learning mechanism. Therefore, when faced with a new task, MADDPG needs to be trained from scratch and cannot quickly adapt to task changes. The COMA algorithm uses an attention mechanism to allow the critic network to focus on the actions of key agents; however, it lacks task adaptability and its initial collaboration effect is poor after the task mode changes. The QMIX algorithm achieves multi-agent Q-value optimization through monotonic value function decomposition, suitable for discrete action spaces, but its policy update speed is slow in dynamic tasks, making it difficult to cope with rapid task switching. The MAML algorithm focuses on the rapid adaptation of a single agent to new tasks but cannot handle multi-agent interaction scenarios, making it difficult to coordinate the policy adjustments of multiple agents. RL 2 The algorithm achieves single-agent meta-learning by storing task experience through recurrent neural networks. However, in multi-agent scenarios, the recurrent state is easily affected by other agents, leading to policy instability.

[0003] Traditional multi-agent reinforcement learning methods typically design loss functions only for the current task, neglecting generalization to new tasks. Therefore, when task parameters change, agents require extensive retraining to adjust their policies, hindering rapid adaptation and resulting in extremely low collaboration efficiency during the initial task launch phase. In existing multi-agent methods, each agent's policy update formula is often calculated independently, without incorporating the policy parameters of other agents. This independent update mechanism leads to inconsistent policy adjustment rhythms when tasks change, easily causing action conflicts and impacting overall collaboration. Furthermore, existing methods have limited ability to handle environmental interference; their state-aware models lack anti-interference mechanisms. When interference such as communication interruptions or sensor noise occurs, the environmental state acquired by the agent becomes biased, leading to inaccurate action policies calculated based on the environmental state. This severely affects the stability of the entire collaborative system, potentially causing task interruption. Single-agent meta-learning methods only apply to parameter adjustments for a single agent in a new task, failing to consider policy interdependencies in multi-agent scenarios. Because of the complex interactions between agents in a multi-agent system, the policy adjustment of a single agent may trigger a chain reaction. Existing single-agent meta-learning methods cannot handle such dynamic dependencies and are difficult to apply directly to multi-agent collaborative scenarios.

[0004] In the healthcare field, tasks involve complex scenarios such as disease diagnosis, treatment selection, and patient care. Existing methods, such as MADDPG, COMA, and QMIX, have significant shortcomings when facing new tasks. These methods require training from scratch and cannot quickly adapt to task changes, leading to a failure to adjust strategies in a timely manner in emergency situations (such as sudden illness or the emergence of new treatment methods). Furthermore, the policy update formulas for each agent are often calculated independently, without incorporating the policy parameters of other agents, resulting in inconsistent policy adjustment rhythms and potential action conflicts. For example, in a medical team, collaboration between doctors and nurses may become chaotic due to inconsistent strategies. Simultaneously, existing methods have limited ability to handle environmental interference, and state-aware models lack anti-interference mechanisms. In medical environments, interference such as communication interruptions and sensor noise are common, which can lead to decisions based on erroneous data, such as incorrect diagnoses or treatment plans. Moreover, single-agent meta-learning methods do not consider the policy interdependencies in multi-agent scenarios, cannot handle dynamic dependencies, and are difficult to directly apply to multi-agent collaborative scenarios.

[0005] In the fintech business, tasks are highly dynamic, involving market volatility, risk assessment, and trading strategy adjustments. Existing methods such as MADDPG and QMIX require retraining from scratch when new tasks (such as new financial products or market environments) emerge, hindering their rapid adaptation. This can lead to a failure to adjust strategies in a timely manner during rapid market changes, reducing the market responsiveness of financial institutions and increasing trading risks. Fintech operations involve multi-agent collaboration, such as traders, risk assessors, and data analysts. The independent update mechanism in existing methods leads to inconsistent agent strategy adjustments, easily resulting in action conflicts. For example, the collaboration between traders and risk assessors in trading decisions may become chaotic due to inconsistent strategies. The fintech environment is subject to various disturbances, such as network latency and data noise. Existing methods have limited ability to handle environmental disturbances, and state-aware models lack anti-interference mechanisms. This can lead to decisions based on erroneous data, such as incorrect risk assessments or trading signals. Summary of the Invention

[0006] The main objective of this invention is to provide a multi-agent collaboration method, apparatus, device, and storage medium based on meta-reinforcement learning, aiming to solve the problems in the prior art where new tasks require training from scratch, collaboration synchronization is insufficient, agent policy updates are asynchronous, and single-agent methods are difficult to apply to multi-agent collaboration.

[0007] To achieve the above objectives, this invention provides a multi-agent cooperation method based on meta-reinforcement learning, including... Acquire multi-source information data and construct a task feature vector based on the multi-source information data; Calculate the task difference between the new task and the training task based on the task feature vector; A meta-training model is constructed using the aforementioned task differences; A sampling task is obtained, and the meta-training model is trained in a double loop using the sampling task to obtain the optimal meta-parameters. The optimal meta-parameters are used as the initial parameters for the new task, and the task difference between the new task and the training task is updated according to the initial parameters. The fine-tuning learning rate is determined according to the updated task difference, and the meta-training model is fine-tuned according to the final layer parameters using the fine-tuning learning rate. The system acquires environmental data for a preset time period and inputs the environmental data into the fine-tuned meta-training model to obtain the action distribution probability. Based on the action distribution probability, the system determines the specific action and causes the multi-agent to execute the specific action.

[0008] Furthermore, to achieve the above objectives, the present invention provides a multi-agent cooperative device based on meta-reinforcement learning, including... The feature vector module is used to acquire multi-source information data and construct task feature vectors based on the multi-source information data. The task difference module is used to calculate the task difference between the new task and the training task based on the task feature vector. The model building module is used to build a meta-training model based on the task differences. The meta-parameter module is used to acquire sampling tasks and perform double-loop training on the meta-training model through the sampling tasks to obtain the optimal meta-parameters. The model fine-tuning module is used to take the optimal meta-parameters as the initial parameters of the new task, update the task difference between the new task and the training task according to the initial parameters, determine the fine-tuning learning rate according to the updated task difference, and use the fine-tuning learning rate to fine-tune the meta-training model according to the parameters of the last layer. The action execution module is used to acquire environmental data for a preset time, input the environmental data into the fine-tuned meta-training model to obtain the action distribution probability, determine the specific action based on the action distribution probability, and enable the multi-agent to execute the specific action.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a multi-agent cooperation program based on meta-reinforcement learning stored in the memory and executable on the processor, wherein when the multi-agent cooperation program based on meta-reinforcement learning is executed by the processor, it implements the steps of the multi-agent cooperation method based on meta-reinforcement learning as described above.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a multi-agent cooperation program based on meta-reinforcement learning, wherein when the multi-agent cooperation program based on meta-reinforcement learning is executed by a processor, it implements the steps of the multi-agent cooperation method based on meta-reinforcement learning as described above.

[0011] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business system platforms such as healthcare and fintech. It discloses a multi-agent collaborative method based on meta-reinforcement learning, comprising: acquiring multi-source information data; constructing a task feature vector based on the multi-source information data; calculating the task difference degree between a new task and a training task based on the task feature vector; constructing a meta-training model using the task difference degree; acquiring a sampled task and performing double-loop training on the meta-training model using the sampled task to obtain optimal meta-parameters; using the optimal meta-parameters as initial parameters for a new task, updating the task difference degree between the new task and the training task based on the initial parameters, determining a fine-tuning learning rate based on the updated task difference degree, fine-tuning the meta-training model using the fine-tuning learning rate and based on the final layer parameters; acquiring environmental data for a preset time and inputting the environmental data into the fine-tuned meta-training model to obtain action distribution probabilities, determining specific actions based on the action distribution probabilities, and having the multi-agents execute the specific actions. This invention constructs vectors based on multi-dimensional task features and calculates task difference. It uses a meta-training model containing a basic network layer, a meta-learning adjustment layer, and a collaborative constraint layer for double-loop training to obtain optimal meta-parameters. New tasks use the optimal meta-parameters as initial parameters, fine-tuning them according to the difference. Finally, it combines temporal features to generate actions and makes collaborative corrections. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments. Figure 1 This is a schematic diagram of an application environment for a multi-agent cooperation method based on meta-reinforcement learning in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the multi-agent cooperation method based on meta-reinforcement learning of the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-agent cooperative device based on meta-reinforcement learning of the present invention. Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0014] The multi-agent cooperation method based on meta-reinforcement learning provided in this invention can be applied to, for example... Figure 1In this application environment, the user terminal communicates with the server via a network. The server can obtain multi-source information data from the user terminal and construct a task feature vector based on the multi-source information data; calculate the task difference degree between the new task and the training task based on the task feature vector; construct a meta-training model based on the task difference degree; obtain a sampled task and perform double-loop training on the meta-training model using the sampled task to obtain optimal meta-parameters; use the optimal meta-parameters as the initial parameters for the new task, update the task difference degree between the new task and the training task based on the initial parameters, determine the fine-tuning learning rate based on the updated task difference degree, and fine-tune the meta-training model using the fine-tuning learning rate and the parameters of the last layer; obtain environmental data for a preset time and input the environmental data into the fine-tuned meta-training model to obtain the action distribution probability; determine the specific action based on the action distribution probability and instruct the multi-agent to execute the specific action. This invention constructs vectors from multi-dimensional task features and calculates task dissimilarity. It employs a meta-training model containing a base network layer, a meta-learning adjustment layer, and a collaborative constraint layer for double-loop training to obtain optimal meta-parameters. New tasks use these optimal meta-parameters as initial parameters, fine-tuned according to the dissimilarity, and finally, actions are generated and collaboratively corrected by combining temporal features. The user end can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server end can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments.

[0015] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the multi-agent cooperation method based on meta-reinforcement learning provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0016] like Figure 2 As shown, the multi-agent cooperative method based on meta-reinforcement learning proposed in this invention includes the following steps: S100: Obtain multi-source information data, and construct a task feature vector based on the multi-source information data; S200. Calculate the task difference between the new task and the training task based on the task feature vector. S300. Construct a meta-training model based on the task difference; S400. Obtain the sampling task and perform double-loop training on the meta-training model through the sampling task to obtain the optimal meta-parameters. S500. The optimal meta-parameters are used as the initial parameters of the new task, and the task difference between the new task and the training task is updated according to the initial parameters. The fine-tuning learning rate is determined according to the updated task difference, and the meta-training model is fine-tuned according to the final layer parameters using the fine-tuning learning rate. S600. Obtain environmental data for a preset time, input the environmental data into the fine-tuned meta-training model to obtain the action distribution probability, determine the specific action based on the action distribution probability, and make the multi-agent execute the specific action.

[0017] In this embodiment, the core process is "input-meta-training-new-task adaptation-output". In the input stage, the focus is on task feature extraction and task difference calculation. Task feature extraction constructs a vector F=[F...] through multi-dimensional encoding. goal ;F env ; F agent ]: F goal For the task target data, discrete features are encoded using one-hot encoding, and continuous features are normalized using min-max normalization; F env When processing environmental data, Kalman filtering is used to denoise sensor observations and integrate information such as obstacle distribution; F maps the discrete features of the agent into low-dimensional vectors through an embedding layer, and then concatenates data such as capability values. The task difference calculation adopts a weighted fusion formula of cosine similarity and Manhattan distance, which transforms the feature vector difference between the new task and the training task into a scalar in the [0,1] interval, providing a quantitative basis for subsequent policy adjustments.

[0018] The meta-training phase relies on a hierarchical collaborative meta-learning mechanism to construct a model containing a base network layer, a meta-learning adjustment layer, and a collaborative constraint layer. The base network layer configures an actor network for each agent (generating individual actions) and shares a critic network (evaluating global value). The meta-learning adjustment layer outputs a dynamic learning rate using a sigmoid function combined with task variability. The collaborative constraint layer calculates the policy association matrix to quantify the collaborative weights between agents. Training employs a dual-loop process: the inner loop updates parameters iteratively in three steps for the sampled task, with the loss function including TD error (ensuring the effectiveness of the current task policy) and a collaborative regularization term (constraining consistent gradient directions); the outer loop collects the results of the multi-task inner loop and updates the meta-parameters using the RMSprop optimizer to achieve cross-task generalization.

[0019] The new task adaptation phase employs a dynamic fine-tuning strategy for rapid adaptation. First, the optimal meta-parameters obtained from meta-training are loaded, and the average of the five smallest differences between the new task and the training task, D, is calculated. avgThe learning rate is determined by the meta-learning adjustment layer; then only the parameters of the last layer of the actor network are updated, iterating for 1-3 steps, while calculating the collaborative error. If the threshold is exceeded, the fine-tuning is repeated, which reduces the amount of computation while ensuring action synchronization.

[0020] The output phase enables real-time collaborative action generation and dynamic optimization. During action generation, LSTM extracts the temporal features of the environment state at consecutive time steps. After the actor network outputs the action distribution, it is sampled by Gumbel-Softmax to obtain specific actions. Then, the collaboration is corrected by fusing partner actions through a policy association matrix. The feedback mechanism evaluates the effect through a multi-dimensional reward function (task progress, collaboration quality, and penalty terms fused by weight). When the reward is lower than the threshold for three consecutive steps, emergency fine-tuning is initiated (increasing the number of inner loop steps and increasing the learning rate) to enhance robustness in complex environments.

[0021] Overall, this method improves the speed of adapting to new tasks and enhances the synchronization of agent collaboration through mechanisms such as multi-dimensional feature encoding, dual-loop training, and dynamic fine-tuning. It is applicable to multi-agent dynamic collaboration scenarios such as industrial manufacturing and logistics warehousing.

[0022] For example, in cross-border payment scenarios within the fintech field, transaction habits and fraud patterns vary significantly across different regions. Traditional anti-fraud models are slow to adapt to new regions and have a high false positive rate. To address this issue, a method is proposed that integrates multi-source information data, including transaction data, user data, and regional feature data. Different features are processed using one-hot encoding, min-max normalization, and Kalman filtering to form a task feature vector along the "transaction-user-region" dimension. When expanding to new regions, a difference coefficient D=0.75 is calculated through a weighted fusion of cosine similarity and Manhattan distance, indicating a significant difference and providing a basis for model adjustment. A meta-training model is constructed based on this difference coefficient. The basic network layer is configured with an actor network and a shared critic network, and the meta-learning adjustment layer dynamically outputs the learning rate based on the difference coefficient. Meta-parameters are optimized through a double-loop training process. The inner loop updates parameters for the sampling task, while the outer loop integrates results from multiple regional tasks to update the meta-parameters. When expanding to new regions, the optimal meta-parameters are loaded, and the learning rate is fine-tuned based on the difference coefficient. Only the parameters of the last layer of the actor network are updated. After iteration, adaptation is completed through collaborative error verification. The system acquires environmental data in real time, extracts temporal features using LSTM, outputs action distributions from the actor network, and executes the final action after sampling and fusion with the policy association matrix. This system reduces the adaptation time of anti-fraud models in new regions from 2 weeks to 4 hours and lowers the false positive rate by 52%.

[0023] In the healthcare field, surgical robots need to adapt to different operational logics in different surgical scenarios. Traditional robots require separate training for each surgical scenario, resulting in poor collaboration when facing complex combined surgeries. To address this issue, multi-source surgical data is integrated, including intraoperative data, surgical type data, and robot parameter data. Kalman filtering is used to denoise the image data, and discrete data is converted into low-dimensional vectors through an embedding layer, concatenating them into a "intraoperative-surgical type-robot" feature vector. When the system transitions from "cholecystectomy" to "hepatectomy," the calculated difference D=0.6 indicates a moderate difference, requiring adjustment of parameters such as "instrument gripping force" and "movement accuracy." A meta-training model is constructed, with the base network layer configuring an actor network and a shared critic network for each robotic arm. The meta-learning adjustment layer outputs a dynamic learning rate based on the difference (set to 0.03 when D=0.6). Meta-parameters are optimized through a dual-loop training process: the inner loop updates parameters based on the sampling task, while the outer loop integrates results from multiple surgical scenarios to update the meta-parameters. When faced with a new surgical procedure, the system loads optimal meta-parameters, fine-tunes the learning rate based on the D-value, and updates only the output layer parameters of the actor network. After iteration, adaptation is completed through collaborative error verification. The system acquires environmental data in real time, extracts temporal features using LSTM, outputs the action distribution from the actor network, and executes the final action after sampling and fusion with the policy association matrix. This system reduces the robot adaptation time across surgical scenarios from 12 hours to 1.5 hours and decreases the surgical complication rate by 40%.

[0024] In one embodiment, step S100 includes S101. Obtain task target data, and construct a target feature sub-vector based on the task target data; S102. Obtain environmental data and construct an environmental feature sub-vector based on the environmental data; S103. Obtain agent data and construct agent feature sub-vectors based on the agent data; S104. Concatenate the target feature vector, the environment feature vector, and the agent feature vector to obtain the task feature vector.

[0025] In this embodiment, the task feature vector F = [F goAl ; F env ; F Agent ] is defined as a vector composed of three sub-vectors, each corresponding to a core element of the task. Wherein, F goAl (Target feature sub-vector) is used to describe the core target attributes of the task, including the target coordinates (e.g., the placement of assembly parts), priority (e.g., the urgency of an order), and task type (e.g., discrete types such as handling and welding). F env(Environmental feature sub-vectors) reflect the external environmental state of task execution, including obstacle distribution (e.g., equipment location in the workshop), communication quality (e.g., signal strength between agents), and real-time environmental data collected by sensors (e.g., radar scan results). F Agent (Agent Feature Subvector) Characterizes the attributes of the agents participating in the collaboration, including agent ID (used to distinguish individuals), role (such as division of labor such as "grabbing" and "transporting"), and capability value (such as quantitative indicators such as movement speed and load).

[0026] Different transformation methods were employed for different feature types of sub-vectors (discrete / continuous, noisy / high-dimensional) to ensure that features could be computed compatibly while retaining key information. For F... goAl For discrete features (such as task type), one-hot encoding is used. If a feature has k categories, it is converted into a k-dimensional vector, with the corresponding category position set to 1 and the rest to 0. For example, for the three task categories "handling," "assembling," and "patrolling," they are encoded as [1,0,0], [0,1,0], and [0,0,1], respectively, thus solving the problem that discrete features cannot be directly used in numerical calculations. Continuous features (such as target coordinates and priority values) are normalized using min-max normalization, mapping the data to the [0,1] interval, as shown in the formula: This eliminates the interference of data of different magnitudes on the model, such as the difference in magnitude between the coordinate value "100 meters" and the priority "5 levels", so that continuous features and discrete features can be calculated together.

[0027] For F env The processing employs Kalman filtering for noise reduction to eliminate sensor noise and environmental interference, thereby improving the accuracy of environmental characteristics. The state equation is x. t = Ax t-1 + w t , where x t Let A be the actual state of the system at time t (including obstacle positions, dynamic trends, etc.), and let A be the state transition matrix (describing the state changes over time). t This represents process noise (uncontrollable random disturbances in the simulated environment, such as the effect of sudden airflow on the sensor). The observation equation is z. t = H xt + v t , where z t Let v be the noisy observation collected by the sensor at time t, H be the observation matrix (mapping the real state to the observation space), and v be the value of the sensor at time t. t To measure noise (reflecting sensor inherent errors, such as radar accuracy deviations), the Kalman gain K is calculated. t By fusing information from the state equation and the observation equation, filtering out noise, and obtaining the denoised environmental state, the F0 is ensured.env It can accurately reflect the actual environment.

[0028] For F Agent The processing employs an embedding layer transformation to map high-dimensional discrete features into low-dimensional dense vectors, reducing computational cost and capturing potential correlations between features. The embedding matrix E∈R is defined. n×d Where n is the number of feature categories (e.g., if there are 5 categories of agent roles, then n=5), and d is the embedding dimension (usually much smaller than n, e.g., d=3). The transformation process first converts the discrete features of the agent (such as roles) into one-hot vectors, and then maps them to low-dimensional vectors through matrix multiplication e = E•onehot (c). For example, the one-hot vectors (5 dimensions) of 5 roles are mapped to 3-dimensional vectors through the embedding matrix, which not only reduces the dimension (from 5 dimensions to 3 dimensions), but also reflects the role association through vector similarity (e.g., the vectors of the "grab" and "carry" roles are more similar).

[0029] By employing multi-dimensional feature encoding, the unstructured information of the task objective, environment, and agents is transformed into a unified structured vector, enabling the model to comprehensively capture key task information. One-hot encoding and normalization ensure compatibility between discrete and continuous features, providing a foundation for subsequent model computation. Kalman filtering improves the accuracy of environmental features and reduces noise interference with policy decisions. The embedding layer reduces feature dimensionality, improves computational efficiency, and simultaneously captures the potential relationships between agent features, providing more valuable input for multi-agent collaboration. In summary, task feature extraction is the foundation of the entire technical solution, and its processing quality directly affects the effectiveness of subsequent meta-training, task adaptation, and action generation.

[0030] For example, in the fintech business, within consumer credit risk control scenarios, platforms need to integrate multi-source data to construct user risk feature vectors. These include: basic user information (age, occupation, etc.), transaction data (frequency of purchases, fluctuations in amount), credit data (historical overdue records, debt ratio), and behavioral data (APP login duration, completeness of information provided). Discrete data (such as occupations categorized as "blue-collar / white-collar / freelance") is encoded using one-hot encoding, while continuous data (such as debt ratio) undergoes min-max normalization. The resulting vector is then concatenated to form a task feature vector containing "identity characteristics, transaction characteristics, and credit characteristics," providing input for subsequent risk assessment.

[0031] In the healthcare field, personalized diabetes diagnosis requires integrating multi-source patient data to construct a disease feature vector. This includes: clinical data (glucose monitoring values, BMI index), medical record data (history of complications, medication records), lifestyle data (exercise duration, dietary structure), and genetic data (relevant susceptibility gene markers). Noisy data (such as occasional abnormal blood glucose values) is denoised using Kalman filtering, and categorical data (such as diet type) is converted into low-dimensional vectors through an embedding layer, ultimately forming a comprehensive feature vector to support the diagnostic model's accurate characterization of an individual's condition.

[0032] In one embodiment, step S200 includes S201. Based on the task feature vector, convert the new task and the training task into a new task feature vector and a training task feature vector; S202. Calculate the cosine similarity and Manhattan distance between the new task feature vector and the training task feature vector; S203. Calculate the task difference based on the cosine similarity and Manhattan distance.

[0033] In this embodiment, a weighted fusion method of cosine similarity and Manhattan distance is used to calculate the difference between tasks. The specific formula is: in, Cosine similarity is used to measure the directional consistency between two vectors; ||Fnew-Fi||1 represents the Manhattan distance, used to measure numerical differences. new For the new task feature vector, F i Let be the feature vector of the i-th training task. The weights 0.6 and 0.4 can be determined through experimental optimization. The final difference D is mapped to the interval [0,1]. The smaller the D value, the more similar the new task is to the training task.

[0034] Cosine similarity primarily focuses on the matching degree of vector "direction". For example, if a new task has similar target type and agent role distribution to a training task (i.e., the vector direction is the same), even if the specific coordinates or values ​​differ, the cosine similarity will still be high, thus reflecting the similarity of task patterns. Manhattan distance, on the other hand, focuses on the difference in vector "values". For example, if the specific values ​​such as target coordinates and number of obstacles differ significantly between two tasks (even if the directions are the same), the Manhattan distance will increase. After normalization (dividing by the maximum possible distance), the proportion of this term in the formula will increase, thus reflecting the differences in details. This weighted fusion method combines the advantages of both indicators: cosine similarity captures the "essential pattern" similarity of tasks, while Manhattan distance captures the differences in "specific parameters", avoiding the one-sidedness that a single indicator may lead to. For example, using only the distance indicator may overlook the similarity of tasks with consistent patterns but minor numerical adjustments.

[0035] This quantization method transforms the differences in high-dimensional feature vectors into a scalar D within the [0,1] interval, thus intuitively reflecting the similarity between the new task and the training task (a smaller D indicates greater similarity). This quantization result can also dynamically adjust the learning rate, providing a basis for subsequent fine-tuning. When D is large (i.e., the new task differs significantly from the training task), the learning rate needs to be increased to accelerate strategy exploration; conversely, when D is small (i.e., the tasks are similar), the learning rate can be decreased for fine-tuning the strategy. This solves the problem of low adaptation efficiency caused by a fixed learning rate in traditional methods. For example, in an industrial robot collaboration scenario, if the new task is "assembling a new type of part" (different part type from the training task but similar assembly process), the D value is small, and the model will fine-tune the strategy with a low learning rate; while if the new task is "switching from assembly to handling" (significant difference in task mode), the D value is large, and the model will quickly explore new collaboration strategies with a high learning rate.

[0036] In summary, task difference calculation quantifies task similarity from multiple dimensions, providing a precise quantitative basis for adaptive adjustment of strategies, and is one of the core mechanisms for improving the speed of adaptation to new tasks.

[0037] For example, in the fintech business field: when a bank launches a "new urban resident exclusive loan product," it needs to calculate the difference between this new product and the historical "white-collar loan product." By extracting the feature vectors of both (new urban resident features include "migrant population marker - income stability - local social security contributions," and white-collar features include "fixed working years - housing provident fund contributions"), cosine similarity is used to measure the vector direction (such as the similarity in the "income stability" dimension), and Manhattan distance is used to calculate the numerical difference (such as the absolute difference in social security contribution duration). The final difference is D=0.7 (significant difference), providing a quantitative basis for subsequent model adjustments.

[0038] In the healthcare sector, during the diagnosis of pediatric pneumonia, it is necessary to calculate the difference between "novel variant viral pneumonia cases" and "traditional bacterial pneumonia cases." Feature vectors are extracted from both (novel cases include "fever peak - viral nucleic acid CT value - respiratory rate," while traditional cases include "white blood cell count - sputum culture results"). The difference is calculated using a formula that fuses similarity and distance, resulting in a difference of D=0.65. This clarifies the pattern differences between new and known cases, avoiding misdiagnosis. In one embodiment, step S300 includes S301. Based on the global state and joint actions, configure an actor network and a shared evaluator network for each agent; S302. Construct the basic network layer of the meta-training model through the actor network and the shared evaluator network; S303. Calculate the dynamic learning rate based on the controller parameters and the task difference using the activation function, and construct the meta-learning adjustment layer of the meta-training model based on the dynamic learning rate; S304. Calculate the collaborative weights of the multi-agents through the policy association matrix, and construct the collaborative constraint layer of the meta-training model based on the collaborative weights.

[0039] In this embodiment, a two-layer architecture integrating MAML (Model Independent Meta-Learning) and MADDPG (Multi-Agent Deep Deterministic Policy Gradient) is adopted, aiming to balance multi-agent collaboration capabilities with the task-adaptive capabilities of meta-learning. This architecture achieves rapid adaptation to new tasks while maintaining the stability of collaboration through the collaborative work of the base network layer and the meta-learning adjustment layer.

[0040] The base network layer is a core component of the two-layer architecture. Its main function is to preserve the collaborative capabilities of multi-agent reinforcement learning, enabling global value assessment and individual action generation. Specifically, the base network layer consists of two main parts: the Actor network and the shared Critic network. Each agent i is equipped with an independent Actor network, whose input is the local state observed by the agent itself, and whose output is the individual action (e.g., movement direction, operation performed), thus achieving distributed action decision-making. The shared Critic network is shared globally, with its input being the global state s = [s1, ..., s...]. n [A1, ..., A2](the set of local states of all agents) and joint action A = [A1, ..., A2] n The set of actions of all agents is used to output a global value assessment, which measures the overall benefit of the current joint actions and achieves centralized value judgment. This "distributed Actor + centralized Critic" model balances individual autonomy and global collaboration, allowing each agent to make flexible decisions while ensuring that actions conform to the overall goal through shared Critic.

[0041] The meta-learning adjustment layer is another core component of the two-layer architecture. Its main function is to introduce a meta-learning mechanism to dynamically adjust the learning rate, enabling the model to quickly adapt to task differences and overcoming the limitation of fixed learning rates in traditional multi-agent methods. The meta-learning adjustment layer includes a parameterized learning rate controller, which contains the parameter θ. α The input is the task variance D (from the "Task Variance Calculation" section), and the output is the dynamic learning rate through the Sigmoid function. This ensures that α∈(0,0.1] (to avoid policy oscillation caused by an excessively large learning rate). The policy update rule module is based on a dynamic learning rate. αThe meta-learning adjustment layer adjusts the parameter update magnitude in the inner loop (task-level adaptation) and fine-tuning stages to achieve "rapid exploration of tasks with large differences and fine-tuning of similar tasks." By binding task differences to the learning rate, the meta-learning adjustment layer enables the model to adaptively adjust the learning pace based on the similarity between new tasks and training tasks, laying the foundation for rapid policy optimization.

[0042] The collaborative mechanism of the two-layer architecture is the foundation for the interaction between the base network layer and the meta-learning adjustment layer. The base network layer provides a "basic policy framework" for multi-agent collaboration, while the meta-learning adjustment layer controls the speed and magnitude of parameter updates within this framework through a dynamic learning rate. For example, when the new task differs significantly from the training task (high D value), the meta-learning adjustment layer outputs a high learning rate, enabling the Actor and Critic parameters of the base network layer to quickly adjust to adapt to the new task. Conversely, when the tasks are similar (low D value), a low learning rate is output, focusing only on fine-tuning the parameters and preserving existing collaborative experience. This mechanism solves the problem of fixed learning rates in traditional multi-agent reinforcement learning (such as MADDPG), where adjustments are either too slow (for similar tasks) or oscillate and become unstable (for tasks with significant differences). By achieving dynamic learning rate adaptation through the meta-learning adjustment layer, both speed and stability of adaptation are balanced.

[0043] Furthermore, the shared Critic in the base network layer ensures that the agent's actions align with the global objective, avoiding individual decision-making conflicts and resolving the issue of "insufficient collaborative synchronization." The dynamic learning rate mechanism in the meta-learning adjustment layer enables the model to flexibly adjust the policy update rhythm according to task differences, addressing the problem of "poor task adaptability." This two-layer architecture also lays the foundation for subsequent dual-loop training, where the inner loop rapidly adjusts intra-task parameters based on the base network layer parameters, while the outer loop optimizes cross-task generalization capabilities through the meta-learning adjustment layer.

[0044] A policy association matrix is ​​a tool used to quantify the degree of dependency between different agents, identifying which agents need to focus their attention on each other's actions. The policy association matrix is ​​as follows: Where, the element M in the matrix ij This represents the collaboration weight of agent i with agent j, with a value ranging from [0,1]. This weight consists of two parts: first, the state similarity between agents i and j (e.g., positional distance, task objective overlap, etc.; a larger value indicates greater state similarity); and second, the denominator is the sum of the state similarities between agent i and all other agents, used for row normalization to ensure that the sum of elements in each row of the matrix is ​​1. For example, in a logistics AGV collaboration scenario, if the AGV... i With AGV j If the distances are close and the target shelves are adjacent, then M ijA high value indicates that both need to coordinate obstacle avoidance; conversely, if the AGV... i With AGV k If the distance is far, then M ik A lower weight indicates a lower priority for collaboration. The higher the weight, the greater the impact of agent j's actions on agent i's final decision, thus ensuring that the actions of "key collaborators" are given priority consideration.

[0045] Furthermore, action correction involves adjusting the actions of collaborating partners based on the individual decisions of the agent to ensure consistency in overall actions. The formula is: Where h is the hidden layer output of the actor network, W and b are the output layer parameters, β is the cooperation coefficient, and σ is the activation function (tanh for continuous actions, softmax for discrete actions). For example, in industrial assembly, if the original action of agent i (grabbing robot) is "move to the left", but through M ij If agent j (the welding robot) is found to be on the left (indicating it is about to perform a welding action), then after the action is corrected, A i It might be adjusted to "move right" to avoid collisions. This mechanism preserves the individual agent's independent judgment of the environment (h term) while solving the problem of "action conflicts caused by independent decisions of each agent" in traditional methods by forcing reference to the state of partners.

[0046] Through the policy association matrix, agents can "perceive who the key partners are," while the action correction mechanism ensures that "actions refer to partner decisions," thus avoiding the problem of "inconsistent rhythm caused by independent policy updates" in traditional multi-agent methods (such as some agents adjusting quickly while others lag). Furthermore, when tasks change (such as target switching or the appearance of obstacles), the policy association matrix updates in real time with the agents' states, and the action correction mechanism dynamically adjusts the collaboration weights, ensuring that collaboration relationships change synchronously with the environment, thereby enhancing the system's dynamic environmental adaptability. Simultaneously, even if individual agents' local decisions are biased, by fusing the actions of multiple partners (especially high-weight partners), the impact of erroneous decisions can be reduced, the risk of task interruption can be decreased, and the system's robustness can be improved. In summary, the collaboration constraint layer, through the process of "quantifying dependencies → correcting action decisions," upgrades independent agent decisions to collaborative decisions, which is the core guarantee for achieving efficient multi-agent collaboration.

[0047] For example, in the fintech business field, in cross-border payment anti-fraud scenarios, a meta-training model needs to be built based on task differences. Fraud patterns vary significantly across different regions. First, the difference in fraud tasks for each region is calculated (e.g., D=0.8 for Southeast Asia and Europe). Then, the model architecture is designed based on the difference: the basic network layer uses a shared critic network to evaluate global fraud risk, the meta-learning adjustment layer dynamically outputs the learning rate based on the difference (the learning rate increases to 0.05 when the difference is large and decreases to 0.01 when the difference is small), and the collaborative constraint layer quantifies the correlation weights of "transaction location-amount-frequency" to ensure that the model adapts to the fraud characteristics of different regions.

[0048] In the healthcare business, for early screening of multiple cancer types, a meta-training model is constructed based on task variability. The screening characteristics for different cancers (lung cancer, breast cancer, and colorectal cancer) vary significantly (e.g., lung cancer is associated with "nodule size - smoking history," and breast cancer with "tumor hardness - family history"). After calculating the variability of each cancer task, the model's base network layer uses a shared critic network to evaluate the screening value. The meta-learning adjustment layer adjusts the parameter update rhythm based on the variability (e.g., D=0.7 for lung cancer and breast cancer), and the collaborative constraint layer quantifies the association weights of "clinical features - imaging features - pathological features," enabling the model to cover the screening needs of multiple cancer types.

[0049] In one embodiment, step S400 includes S401. Obtain sampling tasks from the task distribution, and extract task data from the sampling tasks; S402. The meta-training model is trained using the task data to obtain the inner loop result; S403. The meta-training model is trained using the inner loop results to obtain the optimal meta-parameters.

[0050] This embodiment proposes a dual-loop training process to address the problem of rapid adaptation and stable collaboration among multiple agents in dynamic task scenarios. Its core idea is to enable agents to quickly adapt to new tasks and achieve efficient collaboration through alternating inner and outer loops.

[0051] In the inner loop, the core task is to simulate how an agent quickly adjusts its strategy to adapt to the task requirements in a single specific task, and to ensure synchronization of actions among agents through cooperative constraints. Specifically, firstly, a task τ ~ p(τ) is randomly sampled from a preset task distribution p(τ). This task distribution is a collection of various task scenarios, such as multi-agent cooperative tasks with different target locations and different obstacle distributions, aiming to ensure that the training process covers diverse scenarios. For the currently sampled task, two sets of core parameters are initialized: the Actor network parameters for each agent i (used to generate individual actions) and the shared Critic network parameters (used to evaluate the global action value). Subsequently, task data is acquired, i.e., interaction data such as the environmental state, agent actions, and rewards under the task are collected. This data will serve as the basis for subsequent parameter updates. Through three-step parameter iteration, the agent's policy can quickly adapt to the current task, and its update formula is: ,in, It is the inner loop learning rate, τ base This is the baseline task. Loss function. This includes two terms: the first is the TD error, used to ensure the effectiveness of the policy in the current task; the second is a cooperation regularization term, where λ1 constrains the consistency of the agent's gradient direction, thereby reducing policy conflicts. After the update is complete, the temporary parameters (Actor parameters adapted to the current task) and the updated Critic parameters, as well as task performance metrics (such as the average reward and cooperation success rate of the task), are saved. These metrics will be used for the meta-parameter optimization of the outer loop.

[0052] The core of the outer loop is to optimize the "meta-parameters" (i.e., the initial parameters of the base network) based on the results of the inner loop across multiple tasks. This ensures that the parameters can support rapid inner loop adaptation for all tasks, thus achieving "learning to learn." Specifically, m different tasks (e.g., m = 10) are sampled from the task distribution p(τ). After executing the inner loop, the inner loop results for these m tasks are collected, including the temporary parameters and inner loop loss for each task. The meta-loss, i.e., the average of the inner loop losses for these m tasks, is calculated using the following formula: The meta-loss evaluates the overall performance of the base parameters in "cross-task adaptation." A smaller meta-loss indicates that the base parameters can quickly adapt to different tasks through the inner loop. The meta-parameters (i.e., initial parameters) of the base network are optimized based on the meta-loss, and their update formula is: , , Here, α2 is the outer loop learning rate, which is typically smaller than the inner loop learning rate to ensure more robust updates of the meta-parameters. The optimizer uses the RMSprop optimizer, an adaptive learning rate algorithm that adjusts the gradient through exponential moving averages to avoid gradient oscillations, thereby accelerating the convergence of the meta-parameters.

[0053] The inner and outer loops alternate. After each outer loop updates the meta-parameters, the new meta-parameters become the initial parameters for the next round of the inner loop. Through multiple iterations, the meta-parameters gradually acquire the ability to adapt to new tasks with only a few inner loop steps.

[0054] This technical solution forces all agents to have consistent policy gradient directions through a collaborative regularization term in the inner loop, avoiding action conflicts caused by independent updates from each agent in traditional multi-agent methods (e.g., agents A and B adjusting their pace asynchronously when tasks change, leading to collisions). Furthermore, the outer loop optimizes meta-parameters through multi-task loss, making the basic parameters a "universal initial point." New tasks do not require retraining from scratch; they can be quickly adapted with only a few inner loop steps, solving the shortcomings of traditional methods that "only optimize single-task performance and require retraining when changing tasks." Through the nesting of the inner loop's "learning a single task" and the outer loop's "learning how to quickly learn new tasks," agents master "methods for adapting to task changes," rather than simply memorizing policies for specific tasks, ultimately achieving efficient collaboration in dynamic scenarios.

[0055] In summary, the dual-loop training process is the core of this technical solution's ability to achieve "rapid adaptation to new tasks + stable collaboration." By combining task-level adaptation in the inner loop with meta-parameter generalization in the outer loop, this solution overcomes the limitations of traditional multi-agent reinforcement learning and meta-learning.

[0056] For example, in the fintech business, the optimal meta-parameters are obtained through dual-loop training in the optimization of quantitative trading strategies. Ten typical market tasks (such as "high-volatility bear market" and "low-volatility oscillating market") are sampled from historical market data of "bull market / bear market / sideways market". The inner loop updates the actor network parameters (with the loss function including "return TD error + strategy co-regulation term") in three iterations for a single task (such as the "2022 bear market"), ensuring the strategy's effectiveness in that market. The outer loop collects the results of the inner loop for the ten tasks and updates the meta-parameters using the RMSprop optimizer, ultimately obtaining optimal meta-parameters adaptable to multiple markets (such as the basic threshold for position adjustment and stop-loss trigger conditions), eliminating the need for retraining in new markets.

[0057] In the healthcare business, for medical image recognition (such as CT tumor detection), a dual-loop training and optimization model is used. Eight task categories (such as "lung CT - low dose" and "brain CT - contrast enhancement") are sampled from image data from different hospitals and different equipment. The inner loop focuses on a single task category (such as "lung CT in primary hospitals"), updating the actor network parameters (the loss function includes "recognition accuracy TD error + image feature co-regulation term") to adapt to the noise and resolution characteristics of this type of image. The outer loop integrates the training results of the eight task categories, optimizes the meta-parameters, and obtains general parameters that are compatible with images from multiple scenarios. Subsequent image data from new hospitals only require minor adjustments for adaptation.

[0058] In one embodiment, step S500 includes... S501. The optimal meta-parameters are used as the initial parameters for the new task, and the initial parameters are used to update the task difference between the new task and the training task. S502. According to the filtering rules, obtain several task difference degrees, and calculate the task average value based on the several task difference degrees; S503. Determine the fine-tuning learning rate through the meta-learning adjustment layer based on the average value of the task. S504. Update the final layer parameters of the actor network by fine-tuning the learning rate, and fine-tune the meta-training model according to the final layer parameters.

[0059] In this embodiment, the optimal meta-parameters obtained during the meta-training phase are first loaded. These parameters are the basic network parameters optimized by the outer loop, including the core parameters of the Actor and Critic networks. These meta-parameters, through multi-task learning (dual-loop training), acquire "general collaborative knowledge" across tasks, such as basic cooperation patterns between agents and common processing methods for environmental features. Taking an industrial robot as an example, the meta-parameters may have already learned the "basic motion coordination logic of the grasping and welding robots in assembly tasks." Therefore, when facing new tasks (such as assembling new model parts), this logic can be directly reused without relearning. Compared to traditional multi-agent methods (where new tasks require training from random parameters), this step can shorten the adaptation time by more than 80%, effectively solving the problem of "extremely low collaboration efficiency in the new task startup phase."

[0060] To ensure rapid adaptation to new tasks and maintain agent synchronization, it is necessary to quantify the differences between the new task and the training task and dynamically adjust the strategy. Specifically, this involves calculating the new task τ. new The difference between the training set and all tasks is D1,...,D. m (The method for calculating the difference is the same as above; the smaller the value, the more similar the similarity). From these difference values, select the 5 smallest (i.e., the 5 training tasks most similar to the new task), and calculate the average value D.avg = (1 / 5)ΣD k This D avg It can reliably reflect the overall similarity between the new task and the training task, avoiding the randomness of differences in a single task. For example, if the new task is "warehouse AGV sorting new categories of goods", and there are 5 tasks in the training set that "sort similar categories of goods" (small D value), then D... avg The lower value indicates that the new task is similar to the training task pattern.

[0061] Meta-learning adjustment layer according to D avg Output the fine-tuned learning rate, the formula is:

[0062] Where L final Loss due to new tasks (same as inner loop L) inner Only update the parameters of the last layer of the Actor network (to reduce computation), iterating 1-3 times. If D avg Large differences between the new task and the training task, such as switching from "assembly" to "transportation"), result in a higher learning rate and accelerated strategy exploration (rapid adjustments to adapt to the new mode); if D avg Small tasks (similar to other tasks, such as minor adjustments to target coordinates), low learning rates, and fine-tuned optimization strategies (avoiding drastic adjustments that disrupt existing collaborative patterns). Based on new task data, only the parameters of the last layer (output layer) of the Actor network are updated, iterating 1-3 times. Only the last layer is updated because the earlier layers of the Actor network are responsible for extracting general features (such as environment state and agent role), while the last layer is responsible for outputting specific actions (such as movement direction and execution force). Targeted updates to the last layer can reduce computation by more than 60% while maintaining adaptation effectiveness. The core function is to fine-tune only the output layer of actions directly related to the new task, balancing "adaptation accuracy" and "computational efficiency," based on reusing the "general knowledge" of meta-parameters.

[0063] After fine-tuning, calculate the cooperative error e of all agent actions, using the formula: Where C(n,2) is the number of pairwise combinations of agents (n is the total number of agents), A i A j Let e ​​represent the actions of agents i and j. If e > ε (ε is a threshold, such as 0.1), then fine-tuning is repeated once. Its core function is to verify the synchronization of agent actions. For example, if after fine-tuning, robot A plans to "accelerate" while robot B plans to "decelerate" (e value is large), it indicates a policy conflict, requiring readjustment to correct it. This addresses the problem of "asynchronous policies after individual fine-tuning" in traditional meta-learning methods in multi-agent scenarios.

[0064] The new task adaptation phase achieves three core objectives through a process of "reusing meta-parameters → quantifying differences → dynamic fine-tuning → verifying synergy": rapid adaptation (reusing general knowledge, requiring only 1-3 steps of fine-tuning to adapt to new tasks, overcoming the time-consuming retraining of traditional methods); efficient computation (only updating the last layer parameters of the Actor, significantly reducing computational costs); and stable collaboration (dynamically adjusting the learning rate and verifying collaborative errors through differences to ensure synchronized agent actions and avoid policy conflicts). Ultimately, this phase enables the agent to quickly enter a stable collaborative state after the start of a new task, significantly improving adaptability in dynamic scenarios, such as switching product models on industrial production lines or adjusting shelf layouts in logistics warehouses.

[0065] For example, in the fintech business field: when a payment platform expands to the new scenario of "rural inclusive finance," it fine-tunes its anti-fraud model. First, it loads the optimal meta-parameters obtained from meta-training (fraud identification parameters adapted for urban users), and calculates the difference between the new task (rural users) and the training task (urban users) (e.g., significant difference in the "transaction frequency - device type" dimension, D). avg =0.6), and the fine-tuning learning rate α=0.03 (higher than 0.01 for similar tasks) is determined through the meta-learning adjustment layer; only the parameters of the last layer of the actor network are updated (focusing on new features such as "common payment channels for rural users"). After two iterations, the collaborative error (such as the conflict rate of different risk control rules e=0.08<0.1) is calculated to complete the fine-tuning, which improves the fraud detection rate of the model in rural scenarios from 65% to 89%.

[0066] In the healthcare field: For the new task of "diagnosing sepsis in premature infants," the general diagnostic model was fine-tuned. The optimal meta-parameters from the meta-training (parameters adapted to typical children) were loaded, and the difference between the new task and the training task was calculated (premature infants have greater fluctuations in physiological indicators, D...). avg =0.55), and determined the fine-tuning learning rate α=0.02; only update the parameters of the actor network output layer (focusing on the correlation between "gestational age-weight-inflammatory markers"), and after 3 iterations, verify the collaborative error (the deviation between the diagnostic conclusion and the clinical expert e=0.07<0.1). After fine-tuning, the model's accuracy in early identification of sepsis in preterm infants is improved by 40%.

[0067] In one embodiment, step S600 includes S601. Obtain environmental data for a preset time and extract temporal features through a long short-term memory network; S602. Input the time-series features into the meta-training model to generate action distribution probabilities; S603. Calculate the specific action based on the sampling algorithm and the action distribution probability; S604. The specific actions are collaboratively modified through the policy association matrix, and the multi-agents are made to execute the specific actions after the collaborative modification.

[0068] In this embodiment, real-time collaborative actions are generated through temporal feature capture, action sampling, and collaborative correction. A feedback mechanism is then used to dynamically optimize the strategy, ensuring efficient task execution by multiple agents in complex environments. Specifically, through temporal modeling, probability sampling, and collaborative fusion, real-time environmental information is transformed into synchronized actions of the agents, avoiding short-sighted decision-making and action conflicts. First, the real-time environmental states {s} at three consecutive time points are collected. t , s t-1 , s t-2} (where s t The current state (including obstacle locations, real-time actions of other agents, communication quality, etc.) is then used to extract temporal features via an LSTM (Long Short-Term Memory) network, with the formula h. t = LSTM (h t-1 , s t ). Where h t The hidden state at time t integrates key information from both the current and historical states. This core function addresses the short-sighted decision-making problem inherent in traditional methods that rely solely on the current state. For example, in autonomous driving scenarios, relying solely on the current frame (s...)... t It may be impossible to predict the braking trend of the vehicle in front, while LSTM captures the braking trend through h. t-2 -s t-1 s t By observing the speed change trend, one can slow down and avoid collisions in advance; in industrial robot collaboration, by identifying the "changes in the movement speed of the conveyor belt" through time sequence characteristics, the timing of grasping can be adjusted in advance to avoid missed grasps or collisions.

[0069] Each agent's Actor network is based on the temporal features h output by LSTM. t This generates an action distribution (describing the probability of each possible action, such as the probability distribution of movement direction and force). The specific action is then obtained from the distribution using the Gumbel-Softmax sampling algorithm, with the formula a. i = softmax ( (logπ i + g i ) / τ ), where g i~Gumbel (0,1) is noise sampled from the Gumbel distribution, used to increase the exploratory nature of actions (avoiding the agent from getting stuck in a fixed policy); τ is a temperature parameter (controlling randomness; the smaller the tau, the more concentrated the sampling is on high-probability actions, and the stronger the determinism; the larger the τ, the higher the randomness, suitable for exploring unknown environments). Gumbel-Softmax sampling ensures the "differentiability" of actions. Compared with traditional random sampling (which is not differentiable and cannot be used for parameter updates), its actions can backpropagate gradients, supporting subsequent policy optimization. At the same time, it balances "exploration" and "exploitation," both trying new actions through noise (such as detouring paths around unknown obstacles) and selecting high-probability actions based on existing experience (such as commonly used cooperative routes), improving adaptability in complex environments.

[0070] The original action A obtained from sampling i Final adjustments are made, incorporating the effects of collaborating partners' actions to ensure coordinated movements. For example, in a logistics AGV cluster, if the AGVs... i The original action was "straight ahead", but M ij Display AGV j (A high-powered partner) is approaching from the right (A) j (Corrected to "turn left"), A i It could be adjusted to "slow down and avoid collisions." Compared to the traditional method of "each agent independently outputting actions," this step integrates partner actions, upgrading individual decision-making to global collaborative decision-making, reducing the conflict rate by more than 70%.

[0071] By evaluating action effectiveness through multi-dimensional rewards and triggering emergency adjustments when collaboration fails, the system's robustness in complex environments is enhanced. The reward value r at each time step... t Combining the three dimensions, the formula is r t = ω1×r progress + ω2×r collab + ω3×r penalty， Where, r progress Rewards for task completion progress (such as the percentage of distance goods transported, the percentage of assembled parts completed; higher values ​​indicate smoother task progress); r collab For collaboration quality rewards (e.g., action synchronization degree 1 / e, where e is the collaboration error; the smaller the error, the higher the reward); r penalty The penalty is negative (for collisions, timeouts, communication interruptions, etc.; for example, a collision results in a reward of -5). Weights ω1 = 0.5 (emphasizing task progress), ω2 = 0.3 (encouraging collaboration), and ω3 = -0.2 (suppressing erroneous actions) can be fine-tuned according to the scenario. Its core function is to guide the agent to "focus on both task completion and the quality of collaboration." For example, if the agent ignores collaboration to complete the task quickly, resulting in a collision, the overall reward r...t This will reduce [their] prices, forcing them to adjust their strategies.

[0072] Continuously monitor reward value r t If r is satisfied in 3 consecutive steps t <0.3×r max (r) max To determine the maximum possible reward for the task scenario, the current policy is deemed ineffective, and emergency fine-tuning is initiated: the number of training steps in the inner loop is increased from the usual 1-3 steps to 5 steps (deepening the policy adjustment); the learning rate α3 is increased by 50% (e.g., from 0.01 to 0.015, accelerating policy updates). This mechanism addresses the problem of "cooperation failure caused by sudden interference" in complex environments. For example, sudden sensor noise may cause misjudgment of the environmental state (e.g., misidentifying a stationary obstacle as moving), in which case the reward r... t The rate of interruption can drop sharply, and emergency fine-tuning can quickly correct the strategy to avoid task interruption. Compared with the traditional method of "fixed strategy execution to the end", this mechanism gives the system "self-repair" capability, reducing the task interruption rate by more than 60%.

[0073] The output phase achieves real-time performance through the synergy of an "action generation mechanism" and a "feedback adjustment mechanism": LSTM timing encoding and efficient sampling ensure action generation latency is below milliseconds, meeting the needs of dynamic scenarios; synergy: a collaborative correction mechanism synchronizes agent actions, resolving conflict issues inherent in traditional methods; robustness: feedback and emergency fine-tuning mechanisms quickly correct for environmental interference or policy failure, ensuring stable task execution. Finally, this phase transforms the policies obtained from meta-training and fine-tuning into actual collaborative actions, a crucial step in the practical application of the technology, suitable for scenarios such as robot collaboration in industrial production lines and vehicle platooning in intelligent transportation.

[0074] For example, in the fintech business, real-time cross-border remittance risk control requires the system to dynamically generate risk control actions. This involves acquiring real-time environmental data such as remittance amount, recipient region, remittance frequency, and account history (number of transactions in the past hour), extracting time-series features (e.g., the trend of "remittances to the same high-risk region three times within 10 minutes") using LSTM; the actor network outputs action distributions (probabilities of "allow / warn / intercept"), which are then sampled using Gumbel-Softmax to obtain the "warn" action. Finally, a strategy association matrix is ​​used to fuse partner actions related to "remittances from the same account" (e.g., abnormal remittances from related accounts in the past day), ultimately executing the "suspend remittance + manual review" action, reducing the cross-border fraud interception time from 30 minutes to 2 minutes.

[0075] In the healthcare field, real-time generation of surgical actions is required in robot-assisted laparoscopic surgery. Real-time acquisition of environmental data is crucial: intraoperative laparoscopic images (organ position changes), force sensor data (instrument-tissue contact pressure), and patient vital signs (heart rate, blood oxygenation). Temporal features (such as the "positional movement trend of tissue bleeding points") are extracted using LSTM. The actor network outputs action distributions (the probability of "clamping force - movement angle"), which are sampled to obtain basic actions. These are then fused using a policy association matrix to determine the "assistant robot's field of vision adjustment" action, ultimately executing a "0.5N clamping force + 30° turning for hemostasis" action, improving surgical bleeding control accuracy by 30%.

[0076] In one embodiment, a multi-agent cooperation device based on meta-reinforcement learning is provided, which corresponds one-to-one with the multi-agent cooperation method based on meta-reinforcement learning described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-agent collaborative device based on meta-reinforcement learning of the present invention. The modules include a feature vector module 10, a task difference module 20, a model construction module 30, a meta-parameter module 40, a model fine-tuning module 50, and an action execution module 60. Detailed descriptions of each functional module are as follows. Feature vector module 10 is used to acquire multi-source information data and construct task feature vectors based on the multi-source information data; The task difference module 20 is used to calculate the task difference between the new task and the training task based on the task feature vector. Model building module 30 is used to build a meta-training model based on the task difference. Meta-parameter module 40 is used to acquire sampling tasks and perform double-loop training on the meta-training model through sampling tasks to obtain optimal meta-parameters. The model fine-tuning module 50 is used to take the optimal meta-parameters as the initial parameters of the new task, update the task difference between the new task and the training task according to the initial parameters, determine the fine-tuning learning rate according to the updated task difference, and use the fine-tuning learning rate to fine-tune the meta-training model according to the parameters of the last layer. The action execution module 60 is used to acquire environmental data for a preset time, input the environmental data into the fine-tuned meta-training model to obtain the action distribution probability, determine the specific action based on the action distribution probability, and enable the multi-agent to execute the specific action.

[0077] In one embodiment, the feature vector module 10 includes The target feature sub-vector unit is used to acquire task target data and construct target feature sub-vectors based on the task target data. An environmental feature sub-vector unit is used to acquire environmental data and construct an environmental feature sub-vector based on the environmental data. The agent feature sub-vector unit is used to acquire agent data and construct agent feature sub-vectors based on the agent data; The task feature vector unit is used to concatenate the target feature sub-vector, the environment feature sub-vector, and the agent feature sub-vector to obtain the task feature vector.

[0078] In one embodiment, the task difference module 20 includes A feature vector unit is used to convert the new task and the training task into a new task feature vector and a training task feature vector based on the task feature vector; A vector computation unit is used to calculate the cosine similarity and Manhattan distance between the new task feature vector and the training task feature vector; The task difference unit is used to calculate the task difference based on the cosine similarity and Manhattan distance.

[0079] In one embodiment, the model building module 30 includes A meta-training model unit is used in the meta-training model, which includes a base network layer, a meta-learning adjustment layer, and a cooperative constraint layer. The network configuration unit is used to configure the actor network and the shared evaluator network for each agent based on the global state and joint actions. A basic network layer unit is used to construct the basic network layer of the meta-training model through the actor network and the shared evaluator network; The meta-learning adjustment layer unit is used to calculate the dynamic learning rate based on the controller parameters and the task difference degree through the activation function, and to construct the meta-learning adjustment layer of the meta-training model based on the dynamic learning rate; The collaboration constraint layer unit is used to calculate the collaboration weights of multiple agents through the policy association matrix and to construct the collaboration constraint layer of the meta-training model based on the collaboration weights.

[0080] In one embodiment, the meta-parameter module 40 includes A sampling task unit is used to obtain sampling tasks from the task distribution and extract task data from the sampling tasks. The inner loop unit is used to perform inner loop training on the meta-training model using the task data to obtain the inner loop result; The outer loop unit is used to perform outer loop training on the meta-training model using the results of the inner loop, so as to obtain the optimal meta-parameters.

[0081] In one embodiment, the model fine-tuning module 50 includes: The task difference unit is used to take the optimal meta-parameters as the initial parameters of the new task, and the initial parameters update the task difference between the new task and the training task. The average value unit is used to obtain several task differences according to the filtering rules, and to calculate the task average value based on the several task differences. A learning rate unit is used to determine a fine-tuned learning rate based on the average value of the task through a meta-learning adjustment layer. The fine-tuning unit is used to update the final layer parameters of the actor network through the fine-tuning learning rate, and to fine-tune the meta-trained model according to the final layer parameters.

[0082] In one embodiment, the action execution module 60 includes The environmental data unit is used to acquire environmental data over a preset time and extract temporal features through a long short-term memory network. An action probability unit is used to input the temporal features into a meta-training model to generate action distribution probabilities; The action determination unit is used to calculate the specific action based on the sampling algorithm and the action distribution probability. An action execution unit is used to collaboratively modify specific actions through a policy association matrix and enable the multi-agent to execute the modified specific actions.

[0083] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side multi-agent cooperative method based on meta-reinforcement learning.

[0084] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the user-side functions or steps of a multi-agent cooperative method based on meta-reinforcement learning. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: Acquire multi-source information data and construct a task feature vector based on the multi-source information data; Calculate the task difference between the new task and the training task based on the task feature vector; A meta-training model is constructed using the aforementioned task differences; A sampling task is obtained, and the meta-training model is trained in a double loop using the sampling task to obtain the optimal meta-parameters. The optimal meta-parameters are used as the initial parameters for the new task, and the task difference between the new task and the training task is updated according to the initial parameters. The fine-tuning learning rate is determined according to the updated task difference, and the meta-training model is fine-tuned according to the final layer parameters using the fine-tuning learning rate. The system acquires environmental data for a preset time period and inputs the environmental data into the fine-tuned meta-training model to obtain the action distribution probability. Based on the action distribution probability, the system determines the specific action and causes the multi-agent to execute the specific action.

[0085] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor. Acquire multi-source information data and construct a task feature vector based on the multi-source information data; Calculate the task difference between the new task and the training task based on the task feature vector; A meta-training model is constructed using the aforementioned task differences; A sampling task is obtained, and the meta-training model is trained in a double loop using the sampling task to obtain the optimal meta-parameters. The optimal meta-parameters are used as the initial parameters for the new task, and the task difference between the new task and the training task is updated according to the initial parameters. The fine-tuning learning rate is determined according to the updated task difference, and the meta-training model is fine-tuned according to the final layer parameters using the fine-tuning learning rate. The system acquires environmental data for a preset time period and inputs the environmental data into the fine-tuned meta-training model to obtain the action distribution probability. Based on the action distribution probability, the system determines the specific action and causes the multi-agent to execute the specific action.

[0086] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0087] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0088] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0089] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-agent cooperation method based on meta-reinforcement learning, characterized in that, Includes the following steps: Acquire multi-source information data and construct a task feature vector based on the multi-source information data; Calculate the task difference between the new task and the training task based on the task feature vector; A meta-training model is constructed using the aforementioned task differences; A sampling task is obtained, and the meta-training model is trained in a double loop using the sampling task to obtain the optimal meta-parameters. The optimal meta-parameters are used as the initial parameters for the new task, and the task difference between the new task and the training task is updated according to the initial parameters. The fine-tuning learning rate is determined according to the updated task difference, and the meta-training model is fine-tuned according to the final layer parameters using the fine-tuning learning rate. The system acquires environmental data for a preset time period and inputs the environmental data into the fine-tuned meta-training model to obtain the action distribution probability. Based on the action distribution probability, the system determines the specific action and causes the multi-agent to execute the specific action.

2. The multi-agent cooperative method based on meta-reinforcement learning as described in claim 1, characterized in that, The process of acquiring multi-source information data and constructing a task feature vector based on the multi-source information data includes: Obtain task target data, and construct target feature sub-vectors based on the task target data; Acquire environmental data and construct environmental feature sub-vectors based on the environmental data; Acquire agent data, and construct agent feature sub-vectors based on the agent data; The target feature vector, the environment feature vector, and the agent feature vector are concatenated to obtain the task feature vector.

3. The multi-agent cooperation method based on meta-reinforcement learning as described in claim 1, characterized in that, The step of calculating the task difference between the new task and the training task based on the task feature vector includes: Based on the task feature vector, the new task and the training task are transformed into a new task feature vector and a training task feature vector, respectively. Calculate the cosine similarity and Manhattan distance between the new task feature vector and the training task feature vector; The task difference is calculated based on the cosine similarity and Manhattan distance.

4. The multi-agent cooperative method based on meta-reinforcement learning as described in claim 1, characterized in that, The construction of the meta-training model using the task difference includes: Based on the global state and joint actions, configure an actor network and a shared evaluator network for each agent; The basic network layer of the meta-training model is constructed using the actor network and the shared evaluator network; The dynamic learning rate is calculated using an activation function based on the controller parameters and the task difference, and the meta-learning adjustment layer of the meta-training model is constructed based on the dynamic learning rate. The collaborative weights of the multi-agents are calculated using the policy association matrix, and a collaborative constraint layer of the meta-training model is constructed based on these collaborative weights.

5. The multi-agent cooperation method based on meta-reinforcement learning as described in claim 1, characterized in that, The step of acquiring a sampling task and performing double-loop training on the meta-training model through the sampling task to obtain optimal meta-parameters includes: Obtain sampled tasks from the task distribution, and extract task data from the sampled tasks; The meta-training model is trained using the task data to obtain the inner loop results. The outer loop training is performed on the meta-training model using the results of the inner loop to obtain the optimal meta-parameters.

6. The multi-agent cooperation method based on meta-reinforcement learning as described in claim 4, characterized in that, The step of using the optimal meta-parameters as initial parameters for the new task, updating the task difference between the new task and the training task based on the initial parameters, determining the fine-tuning learning rate based on the updated task difference, and fine-tuning the meta-training model using the fine-tuning learning rate and based on the final layer parameters includes: The optimal meta-parameters are used as the initial parameters for the new task, and the initial parameters are used to update the task difference between the new task and the training task. Based on the filtering rules, several task differences are obtained, and the average task value is calculated based on the several task differences. The fine-tuning learning rate is determined by a meta-learning adjustment layer based on the average value of the task. The final layer parameters of the actor network are updated by fine-tuning the learning rate, and the meta-training model is fine-tuned based on the final layer parameters.

7. The multi-agent cooperative method based on meta-reinforcement learning as described in claim 1, characterized in that, The process of acquiring environmental data for a preset time, inputting the environmental data into a fine-tuned meta-training model to obtain action distribution probabilities, determining specific actions based on the action distribution probabilities, and having the multi-agent execute the specific actions includes: Acquire environmental data for a preset time period and extract temporal features through a long short-term memory network; The temporal features are input into the meta-training model to generate action distribution probabilities; The specific action is calculated based on the sampling algorithm and the probability distribution of the action. The specific actions are collaboratively modified using a policy association matrix, and the multi-agents then execute the modified actions.

8. A multi-agent cooperative device based on meta-reinforcement learning, characterized in that, The multi-agent cooperative device based on meta-reinforcement learning includes: The feature vector module is used to acquire multi-source information data and construct task feature vectors based on the multi-source information data. The task difference module is used to calculate the task difference between the new task and the training task based on the task feature vector. The model building module is used to build a meta-training model based on the task differences. The meta-parameter module is used to acquire sampling tasks and perform double-loop training on the meta-training model through the sampling tasks to obtain the optimal meta-parameters. The model fine-tuning module is used to take the optimal meta-parameters as the initial parameters of the new task, update the task difference between the new task and the training task according to the initial parameters, determine the fine-tuning learning rate according to the updated task difference, and use the fine-tuning learning rate to fine-tune the meta-training model according to the parameters of the last layer. The action execution module is used to acquire environmental data for a preset time, input the environmental data into the fine-tuned meta-training model to obtain the action distribution probability, determine the specific action based on the action distribution probability, and enable the multi-agent to execute the specific action.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a multi-agent cooperation program based on meta-reinforcement learning stored in the memory and executable on the processor. When executed by the processor, the multi-agent cooperation program based on meta-reinforcement learning implements the steps of the multi-agent cooperation method based on meta-reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a multi-agent cooperation program based on meta-reinforcement learning, which, when executed by a processor, implements the steps of the multi-agent cooperation method based on meta-reinforcement learning as described in any one of claims 1-7.

Images