Multi-modal learning fusion agent system and learning method thereof

Abstract

The invention particularly relates to a multi-modal learning fusion agent system and a learning method thereof. The multi-modal learning fusion agent system comprises a core model module, a data preprocessing module, a task scheduling module, a resource management module, an interface module and a configuration and maintenance module. According to the multi-modal learning fusion agent system and the learning method thereof, the system can monitor and analyze data and task feedback in the running process in real time, the weight and parameters of the model are automatically adjusted so as to keep the optimal performance and adaptability, the importance of resource management is emphasized, and the learning efficiency is improved. Therefore, the intelligent agent system can maintain high performance, has the capability of quickly responding to new conditions, and can meet the requirements of a modern intelligent agent system for running in a complex and changeable environment.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent devices, and in particular to a multimodal learning fusion intelligent agent system and a learning method thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent systems play an increasingly important role in handling increasingly complex data analysis tasks and decision-making. These systems need to be highly flexible and adaptable to cope with ever-changing environments and task requirements.

[0003] However, due to its inherent rigid structure, traditional single-frame intelligent systems are difficult to quickly adapt to new algorithms and technologies, which limits their performance in dynamic environments. In addition, with the explosive growth of data and the increasing shortage of computing resources, how to efficiently manage and schedule resources, and how to quickly and accurately process and analyze data have become two major challenges in the design of intelligent systems.

[0004] Based on the above situation, the present invention proposes a multimodal learning fusion intelligent agent system and a learning method thereof. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a simple and efficient multi-modal learning fusion intelligent agent system and a learning method thereof.

[0006] The present invention is achieved through the following technical solutions:

[0007] A multimodal learning fusion agent system, characterized by comprising a core model module, a data preprocessing module, a task scheduling module, a resource management module, an interface module and a configuration and maintenance module;

[0008] The core model module is the technical core of the intelligent agent, integrating advanced machine learning and deep learning algorithms to handle complex data analysis tasks, and supports replacing or upgrading models through a plug-in architecture to adapt to new algorithms and business needs;

[0009] The data preprocessing module, the data conversion center of the agent, uses a series of data processing techniques to prepare data for use by the core model, uses regular expressions and natural language processing tools for data cleaning, uses the Pandas library and NumPy library for data conversion, and uses the Scikit-learn library for feature extraction;

[0010] The data preprocessing module can also dynamically load custom processing sub-modules such as Pillow or OpenCV according to the type and source of the data.

[0011] The task scheduling module is a workflow manager for intelligent agents, which uses scheduling algorithms (such as shortest job first or round-robin scheduling) to optimize the order of task execution, integrates task queues to manage task requests, and uses workflow definition languages ​​to define and execute complex workflows to ensure reasonable allocation and use of resources;

[0012] The resource management module is a performance optimizer for the intelligent agent. It uses the containerization technology Docker and the cluster management tool Kubernetes to dynamically allocate and manage hardware resources, monitor the usage of CPU, GPU and memory, and use resource quotas and limits to optimize resource usage.

[0013] The interface module is the communication hub of the intelligent body, responsible for providing multiple communication protocols to interact with external systems, supporting multiple data formats, integrating API gateways to manage API requests, and using JWT technology to ensure security;

[0014] When modules interact with each other, independent development and testing between modules are achieved through clearly defined API interfaces and messaging protocols; each module can be independently developed and tested, and then easily integrated into the system through the microservice architecture;

[0015] The configuration and maintenance module uses a centralized configuration management system (such as Spring Cloud Config) to uniformly manage the configuration parameters of each module, which not only reduces the possibility of configuration errors, but also makes system upgrades and maintenance easier; and implements a unified logging and monitoring system (ELK Stack and Prometheus+Grafana) to facilitate problem tracking and performance analysis.

[0016] The core model modules include convolutional neural network CNN, recurrent neural network RNN, long short-term memory network LSTM and Transformer model.

[0017] The task scheduling module uses RabbitMQ or Redis to manage task requests and uses business process modeling and annotation BPMN (Business Process Model and Notation) to define complex workflows; at the same time, it monitors task execution to ensure reasonable allocation and use of resources.

[0018] The resource management module is also integrated with a load balancer for distributing network traffic to ensure high availability and scalability of the system.

[0019] The interface module supports RESTful API, WebSocket communication protocol and gRPC communication protocol, supports JSON, XML and Protocol Buffers data formats, and can be adaptively adjusted as needed.

[0020] A learning method for a multimodal learning fusion agent system comprises the following steps:

[0021] Step S1: Real-time monitoring and analysis

[0022] Use system monitoring tools (such as Prometheus and Grafana) to collect CPU usage, memory usage, network latency, and disk I / O data in real time, and use real-time big data processing frameworks (such as Apache Kafka and Apache Spark Streaming) to clean, transform, and aggregate data.

[0023] Use ensemble machine learning algorithms (such as Isolation Forest, Autoencoders) as anomaly detection algorithms to identify abnormal patterns in the data;

[0024] Step S2: Automatically start the learning process

[0025] Use gradient descent algorithms and optimizers (Adam, RMSprop) to adjust the weights and parameters of deep learning models, and apply transfer learning to apply pre-trained models on new data distributions, and use fine-tuning strategies to quickly adapt to new tasks to reduce learning time and resource consumption;

[0026] Based on the task requirements and data characteristics, the agent system dynamically selects the most appropriate learning algorithm from the algorithm library, such as decision tree, support vector machine or neural network;

[0027] Step S3: Combining reinforcement learning with supervised learning and updating the supervised learning model

[0028] Use convolutional neural networks (CNN) and recurrent neural networks (RNN) for model updates, and apply Q-learning and Deep Q-Networks (DQN) for policy optimization;

[0029] In the presence of labeled data, fast model updates using convolutional neural networks (CNNs) for image recognition or recurrent neural networks (RNNs) for natural language processing;

[0030] In unlabeled data or exploratory tasks, algorithms such as Q-learning and Deep Q-Networks (DQN) are applied to optimize policies and gradually improve decision-making policies through interaction with the environment.

[0031] In multimodal learning fusion, the TensorFlow Agents (TF-Agents) framework is used to enable supervised learning and reinforcement learning to work together in the same framework and share knowledge to improve learning efficiency.

[0032] Step S4: Model evaluation

[0033] Use accuracy, recall and F1 score as column quantitative indicators, and use the LMMs-Eval automated testing framework to evaluate the model regularly;

[0034] Use the real-time monitoring system to track the performance of the monitoring model in real time, so as to promptly detect performance degradation and trigger the adjustment mechanism; Step S5, the learning and adjustment mechanism is triggered

[0035] When model evaluation or real-time monitoring finds that the performance degradation exceeds a custom threshold, the intelligent system automatically retrains the model or introduces a new model architecture to adapt to new data and task requirements.

[0036] A multimodal learning fusion intelligent agent device, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0037] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0038] The beneficial effects of the present invention are: the multimodal learning fusion intelligent agent system and its learning method not only enable the system to monitor and analyze data and task feedback in real time during operation, and automatically adjust the weights and parameters of the model to maintain optimal performance and adaptability, but also emphasize the importance of resource management, so that the intelligent agent system can maintain high performance while also having the ability to respond quickly to new situations, and can meet the needs of modern intelligent agent systems operating in complex and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Attached Figure 1 Schematic diagram of the multimodal learning fusion agent system of the present invention.

[0041] Attached Figure 2 Schematic diagram of the learning method of the multimodal learning fusion agent system of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] The multimodal learning fusion intelligent agent system includes a core model module, a data preprocessing module, a task scheduling module, a resource management module, an interface module and a configuration and maintenance module;

[0044] The core model module is the technical core of the intelligent agent, integrating advanced machine learning and deep learning algorithms to handle complex data analysis tasks, and supports replacing or upgrading models through a plug-in architecture to adapt to new algorithms and business needs;

[0045] The data preprocessing module, the data conversion center of the agent, uses a series of data processing techniques to prepare data for use by the core model, uses regular expressions and natural language processing tools for data cleaning, uses the Pandas library and NumPy library for data conversion, and uses the Scikit-learn library for feature extraction;

[0046] The data preprocessing module can also dynamically load custom processing sub-modules such as Pillow or OpenCV according to the type and source of the data.

[0047] The task scheduling module is a workflow manager for intelligent agents, which uses scheduling algorithms (such as shortest job first or round-robin scheduling) to optimize the order of task execution, integrates task queues to manage task requests, and uses workflow definition languages ​​to define and execute complex workflows to ensure reasonable allocation and use of resources;

[0048] The resource management module is a performance optimizer for the intelligent agent. It uses the containerization technology Docker and the cluster management tool Kubernetes to dynamically allocate and manage hardware resources, monitor the usage of CPU, GPU and memory, and use resource quotas and limits to optimize resource usage.

[0049] The interface module is the communication hub of the intelligent body, responsible for providing multiple communication protocols to interact with external systems, supporting multiple data formats, integrating API gateways to manage API requests, and using JWT technology to ensure security;

[0050] When modules interact with each other, independent development and testing between modules are achieved through clearly defined API interfaces and messaging protocols; each module can be independently developed and tested, and then easily integrated into the system through the microservice architecture;

[0051] The configuration and maintenance module uses a centralized configuration management system (such as Spring Cloud Config) to uniformly manage the configuration parameters of each module, which not only reduces the possibility of configuration errors, but also makes system upgrades and maintenance easier; and implements a unified logging and monitoring system (ELK Stack and Prometheus+Grafana) to facilitate problem tracking and performance analysis.

[0052] The core model modules include convolutional neural network CNN, recurrent neural network RNN, long short-term memory network LSTM and Transformer model.

[0053] The task scheduling module uses RabbitMQ or Redis to manage task requests and uses business process modeling and annotation BPMN (Business Process Model and Notation) to define complex workflows; at the same time, it monitors task execution to ensure reasonable allocation and use of resources.

[0054] The resource management module is also integrated with a load balancer for distributing network traffic to ensure high availability and scalability of the system.

[0055] The interface module supports RESTful API, WebSocket communication protocol and gRPC communication protocol, supports JSON, XML and Protocol Buffers data formats, and can be adaptively adjusted as needed.

[0056] The learning method of the multimodal learning fusion agent system includes the following steps:

[0057] Step S1: Real-time monitoring and analysis

[0058] Use system monitoring tools (such as Prometheus and Grafana) to collect CPU usage, memory usage, network latency, and disk I / O data in real time, and use real-time big data processing frameworks (such as Apache Kafka and Apache Spark Streaming) to clean, transform, and aggregate data.

[0059] Use ensemble machine learning algorithms (such as Isolation Forest, Autoencoders) as anomaly detection algorithms to identify abnormal patterns in the data;

[0060] Step S2: Automatically start the learning process

[0061] Use gradient descent algorithms and optimizers (Adam, RMSprop) to adjust the weights and parameters of deep learning models, and apply transfer learning to apply pre-trained models on new data distributions, and use fine-tuning strategies to quickly adapt to new tasks to reduce learning time and resource consumption;

[0062] Based on the task requirements and data characteristics, the agent system dynamically selects the most appropriate learning algorithm from the algorithm library, such as decision tree, support vector machine or neural network;

[0063] Step S3: Combining reinforcement learning with supervised learning and updating the supervised learning model

[0064] Use convolutional neural networks (CNN) and recurrent neural networks (RNN) for model updates, and apply Q-learning and Deep Q-Networks (DQN) for policy optimization;

[0065] In the presence of labeled data, fast model updates using convolutional neural networks (CNNs) for image recognition or recurrent neural networks (RNNs) for natural language processing;

[0066] In unlabeled data or exploratory tasks, algorithms such as Q-learning and Deep Q-Networks (DQN) are applied to optimize policies and gradually improve decision-making policies through interaction with the environment.

[0067] In multimodal learning fusion, the TensorFlow Agents (TF-Agents) framework is used to enable supervised learning and reinforcement learning to work together in the same framework and share knowledge to improve learning efficiency.

[0068] Step S4: Model evaluation

[0069] Use accuracy, recall and F1 score as column quantitative indicators, and use the LMMs-Eval automated testing framework to evaluate the model regularly;

[0070] Use the real-time monitoring system to track the performance of the monitoring model in real time, so as to promptly detect performance degradation and trigger the adjustment mechanism; Step S5, the learning and adjustment mechanism is triggered

[0071] When model evaluation or real-time monitoring finds that the performance degradation exceeds a custom threshold, the intelligent system automatically retrains the model or introduces a new model architecture to adapt to new data and task requirements.

[0072] The multimodal learning fusion intelligent agent device includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0073] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0074] The multimodal learning fusion agent system and its learning method improve the manageability, scalability and flexibility of the system by decomposing the agent system into multiple functional modules. This design not only allows the agent system to easily replace or upgrade the model to adapt to new algorithms and business needs, but also enables the system to monitor and analyze data and task feedback in real time during operation through an adaptive learning mechanism, and automatically adjust the weights and parameters of the model to maintain optimal performance and adaptability.

[0075] In addition, the multimodal learning fusion agent system and its learning method also emphasize the importance of resource management, dynamically allocate and manage hardware resources through containerization technology and cluster management tools, and optimize data flow and improve system response speed through real-time big data processing framework and anomaly detection algorithm. The combination of these technologies enables the agent system to maintain high performance while also having the ability to respond quickly to new situations, meeting the needs of modern agent systems operating in complex and changing environments.

[0076] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal learning fusion agent system, characterized by: It includes core model module, data preprocessing module, task scheduling module, resource management module, interface module and configuration and maintenance module; The core model module integrates machine learning and deep learning algorithms to process data analysis tasks, and supports replacing or upgrading models through a plug-in architecture to adapt to new algorithms and business needs; The data preprocessing module uses regular expressions and natural language processing tools to clean data, uses Pandas and NumPy libraries to convert data, and uses Scikit-learn library to extract features; The data preprocessing module can also dynamically load custom processing submodules based on the type and source of the data; The task scheduling module uses a scheduling algorithm to optimize the order of task execution, integrates a task queue to manage task requests, and uses a workflow definition language to define and execute complex workflows to ensure reasonable allocation and use of resources; The resource management module uses the containerization technology Docker and the cluster management tool Kubernetes to dynamically allocate and manage hardware resources, monitor the usage of CPU, GPU and memory, and use resource quotas and limits to optimize resource usage; The interface module is responsible for interacting with external systems through communication protocols, integrating an API gateway to manage API requests, and using JWT technology to ensure security; When modules interact with each other, independent development and testing between modules are achieved through API interfaces and messaging protocols; each module can be independently developed and tested, and then easily integrated into the system through the microservice architecture; The configuration and maintenance module adopts a centralized configuration management system to uniformly manage the configuration parameters of each module, and implements a unified logging and monitoring system to facilitate problem tracking and performance analysis.

2. The multimodal learning fusion agent system according to claim 1, characterized in that: The core model modules include convolutional neural network CNN, recurrent neural network RNN, long short-term memory network LSTM and Transformer model.

3. The multimodal learning fusion agent system according to claim 1, characterized in that: The task scheduling module uses RabbitMQ or Redis to manage task requests and uses business process modeling and annotation BPMN to define workflows; at the same time, it monitors task execution to ensure reasonable allocation and use of resources.

4. The multimodal learning fusion agent system according to claim 1, characterized in that: The resource management module is also integrated with a load balancer for distributing network traffic to ensure high availability and scalability of the system.

5. The multimodal learning fusion agent system according to claim 1, characterized in that: The interface module supports RESTful API, WebSocket communication protocol and gRPC communication protocol, supports JSON, XML and Protocol Buffers data formats, and can be adaptively adjusted as needed.

6. A learning method for a multimodal learning fusion agent system, characterized by: The following steps are involved: Step S1: Real-time monitoring and analysis Use system monitoring tools to collect CPU usage, memory usage, network latency, and disk I / O data in real time, and use real-time big data processing frameworks to clean, transform, and aggregate data; Step S2: Automatically start the learning process Use gradient descent algorithms and optimizers to adjust the weights and parameters of deep learning models, and apply transfer learning to apply pre-trained models on new data distributions, and fine-tune strategies to quickly adapt to new tasks to reduce learning time and resource consumption; Based on task requirements and data characteristics, the agent system dynamically selects learning algorithms from the algorithm library; Step S3: Combining reinforcement learning with supervised learning and updating the supervised learning model Use convolutional neural network (CNN) and recurrent neural network (RNN) to update the model, and apply Q-learning algorithm and Deep Q-Networks algorithm for strategy optimization; In multimodal learning fusion, the TensorFlow Agents framework is used to enable supervised learning and reinforcement learning to work together in the same framework and share knowledge to improve learning efficiency. Step S4: Model evaluation Use accuracy, recall and F1 score as column quantitative indicators, and use the LMMs-Eval automated testing framework to evaluate the model regularly; Use real-time monitoring systems to track and monitor model performance in real time, so as to promptly detect performance degradation and trigger adjustment mechanisms; Step S5: Learning and adjustment mechanism triggering When model evaluation or real-time monitoring finds that the performance degradation exceeds a custom threshold, the intelligent system automatically retrains the model or introduces a new model architecture to adapt to new data and task requirements.

7. The learning method of the multimodal learning fusion agent system according to claim 6, characterized in that: In step S1, the CPU usage, memory usage, network latency and disk I / O data are collected in real time using the Prometheus tool or the Grafana tool, and the Isolation Forest algorithm or the Autoencoders algorithm is used as an anomaly detection algorithm to identify abnormal patterns in the data.

8. The learning method of the multimodal learning fusion agent system according to claim 6, characterized in that: In step S3, when there is labeled data, a convolutional neural network (CNN) is used to update the model of image recognition, or a recurrent neural network (RNN) is used to update the model of natural language processing; In unlabeled data or exploratory tasks, Q-learning algorithm or Deep Q-Networks algorithm is applied for policy optimization to gradually improve the decision-making strategy through interaction with the environment.

9. A multimodal learning fusion agent device, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 5 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images