Remote application automation system based on RPC

Abstract

The invention relates to the technical field of process automation, and discloses a remote application automation system based on RPC, which comprises a system and communication basic module used for providing basic support for RPC communication and micro-service architecture, and a transaction management module used for data consistency in a distributed system. The fault-tolerant and redundancy mechanism module is used for improving the fault-tolerant capability of the system, the cross-language and protocol compatible module is used for supporting RPC interaction of a multi-language platform, and the security and authentication management module is used for security and access control of data and services. And the data persistence and distributed storage module is used for storage and quick access of read data. The cross-language support is realized through the gRPC and the Protobuf, so that the system can seamlessly carry out efficient RPC communication among different programming languages, the communication efficiency is higher, and the effects of transmission and data exchange can still be stably carried out in a micro-service environment needing high-frequency data exchange.

Description

Technical Field

[0001] The present invention relates to the technical field of process automation, and in particular to a remote application automation system based on RPC. Background Art

[0002] Remote application automation systems based on RPC remote procedure calls use RPC technology to remotely control or automate the operation and management of applications. RPC allows different computer systems to communicate over the network, so that programs executed on the local computer can call functions on the remote server, just like on the local computer. This provides great convenience for the implementation of distributed systems, cross-platform applications and automated tasks.

[0003] With the widespread application of microservice architecture, cross-language and cross-platform system integration has become a common requirement. In traditional distributed systems, communication between services usually relies on HTTP / RESTful API or SOAP protocol. Although these technologies are simple and easy to use, they have certain limitations in performance and scalability. RESTful API is based on text-based communication (such as JSON), which is easy to read and debug. However, due to its large message overhead and parsing cost, it is easy to increase communication delays in high-frequency data exchange scenarios. Especially in microservice architecture, when multiple services interact frequently, performance bottlenecks are particularly prominent.

[0004] In the existing technology, although gRPC and Protobuf can provide more efficient communication in the microservice architecture, they still have some shortcomings in cross-language support and protocol scalability. Although the existing RPC framework can support multiple languages, during the implementation process, cross-language communication may lead to compatibility issues due to differences in the processing of data structures and types between different languages. With the continuous advancement of technology, the system often needs to support new services, protocols or frameworks, which puts higher requirements on the scalability of the traditional RPC architecture. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a remote application automation system based on RPC, which solves the problem that although gRPC supports multiple programming languages, different programming languages ​​have differences in the processing and serialization methods of data types, and with the continuous development of new technologies and the increase of services, the traditional RPC framework cannot flexibly support the integration of future technologies or protocols, and lacks the ability to seamlessly expand and adapt to new services and new protocols.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A remote application automation system based on RPC, comprising:

[0008] System and communication basic module, transaction management module, fault tolerance and redundancy mechanism module, monitoring and fault warning module, traffic management and load balancing module, cross-language and protocol compatibility module, asynchronous task and event-driven processing module, security and authentication management module and data persistence and distributed storage module;

[0009] The system and communication basic module is used to provide basic support for RPC communication and microservice architecture;

[0010] The transaction management module is used for data consistency in a distributed system;

[0011] The fault tolerance and redundancy mechanism module is used to improve the fault tolerance capability of the system;

[0012] The monitoring and fault warning module is used to monitor the system status and detect faults and issue alarms in a timely manner;

[0013] The traffic management and load balancing module is used to optimize the distribution of requests and load management;

[0014] The cross-language and protocol compatible module is used to support RPC interaction of multi-language platforms;

[0015] The asynchronous task and event-driven processing module is used to bear the concurrency and response speed of task processing;

[0016] The security and authentication management module is used for security and access control of data and services;

[0017] The data persistence and distributed storage module is used for storage and quick access of read data.

[0018] Preferably, the system and communication infrastructure module includes an RPC communication protocol unit, a microservice architecture unit and a service discovery and load balancing unit. The RPC communication protocol unit is responsible for defining the RPC communication protocol between services in the system. The microservice architecture unit is used to design and implement a microservice architecture. The service discovery and load balancing unit is used to register and discover services through Consul or etcd and to use Envoy or Nginx to achieve balanced distribution of traffic. The microservice architecture includes service splitting, containerized deployment, automated orchestration, and management of inter-service communication. The RPC communication protocol is used to ensure reliable communication between services. The RPC communication protocol uses gRPC and Protobuf as communication protocols and data formats.

[0019] Preferably, the transaction management module includes a distributed transaction management unit, a consistency protocol unit and an event-driven architecture unit. The distributed transaction management unit is used to adopt the TCC or Saga mode to process cross-service transactions, the consistency protocol unit is used to select a consistency protocol according to system requirements, and is used to dynamically switch the role of the consistency model. The event-driven architecture unit is used to adopt Event Sourcing and CQRS modes to perform an event-driven architecture, and the event-driven architecture is used for cross-service data synchronization and management.

[0020] Preferably, the fault-tolerant and redundancy mechanism module includes a redundant copy and failover unit, a circuit breaker and fallback unit, and an automatic fault-tolerant adjustment unit. The redundant copy and failover unit is responsible for the availability of the redundant copy mechanism of the microservice. The circuit breaker and fallback unit is used to prevent the spread of faults by using a circuit breaker mechanism by utilizing Hystrix or Resilience4j. The automatic fault-tolerant adjustment unit is used to intelligently adjust the service fault tolerance level and dynamically optimize the fault-tolerant strategy through historical data analysis. The redundant copy mechanism availability is used to automatically transfer traffic to a healthy instance when an instance fails. The circuit breaker mechanism is used to trigger a fallback mechanism when the service is unavailable. The fault-tolerant strategy includes automatically expanding the number of instances and adjusting service copies.

[0021] Preferably, the monitoring and fault warning module includes a performance monitoring and health check unit, a log collection and exception analysis unit, and a fault warning and automatic response unit. The performance monitoring and health check unit is used to monitor services through Prometheus and visualize them through Grafana. The log collection and exception analysis unit is used to adopt ELK Stack to centrally manage logs and perform anomaly detection, and is responsible for collecting and analyzing application logs. The fault warning and automatic response unit is used to monitor and log analysis data to trigger fault warnings, and combine machine learning algorithms to predict potential faults. The predicted potential faults are used to automatically adjust related service configurations. The Prometheus monitoring services include CPU, memory, response time, and error rate. The machine learning algorithms include time series prediction algorithms, anomaly detection algorithms, classification algorithms, clustering algorithms, and deep learning algorithms.

[0022] Preferably, the traffic management and load balancing module includes an intelligent load balancing unit, a traffic routing and traffic control unit, and a current limiting and fuse unit. The intelligent load balancing unit uses Envoy or Nginx to enable intelligent load balancing, which is used to dynamically monitor the health status of service instances and optimize the distribution of traffic. The traffic routing and traffic control unit uses Istio or Spring Cloud Gateway to enable intelligent routing of traffic, which is used for content-based routing policies and to control traffic distribution and priority. The current limiting and fuse unit is responsible for current limiting control of traffic, which will not cause service overload under high concurrency conditions. The fusing and degradation of the current limiting and fuse unit adopt Rate Limiting and Circuit Breaker.

[0023] Preferably, the cross-language and protocol compatibility module includes a cross-language RPC support unit, a multi-platform compatibility unit and a protocol extension and adaptation unit. The cross-language RPC support unit is used to provide cross-language support through gRPC and Protobuf, so that the system can seamlessly perform RPC communication between different languages. The multi-platform compatibility unit is used to support communication and integration between multiple platforms by designing a system API interface. The protocol extension and adaptation unit is used to provide an extended protocol that can be used for future new services or new technologies. The extended protocol is used for the long-term scalability of the system. The different languages ​​include Java, Python or Go, and the multiple platforms include Web, mobile terminals, and backend services.

[0024] Preferably, the asynchronous task and event-driven processing module includes an asynchronous task processing unit, an event-driven architecture unit and a long task scheduling and management unit. The asynchronous task processing unit is used to schedule and execute asynchronous tasks through the message queue of Kafka or RabbitMQ, so that tasks can be executed on demand to avoid blocking the main thread. The event-driven architecture unit is used to synchronize and manage data across services in an event-driven manner through Event Sourcing and CQRS modes. The long task scheduling and management unit is used to schedule and manage long-running tasks through Quartz Scheduler or Celery.

[0025] Preferably, the security and authentication management module includes an identity authentication and authorization unit, an encryption and data protection unit and an API security control unit. The identity authentication and authorization unit is used for identity authentication and authorization control. The identity authentication and authorization control is used to confirm that only legitimate users and services in the system can access the API. The encryption and data protection unit is used to protect the security of data during transmission through TLS / SSL encrypted communication. The encrypted communication is used to prevent data from being tampered with or leaked. The API security control unit is used to use API Gateway to perform access control, traffic monitoring and restriction on the API to prevent unauthorized access. The identity authentication and authorization control methods include OAuth2.0 or JWT.

[0026] Preferably, the data persistence and distributed storage module includes a distributed database management unit, a cache and data acceleration unit and a distributed file storage unit. The distributed database management unit is responsible for the deployment and management of the distributed database. The cache and data acceleration unit is used to accelerate the reading of frequently accessed data through cache technology. The cache technology is used to reduce the burden on the database. The distributed file storage unit is used for storing distributed files and for persistent storage and fast access to massive data. The distributed database includes Cassandra and CockroachDB. The cache technology includes at least any one of Redis or Memcached. The distributed file storage method includes HDFS and Ceph. The cache technology includes hardware cache, software cache and cache strategy.

[0027] The present invention provides a remote application automation system based on RPC, which has the following beneficial effects:

[0028] 1. The present invention realizes cross-language support through gRPC and Protobuf, so that the system can seamlessly perform efficient RPC communication between different programming languages, and adopts an extensible protocol architecture to support the integration of future technologies or services, with higher communication efficiency, lower latency and stronger system scalability, and can still stably perform transmission and data exchange effects in a microservice environment that requires high-frequency data exchange.

[0029] 2. The present invention introduces redundant copies and fault transfer mechanisms, circuit breakers and fallback mechanisms, and automatic fault-tolerant adjustment strategies. When a service instance fails or is overloaded, it can automatically transfer traffic to a healthy instance, and prevent the fault from spreading through a circuit breaker mechanism. In the face of high-concurrency requests and partial service failures, it can still ensure service continuity and high availability. It not only improves the fault tolerance and stability of the system, but also can respond to dynamically changing loads and fault scenarios faster and more intelligently than traditional single-point redundancy solutions.

[0030] 3. The distributed database, cache technology and distributed file storage of the present invention ensure efficient storage and fast access to massive data, achieve high availability and horizontal expansion of data through distributed databases such as Cassandra and CockroachDB, use cache technologies such as Redis or Memcached to accelerate the reading of frequently accessed data, and provide distributed file storage through HDFS or Ceph. When processing large amounts of data, it can significantly reduce the database burden, improve the speed of data access and the scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A system architecture diagram of a remote application automation system based on RPC according to the present invention;

[0032] Figure 2 This is a system and communication basic module architecture diagram of a remote application automation system based on RPC of the present invention;

[0033] Figure 3 This is a diagram of the architecture of a transaction management module of a remote application automation system based on RPC of the present invention;

[0034] Figure 4 This is a fault-tolerant and redundant mechanism module architecture diagram of a remote application automation system based on RPC of the present invention;

[0035] Figure 5 This is a monitoring and fault warning module architecture diagram of a remote application automation system based on RPC in the present invention;

[0036] Figure 6 This is a flow management and load balancing module architecture diagram of a remote application automation system based on RPC of the present invention;

[0037] Figure 7 A cross-language and protocol compatible module architecture diagram of a remote application automation system based on RPC of the present invention;

[0038] Figure 8 This is an architecture diagram of an asynchronous task and event-driven processing module of an RPC-based remote application automation system of the present invention;

[0039] Fig. 9 This is a security and authentication management module architecture diagram of a remote application automation system based on RPC of the present invention;

[0040] Fig.10 This is a data persistence and distributed storage module architecture diagram of a remote application automation system based on RPC in the present invention;

[0041] Fig.11This is a system and communication basic module relationship architecture diagram of a remote application automation system based on RPC of the present invention;

[0042] Fig.12 This is a relationship architecture diagram of a transaction management module of a remote application automation system based on RPC of the present invention;

[0043] Fig.13 This is a fault tolerance and redundancy mechanism module relationship architecture diagram of a remote application automation system based on RPC of the present invention;

[0044] Fig.14 This is a relationship architecture diagram of monitoring and fault warning modules of a remote application automation system based on RPC of the present invention;

[0045] Fig.15 This is a relationship architecture diagram of the traffic management and load balancing modules of a remote application automation system based on RPC of the present invention;

[0046] Fig.16 A cross-language and protocol compatible module relationship architecture diagram of a remote application automation system based on RPC of the present invention;

[0047] Fig.17 This is a relationship architecture diagram of an asynchronous task and event-driven processing module of a remote application automation system based on RPC of the present invention;

[0048] Fig.18 This is an architecture diagram of the relationship between the security and authentication management module and the data persistence and distributed storage module of an RPC-based remote application automation system of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Please refer to the attached Figure 1 -Attached Fig.10 The embodiment of the present invention provides a remote application automation system based on RPC, including:

[0051] System and communication basic module, transaction management module, fault tolerance and redundancy mechanism module, monitoring and fault warning module, traffic management and load balancing module, cross-language and protocol compatibility module, asynchronous task and event-driven processing module, security and authentication management module and data persistence and distributed storage module;

[0052] The system and communication basic module is used to provide basic support for RPC communication and microservice architecture;

[0053] The transaction management module is used for data consistency in distributed systems;

[0054] The fault tolerance and redundancy mechanism module is used to improve the fault tolerance of the system;

[0055] The monitoring and fault warning module is used to monitor the system status and detect faults and issue alarms in a timely manner;

[0056] Traffic management and load balancing modules are used to optimize request distribution and load management;

[0057] Cross-language and protocol compatible modules are used to support RPC interactions on multi-language platforms;

[0058] Asynchronous task and event-driven processing modules are used to handle the concurrency and response speed of task processing;

[0059] The security and authentication management module is used for security and access control of data and services;

[0060] The data persistence and distributed storage module is used for storage and fast access of read data.

[0061] The system and communication basic modules include the RPC communication protocol unit, the microservice architecture unit, and the service discovery and load balancing unit. The RPC communication protocol unit is responsible for defining the RPC communication protocol between services in the system. The microservice architecture unit is used to design and implement the microservice architecture. The service discovery and load balancing unit is used to register and discover services through Consul or etcd and use Envoy or Nginx to achieve balanced distribution of traffic. The microservice architecture includes service splitting, containerized deployment, automated orchestration, and management of inter-service communication. The RPC communication protocol is used to ensure reliable communication between services. The RPC communication protocol uses gRPC and Protobuf as the communication protocol and data format.

[0062] Specifically, the RPC communication protocol unit defines an efficient communication protocol between microservices in the system by adopting the gRPC protocol and Protobuf data format, achieving reliable communication between services, reducing bandwidth consumption and improving the security of data exchange;

[0063] The microservice architecture unit is responsible for designing the microservice architecture, improving the scalability and flexibility of the system through service splitting, containerized deployment (such as Docker), automated orchestration (such as Kubernetes), and inter-service communication management;

[0064] The service discovery and load balancing unit uses Consul or etcd to register and discover services, ensuring the dynamic availability of microservices, and uses Envoy or Nginx to achieve balanced distribution of traffic, ensuring high availability and load balancing of the system. It enables efficient and stable communication between services in the system and can dynamically adapt to load changes.

[0065] The transaction management module includes a distributed transaction management unit, a consistency protocol unit and an event-driven architecture unit. The distributed transaction management unit is used to process cross-service transactions using the TCC or Saga mode. The consistency protocol unit is used to select the consistency protocol according to system requirements and to dynamically switch the consistency model. The event-driven architecture unit is used to adopt the Event Sourcing and CQRS modes to perform event-driven architecture. The event-driven architecture is used for cross-service data synchronization and management.

[0066] Specifically, the distributed transaction management unit uses TCC (Try-Confirm-Cancel) or Saga mode to handle cross-service transactions, ensuring the consistency and reliability of transactions in a distributed environment. The TCC mode ensures the final consistency of transactions between services through three-phase operations: try, confirm, and cancel. The Saga mode ensures the high availability and fault tolerance of the system by breaking down long transactions into multiple local transactions and handling failed transactions through a compensation mechanism.

[0067] The consistency protocol unit selects the appropriate consistency protocol according to system requirements and supports dynamic switching of consistency models to flexibly balance the system's strong consistency and final consistency requirements. It ensures data consistency across services by selecting appropriate protocols such as two-phase commit, Paxos, or Raft, and adjusts consistency strategies according to specific scenarios;

[0068] The event-driven architecture unit drives the system's event management and data synchronization by adopting the Event Sourcing and CQRS modes. Event Sourcing makes each business event persistent and serves as the only source of system status, thereby ensuring the traceability and consistency of system data; CQRS separates commands from query operations, optimizes read and write performance, and ensures data synchronization and consistency management across services. Through flexible transaction management and event processing mechanisms, the transaction integrity and efficient data synchronization of distributed systems are guaranteed.

[0069] The fault tolerance and redundancy mechanism module includes redundant replicas and failover units, circuit breakers and fallback units, and automatic fault tolerance adjustment units. The redundant replicas and failover units are responsible for the availability of the redundant replica mechanisms of microservices. The circuit breakers and fallback units are used to prevent the spread of faults by using circuit breaker mechanisms through Hystrix or Resilience4j. The automatic fault tolerance adjustment unit is used to intelligently adjust the service fault tolerance level and dynamically optimize the fault tolerance strategy through historical data analysis. The availability of the redundant replica mechanism is used to automatically transfer traffic to healthy instances when an instance fails. The circuit breaker mechanism is used to trigger the fallback mechanism when the service is unavailable. The fault tolerance strategy includes automatically expanding the number of instances and adjusting service replicas.

[0070] Specifically, the redundant copy and failover unit implements the redundant copy mechanism of microservices to ensure that when a service instance fails, the traffic can be automatically transferred to the healthy copy to ensure the high availability of the system. The service status is monitored in real time through the load balancing and health check mechanism to prevent single point failures from affecting the operation of the overall system.

[0071] The circuit breaker and fallback unit uses circuit breaker frameworks such as Hystrix or Resilience4j to achieve fault isolation and prevent fault propagation. When a service fails continuously, the circuit breaker will cut off the call to the service to avoid further affecting system performance, and trigger the fallback mechanism such as returning to the default value or calling the backup service to ensure that the system can continue to provide services;

[0072] The automatic fault-tolerance adjustment unit analyzes historical data, intelligently adjusts the system's fault-tolerance level, and dynamically optimizes the fault-tolerance strategy to ensure that the system can flexibly respond to different load and fault conditions. Fault-tolerance strategies include automatically expanding the number of instances and adjusting the number of service replicas to cope with different traffic fluctuations and fault scenarios, thereby improving the robustness and adaptability of the system. This module ensures the continuous availability and stability of the system in the face of high concurrency, faults, or load fluctuations through redundancy mechanisms, circuit breaker protection, and automatic adjustment strategies.

[0073] The monitoring and fault warning module includes a performance monitoring and health check unit, a log collection and anomaly analysis unit, and a fault warning and automatic response unit. The performance monitoring and health check unit is used to monitor services through Prometheus and visualize them through Grafana. The log collection and anomaly analysis unit is used to use ELK Stack to centrally manage logs and perform anomaly detection. It is responsible for collecting and analyzing application logs. The fault warning and automatic response unit is used to monitor and log analysis data to trigger fault warnings, and combine machine learning algorithms to predict potential faults. The predicted potential faults are used to automatically adjust the configuration of related services. Prometheus monitoring services include CPU, memory, response time, and error rate. Machine learning algorithms include time series prediction algorithms, anomaly detection algorithms, classification algorithms, clustering algorithms, and deep learning algorithms.

[0074] Specifically, the performance monitoring and health check unit uses Prometheus to monitor the performance of microservices, collects key indicators such as CPU usage, memory usage, response time and error rate of services in real time, and visualizes them through Grafana to help operation and maintenance personnel understand the health status of the system in real time. This unit can detect system bottlenecks or anomalies in a timely manner to ensure the stable operation of services;

[0075] The log collection and exception analysis unit uses ELK Stack to implement centralized log management and exception analysis, and is responsible for collecting, storing, and analyzing log data generated by each microservice. By classifying and analyzing logs in real time, it can detect abnormal behaviors and potential problems in the system, and provide data support for fault diagnosis and performance optimization. The fault warning and automatic response unit automatically triggers fault warnings by combining performance monitoring data and log analysis results, and also combines machine learning algorithms such as time series prediction algorithms, anomaly detection algorithms, classification algorithms, clustering algorithms, and deep learning algorithms to predict potential faults. Through these analysis and response mechanisms, the system can make corresponding adjustments or warnings before a failure occurs, thereby improving the stability and reliability of the system.

[0076] The traffic management and load balancing module includes an intelligent load balancing unit, a traffic routing and traffic control unit, and a current limiting and fuse unit. The intelligent load balancing unit uses Envoy or Nginx to enable intelligent load balancing, which is used to dynamically monitor the health status of service instances and optimize traffic distribution. The traffic routing and traffic control unit uses Istio or SpringCloud Gateway to enable intelligent routing of traffic, which is used for content-based routing policies to control traffic distribution and priority. The current limiting and fuse unit is responsible for current limiting control of traffic, which will not cause service overload under high concurrency conditions. The fuse and degradation of the current limiting and fuse unit use Rate Limiting and Circuit Breaker.

[0077] Specifically, the intelligent load balancing unit uses Envoy or Nginx to implement intelligent load balancing, dynamically monitor the health status of service instances, and optimize traffic distribution. When a service instance is unavailable or overloaded, traffic is automatically forwarded to healthy service instances, thereby ensuring high availability and load balancing of the system. It can automatically adjust traffic distribution strategies under different network conditions and load environments to improve service response speed and fault tolerance.

[0078] Traffic routing and traffic control units implement intelligent traffic routing through Istio or Spring Cloud Gateway, and are used for content-based routing strategies, such as determining the traffic forwarding direction based on the request path, HTTP header, or request parameters. This enables the system to flexibly control traffic distribution and priority between different services, ensuring that different types of requests can access key services first, improving the overall efficiency and responsiveness of the system;

[0079] The current limiting and circuit breaker unit is used to control traffic in high concurrency situations to avoid overloading a single service instance. Rate Limiting controls the maximum number of requests per unit time to prevent service crashes caused by sudden traffic; Circuit Breaker mechanism quickly cuts off requests to the service when the service is unavailable to prevent a backlog of requests from causing system crashes, while triggering a fallback strategy to provide fault tolerance. This unit ensures that the system can still run stably under high load conditions, avoids risks such as service overload and downtime, and improves the stability and robustness of the system.

[0080] The cross-language and protocol compatibility module includes a cross-language RPC support unit, a multi-platform compatibility unit, and a protocol extension and adaptation unit. The cross-language RPC support unit is used to provide cross-language support through gRPC and Protobuf, so that the system can seamlessly perform RPC communication between different languages. The multi-platform compatibility unit is used to support communication and integration between multiple platforms by designing the system API interface. The protocol extension and adaptation unit is used to provide extended protocols that can be used for future new services or new technologies. The extended protocol is used for the long-term scalability of the system. Different languages ​​include Java, Python or Go, and multiple platforms include Web, mobile, and background services.

[0081] Specifically, the cross-language RPC support unit provides cross-language support by adopting gRPC and Protobuf, ensuring that services within the system can perform seamless RPC communication between different programming languages. As an efficient remote procedure call protocol, gRPC enables services to communicate across languages ​​by supporting multiple programming languages ​​(such as Java, Python, Go, etc.), while Protobuf provides a compact and efficient data serialization format to ensure consistency and efficiency of data exchange between different languages. It enables the system to flexibly respond to the needs of multiple development languages ​​and promotes the cross-language interoperability of the system;

[0082] The multi-platform compatibility unit enables the system to support communication and integration between different platforms by designing a unified system API interface. These platforms can include Web, mobile, and backend services. This unit ensures that different platforms can efficiently access system services and interact with data through standardized APIs and protocols. Whether it is a Web application on the PC, a client on a mobile device, or a backend microservice, they can all interact and integrate smoothly with the core system, improving the cross-platform compatibility of the system.

[0083] The protocol extension and adaptation unit provides the ability to extend the protocol, which can seamlessly integrate new services or new technologies in the future. By designing an extensible protocol architecture, the unit allows the system to quickly adapt to new services and protocols when the technology stack changes or new requirements are added, ensuring the long-term scalability and flexibility of the system. This expansion mechanism not only ensures that the system is compatible with existing technologies, but also enables it to flexibly respond to and quickly integrate new technologies and services when future technologies are updated.

[0084] The asynchronous task and event-driven processing module includes an asynchronous task processing unit, an event-driven architecture unit, and a long task scheduling and management unit. The asynchronous task processing unit is used to schedule and execute asynchronous tasks through the message queue of Kafka or RabbitMQ, so that tasks can be executed on demand to avoid blocking the main thread. The event-driven architecture unit is used to synchronize and manage data across services in an event-driven manner through EventSourcing and CQRS modes. The long task scheduling and management unit is used to schedule and manage long-running tasks through Quartz Scheduler or Celery.

[0085] Specifically, the asynchronous task processing unit uses message queue systems such as Kafka or RabbitMQ to schedule and execute asynchronous tasks. These message queues can decouple task producers and consumers, ensuring that tasks are executed on demand without blocking the main thread or other business processes. This design can improve the system's responsiveness and throughput, avoiding performance bottlenecks caused by synchronous tasks, which is particularly important in high-concurrency scenarios;

[0086] Event-driven architecture unit, synchronizes and manages data across services in an event-driven manner. EventSourcing ensures that all data changes are persisted through events, and each event records the changes in the system state, which not only improves the consistency and traceability of the data, but also enables the restoration of the system state through event playback. The CQRS pattern optimizes query performance by separating data write operations from read operations, and enables more efficient data synchronization between services. Through this event-driven architecture, the system can achieve loose coupling between services and provide higher scalability and flexibility;

[0087] The long task scheduling and management unit manages and schedules long-running tasks through scheduling frameworks such as Quartz Scheduler or Celery. Quartz Scheduler is a powerful task scheduling tool that supports complex scheduling scenarios such as scheduled execution of tasks, repeated execution of tasks, and task dependencies; Celery is a distributed task queue framework that can manage large-scale asynchronous tasks in a distributed environment and provide task execution status tracking and management. This unit can ensure that long-running tasks can be executed efficiently and stably, avoiding system resource waste or service blocking due to long tasks.

[0088] The security and authentication management module includes an authentication and authorization unit, an encryption and data protection unit and an API security control unit. The authentication and authorization unit is used for authentication and authorization control. The authentication and authorization control is used to confirm that only legitimate users and services in the system can access the API. The encryption and data protection unit is used to protect the security of data during transmission through TLS / SSL encrypted communication. The encrypted communication is used to prevent data from being tampered with or leaked. The API security control unit is used to use API Gateway to perform access control, traffic monitoring and restriction on the API to prevent unauthorized access. The authentication and authorization control methods include OAuth2.0 or JWT.

[0089] Specifically, the authentication and authorization unit is responsible for authentication and authorization control to ensure that only legitimate users and services can access the system API. By using standard authentication mechanisms such as OAuth2.0 or JWT, the system can verify the identity of the user or service and grant corresponding access rights. These authentication protocols provide a secure token management mechanism to ensure the legitimacy and validity of users or services when accessing protected resources;

[0090] The encryption and data protection unit encrypts communications through the TLS / SSL protocol to ensure that data will not be tampered with or leaked during transmission. The TLS / SSL encryption protocol can provide data encryption, authentication, and data integrity protection for communications between the client and the server, preventing third parties from conducting man-in-the-middle attacks or eavesdropping on sensitive information. The unit protects all sensitive data in network communications through encryption, ensuring the confidentiality and security of data during transmission;

[0091] The API security control unit uses API Gateway to perform access control, traffic monitoring and restriction of APIs to prevent unauthorized access. As a unified entrance, API Gateway can centrally manage access rights of all APIs, monitor traffic, and limit the frequency and number of API requests based on different access policies, thereby effectively preventing API abuse and DDoS attacks. In addition, API Gateway can also combine identity authentication and authorization mechanisms to ensure that only authenticated users or services can access specific APIs, further improving the security of the system.

[0092] The data persistence and distributed storage module includes a distributed database management unit, a cache and data acceleration unit and a distributed file storage unit. The distributed database management unit is responsible for the deployment and management of the distributed database. The cache and data acceleration unit is used to accelerate the reading of frequently accessed data through cache technology. The cache technology is used to reduce the burden on the database. The distributed file storage unit is used for storing distributed files and for persistent storage and fast access to massive data. The distributed databases include Cassandra and CockroachDB. The cache technology includes at least any one of Redis or Memcached. The distributed file storage methods include HDFS and Ceph. The cache technology includes hardware cache, software cache and cache strategy.

[0093] Specifically, the distributed database management unit is responsible for the deployment and management of distributed databases, supporting the high availability and horizontal expansion of system data. Commonly used distributed databases include Cassandra and CockroachDB. Cassandra has a decentralized design and high fault tolerance, and is suitable for scenarios that require high write performance and linear scalability; CockroachDB supports strong consistency and automatic data sharding, and is suitable for application scenarios with high requirements for transaction consistency. Through these databases, the system can distribute and store data on multiple nodes to ensure data reliability and fast access capabilities;

[0094] The cache and data acceleration unit uses cache technology to accelerate the reading of frequently accessed data, reduce the burden on the backend database, and improve system response speed. Caching technology includes software cache solutions such as Redis or Memcached. Redis provides rich data structure support and high-performance reading and writing, which is suitable for caching hot data and distributed transaction scenarios; Memcached has a lightweight architecture and is suitable for simple key-value caching needs. In addition, it also combines hardware cache (such as SSD, NVM storage) and cache strategies (such as LRU, LFU, TTL expiration strategies) to optimize cache management to ensure efficient data access and rational use of cache space;

[0095] Distributed file storage units are used to store distributed files. They are mainly responsible for the persistent storage and fast access of massive data, and are suitable for unstructured data such as log files and large object storage. Distributed file storage systems such as HDFS and Ceph are supported. HDFS is suitable for high-throughput and batch data storage scenarios, and improves fault tolerance and availability through data segmentation and replication mechanisms; Ceph provides unified object, block and file storage, which is suitable for cloud storage and high-performance storage requirements, and supports automatic fault recovery and high scalability.

[0096] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A remote application automation system based on RPC, characterized in that: include: System and communication basic module, transaction management module, fault tolerance and redundancy mechanism module, monitoring and fault warning module, traffic management and load balancing module, cross-language and protocol compatibility module, asynchronous task and event-driven processing module, security and authentication management module and data persistence and distributed storage module; The system and communication basic module is used to provide basic support for RPC communication and microservice architecture; The transaction management module is used for data consistency in a distributed system; The fault tolerance and redundancy mechanism module is used to improve the fault tolerance capability of the system; The monitoring and fault warning module is used to monitor the system status and detect faults and issue alarms in a timely manner; The traffic management and load balancing module is used to optimize the distribution of requests and load management; The cross-language and protocol compatible module is used to support RPC interaction of multi-language platforms; The asynchronous task and event-driven processing module is used to bear the concurrency and response speed of task processing; The security and authentication management module is used for security and access control of data and services; The data persistence and distributed storage module is used for storage and quick access of read data.

2. The RPC-based remote application automation system according to claim 1, characterized in that: The system and communication infrastructure module includes an RPC communication protocol unit, a microservice architecture unit, and a service discovery and load balancing unit. The RPC communication protocol unit is responsible for defining the RPC communication protocol between services in the system. The microservice architecture unit is used to design and implement a microservice architecture. The service discovery and load balancing unit is used to register and discover services through Consul or etcd and to use Envoy or Nginx to achieve balanced distribution of traffic. The microservice architecture includes service splitting, containerized deployment, automated orchestration, and management of inter-service communication. The RPC communication protocol is used to ensure reliable communication between services. The RPC communication protocol uses gRPC and Protobuf as communication protocols and data formats.

3. The RPC-based remote application automation system according to claim 1, characterized in that: The transaction management module includes a distributed transaction management unit, a consistency protocol unit and an event-driven architecture unit. The distributed transaction management unit is used to adopt the TCC or Saga mode to process cross-service transactions. The consistency protocol unit is used to select a consistency protocol according to system requirements and to dynamically switch the role of the consistency model. The event-driven architecture unit is used to adopt the Event Sourcing and CQRS modes to perform an event-driven architecture. The event-driven architecture is used for cross-service data synchronization and management.

4. The RPC-based remote application automation system according to claim 1, characterized in that: The fault-tolerant and redundant mechanism module includes a redundant copy and failover unit, a circuit breaker and fallback unit, and an automatic fault-tolerant adjustment unit. The redundant copy and failover unit is responsible for the availability of the redundant copy mechanism of the microservice. The circuit breaker and fallback unit is used to prevent the spread of faults by using a circuit breaker mechanism by using Hystrix or Resilience4j. The automatic fault-tolerant adjustment unit is used to intelligently adjust the service fault tolerance level and dynamically optimize the fault tolerance strategy through historical data analysis. The redundant copy mechanism availability is used to automatically transfer traffic to a healthy instance when an instance fails. The circuit breaker mechanism is used to trigger a fallback mechanism when a service is unavailable. The fault-tolerant strategy includes automatically expanding the number of instances and adjusting service copies.

5. The RPC-based remote application automation system according to claim 1, characterized in that: The monitoring and fault warning module includes a performance monitoring and health check unit, a log collection and anomaly analysis unit, and a fault warning and automatic response unit. The performance monitoring and health check unit is used to monitor services through Prometheus and visualize them through Grafana. The log collection and anomaly analysis unit is used to use ELK Stack to centrally manage logs and perform anomaly detection, and is responsible for collecting and analyzing application logs. The fault warning and automatic response unit is used to monitor and log analysis data to trigger fault warnings, and combine machine learning algorithms to predict potential faults. The predicted potential faults are used to automatically adjust related service configurations. The Prometheus monitoring services include CPU, memory, response time, and error rate. The machine learning algorithms include time series prediction algorithms, anomaly detection algorithms, classification algorithms, clustering algorithms, and deep learning algorithms.

6. The RPC-based remote application automation system according to claim 1, characterized in that: The traffic management and load balancing module includes an intelligent load balancing unit, a traffic routing and traffic control unit, and a current limiting and fuse unit. The intelligent load balancing unit uses Envoy or Nginx to enable intelligent load balancing, which is used to dynamically monitor the health status of service instances and optimize the distribution of traffic. The traffic routing and traffic control unit uses Istio or SpringCloud Gateway to enable intelligent routing of traffic, which is used for content-based routing policies to control traffic distribution and priority. The current limiting and fuse unit is responsible for current limiting control of traffic, which will not cause service overload under high concurrency conditions. The fusing and degradation of the current limiting and fuse unit adopt Rate Limiting and CircuitBreaker.

7. The RPC-based remote application automation system according to claim 1, characterized in that: The cross-language and protocol compatibility module includes a cross-language RPC support unit, a multi-platform compatibility unit and a protocol extension and adaptation unit. The cross-language RPC support unit is used to provide cross-language support through gRPC and Protobuf, so that the system can seamlessly perform RPC communication between different languages. The multi-platform compatibility unit is used to support communication and integration between multiple platforms by designing the system API interface. The protocol extension and adaptation unit is used to provide an extended protocol that can be used for future new services or new technologies. The extended protocol is used for the long-term scalability of the system. The different languages ​​include Java, Python or Go, and the multiple platforms include Web, mobile terminals, and background services.

8. The RPC-based remote application automation system according to claim 1, characterized in that: The asynchronous task and event-driven processing module includes an asynchronous task processing unit, an event-driven architecture unit and a long task scheduling and management unit. The asynchronous task processing unit is used to schedule and execute asynchronous tasks through the message queue of Kafka or RabbitMQ, so that tasks can be executed on demand to avoid blocking the main thread. The event-driven architecture unit is used to synchronize and manage data across services in an event-driven manner through EventSourcing and CQRS modes. The long task scheduling and management unit is used to schedule and manage long-running tasks through Quartz Scheduler or Celery.

9. The RPC-based remote application automation system according to claim 1, characterized in that: The security and authentication management module includes an identity authentication and authorization unit, an encryption and data protection unit and an API security control unit. The identity authentication and authorization unit is used for identity authentication and authorization control. The identity authentication and authorization control is used to confirm that only legitimate users and services in the system can access the API. The encryption and data protection unit is used to protect the security of data during transmission through TLS / SSL encrypted communication. The encrypted communication is used to prevent data from being tampered with or leaked. The API security control unit is used to use API Gateway to perform access control, traffic monitoring and restriction on the API to prevent unauthorized access. The identity authentication and authorization control methods include OAuth2.0 or JWT.

10. The RPC-based remote application automation system according to claim 1, characterized in that: The data persistence and distributed storage module includes a distributed database management unit, a cache and data acceleration unit and a distributed file storage unit. The distributed database management unit is responsible for the deployment and management of the distributed database. The cache and data acceleration unit is used to accelerate the reading of frequently accessed data through cache technology. The cache technology is used to reduce the burden on the database. The distributed file storage unit is used for storing distributed files and for persistent storage and fast access to massive data. The distributed database includes Cassandra and CockroachDB. The cache technology includes at least any one of Redis or Memcached. The distributed file storage method includes HDFS and Ceph. The cache technology includes hardware cache, software cache and cache strategy.

Images