Multi-modal optimization-based carrier-class distributed database implementation method

Abstract

The invention discloses a carrier-class distributed database implementation method based on multi-modal optimization, and belongs to the technical field of databases, a transaction processing module, a query execution module and a streaming processing module are configured in a calculation layer to realize multi-modal calculation capability, and a multi-modal optimization mechanism is introduced in an optimization layer to realize multi-modal calculation capability. According to the method, differential optimization strategies are executed for transaction type, analysis type and streaming data processing scenes respectively, intelligent adjustment and optimization are achieved by combining machine learning and reinforcement learning, the system constructs a unified data platform, fusion processing of transaction, analysis and streaming data is supported originally so as to cope with a multi-modal business scene, and the system has the advantages of being high in practicability and easy to popularize. The kernel can adapt to service load change through an intelligent adjusting and optimizing mechanism, double improvement of performance and autonomous ability is achieved, the bottom layer is based on distributed storage and a high-consistency protocol, the carrier-level high-reliability requirement is met, and the top layer remarkably reduces operation and maintenance complexity and labor cost through a highly-visual and automatic operation and maintenance system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of databases, and particularly relates to a telecommunication-level distributed database implementation method based on multi-modal optimization. BACKGROUND

[0002] The telecommunication-level distributed database is a high-performance, high-availability and highly scalable database system specially designed for the telecommunication industry. Through data sharding, replication and distributed transaction processing technology, it meets the needs of massive data storage, high-concurrency access and real-time response in the telecommunication industry, while ensuring data consistency and system fault tolerance.

[0003] The rapid development of 5G, Internet of Things and cloud computing is driving the telecommunication business to a new stage, which requires the underlying database system to support massive transaction processing, complex analysis query and real-time stream computing at the same time. However, traditional databases are usually designed for single load and are difficult to handle such multi-modal mixed scenarios, resulting in serious challenges such as throughput bottleneck, high response delay and low resource utilization in typical applications such as high-concurrency billing and real-time log analysis.

[0004] Based on this, the application designs a telecommunication-level distributed database implementation method based on multi-modal optimization to solve the above problems. SUMMARY

[0005] The purpose of the application is to solve the problem that traditional databases are usually designed for single load and are difficult to handle such multi-modal mixed scenarios, resulting in serious challenges such as throughput bottleneck, high response delay and low resource utilization in typical applications such as high-concurrency billing and real-time log analysis, and to propose a telecommunication-level distributed database implementation method based on multi-modal optimization.

[0006] In order to achieve the above purpose, the application adopts the following technical solutions: The telecommunication-level distributed database implementation method based on multi-modal optimization comprises the following steps: In the computing layer, configure transaction processing modules, query execution modules and stream processing modules to realize multi-modal computing capabilities; In the optimization layer, introduce a multi-modal optimization mechanism, respectively execute differentiated optimization strategies for transaction-type, analysis-type and stream data processing scenarios, and realize intelligent tuning combined with machine learning and reinforcement learning; In the storage layer, use distributed storage pools, multi-copy consistency protocols and encryption algorithms to ensure the reliability and security of data; In the operation and maintenance layer, realize intelligent database management through monitoring and alarming, abnormality detection and automatic expansion, and visual operation and maintenance platform.

[0007] As a further description of the above technical solutions: The core mechanism of the transaction optimization is to use the log buffer pool and parallel commit together to ensure the throughput of transaction processing in high concurrency scenarios, where the log buffer pool is used to reduce I / O delay.

[0008] As a further description of the above technical solution: The optimization of the analytical load uses columnar storage to optimize data layout, introduces a vectorized execution engine to improve CPU processing efficiency, and uses dynamic indexing to achieve on-demand acceleration during query.

[0009] As a further description of the above technical solution: The streaming data processing uses memory queues to achieve high throughput, uses ring buffers to ensure stable reuse of memory, and relies on window computing mechanism to complete real-time aggregation of unbounded data.

[0010] As a further description of the above technical solution: The intelligent tuning engine integrates three core AI capabilities, namely machine learning, reinforcement learning, and time series prediction.

[0011] As a further description of the above technical solution: The machine learning is used to accurately predict SQL execution plans to improve single query efficiency, the reinforcement learning is used to adaptively adjust system parameters online to optimize overall workload performance, and the time series prediction is used to accurately estimate resource requirements to achieve optimal balance between cost and performance.

[0012] As a further description of the above technical solution: The machine learning, reinforcement learning, and time series prediction together build an end-to-end autonomous optimization system, which drives the database system to evolve towards autonomy and intelligence.

[0013] As a further description of the above technical solution: The core mechanism of the storage layer uses Raft or Paxos protocol to ensure strong consistency and high availability of data, providing strong consistency guarantee for upper-layer services.

[0014] As a further description of the above technical solution: The storage layer uses national cryptographic algorithm as the core encryption technology to ensure the confidentiality and compliance of static data storage.

[0015] As a further description of the above technical solution: The operation and maintenance layer builds an automatic closed loop from monitoring, decision-making to execution, which detects anomalies in real time, automatically triggers capacity expansion decisions, and at the same time, all states are monitored and manually intervened through a visual operation and maintenance platform.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. In this invention, the system constructs a unified data platform that natively supports the integrated processing of transactional, analytical, and streaming data to cope with multimodal business scenarios. Its kernel can adapt to changes in business load through an intelligent tuning mechanism, achieving a dual improvement in performance and autonomy. The underlying layer is based on distributed storage and a strong consistency protocol to meet the high reliability requirements of telecom-grade systems. The top layer significantly reduces the complexity of operation and maintenance and labor costs through a highly visualized and automated operation and maintenance system.

[0017] 2. In this invention, by working together with the transaction, analysis and streaming modules of the computing layer, and combined with the differentiated strategies of the optimization layer, the system achieves unified management and efficient processing of mixed loads, eliminating the complexity and latency caused by data migration between different systems.

[0018] 3. In this invention, an AI engine integrating machine learning, reinforcement learning, and time series prediction is constructed to build an end-to-end autonomous system from SQL execution and parameter tuning to resource planning, enabling the database to adapt to changes in business load, significantly improving performance and reducing manual management costs.

[0019] 4. In this invention, the storage layer is based on the Raft / Paxos strong consistency protocol and distributed architecture, which ensures the consistency of data and the high availability of services in a multi-replica environment, meeting the extreme requirements of data reliability in multiple key business scenarios in finance and telecommunications.

[0020] 5. In this invention, static data is encrypted using a built-in national cryptographic algorithm, which not only strengthens the data security defense at the technical level, but also meets the increasingly stringent requirements for localization and industry compliance at the regulatory level. Attached Figure Description

[0021] Figure 1 This is an overall architecture diagram of the telecom-grade distributed database implementation method based on multimodal optimization proposed in this invention; Figure 2 This is a diagram showing the internal structure of the optimization layer in the telecom-grade distributed database implementation method based on multimodal optimization proposed in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see the appendix Figure 1 - Appendix Figure 2 This invention provides a technical solution: a carrier-grade distributed database implementation method based on multimodal optimization, comprising the following steps: Configure transaction processing, query execution, and stream processing modules in the computing layer to achieve multimodal computing capabilities; A multimodal optimization mechanism is introduced into the optimization layer to execute differentiated optimization strategies for transactional, analytical, and streaming data processing scenarios, and intelligent tuning is achieved by combining machine learning and reinforcement learning. The storage layer employs distributed storage pools, multi-replica consistency protocols, and encryption algorithms to ensure data reliability and security. At the operations and maintenance layer, intelligent database management is achieved through monitoring and alarms, anomaly detection and automatic expansion, and a visual operations and maintenance platform.

[0024] Specifically, the core mechanism of the transactional optimization lies in the coordinated use of log buffer pool and parallel commit to ensure the throughput of transaction processing in high-concurrency scenarios, where log buffer pool is used to reduce I / O latency.

[0025] Specifically, the optimization of the analytical workload involves using columnar storage to optimize data layout, introducing a vectorized execution engine to improve CPU processing efficiency, and leveraging dynamic indexes to achieve on-demand acceleration during queries.

[0026] Specifically, in the streaming data processing, the system uses a memory queue to achieve high throughput, utilizes a circular buffer to ensure stable memory reuse, and relies on a window calculation mechanism to complete the real-time aggregation of unbounded data.

[0027] Specifically, the intelligent tuning engine integrates three core AI capabilities: machine learning, reinforcement learning, and time series prediction.

[0028] Specifically, the machine learning is used to accurately predict SQL execution plans to improve the efficiency of a single query; the reinforcement learning is used to adaptively adjust system parameters online to optimize overall workload performance; and the time series prediction is used to accurately estimate resource requirements to achieve the optimal balance between cost and performance.

[0029] Specifically, machine learning, reinforcement learning, and time series prediction together construct an end-to-end autonomous optimization system, driving the database system towards autonomy and intelligence.

[0030] Specifically, the core mechanism of the storage layer adopts the Raft or Paxos protocol to ensure strong data consistency and high availability, providing strong consistency guarantees for upper-layer services.

[0031] Specifically, the storage layer uses national cryptographic algorithms as the core encryption technology to ensure the confidentiality and compliance of static data storage.

[0032] Specifically, the operation and maintenance layer constructs an automated closed loop from monitoring and decision-making to execution. Anomaly detection provides real-time load awareness and automatically triggers capacity expansion decisions. Simultaneously, all states are centrally monitored and manually intervened through a visualized operation and maintenance platform. The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the technical scope disclosed in this invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of this invention.

Claims

1. A method for implementing a carrier-grade distributed database based on multimodal optimization, characterized in that: Includes the following steps: Configure transaction processing, query execution, and stream processing modules in the computing layer to achieve multimodal computing capabilities; A multimodal optimization mechanism is introduced into the optimization layer to execute differentiated optimization strategies for transactional, analytical, and streaming data processing scenarios, and intelligent tuning is achieved by combining machine learning and reinforcement learning. The storage layer employs distributed storage pools, multi-replica consistency protocols, and encryption algorithms to ensure data reliability and security. At the operations and maintenance layer, intelligent database management is achieved through monitoring and alarms, anomaly detection and automatic expansion, and a visual operations and maintenance platform.

2. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 1, characterized in that, The core mechanism of the transactional optimization lies in the coordinated use of log buffer pool and parallel commit to ensure the throughput of transaction processing in high-concurrency scenarios, where log buffer pool is used to reduce I / O latency.

3. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 1, characterized in that, The optimization of the analytical workload involves using columnar storage to optimize data layout, introducing a vectorized execution engine to improve CPU processing efficiency, and leveraging dynamic indexes to achieve on-demand acceleration during queries.

4. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 1, characterized in that, The streaming data processing system employs a memory queue to achieve high throughput, utilizes a circular buffer to ensure stable memory reuse, and relies on a window calculation mechanism to complete real-time aggregation of unbounded data.

5. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 1, characterized in that, The intelligent tuning engine integrates three core AI capabilities: machine learning, reinforcement learning, and time series prediction.

6. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 5, characterized in that, The machine learning is used to accurately predict SQL execution plans to improve the efficiency of a single query; the reinforcement learning is used to adaptively adjust system parameters online to optimize overall workload performance; and the time series prediction is used to accurately estimate resource requirements to achieve the optimal balance between cost and performance.

7. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 6, characterized in that, The machine learning, reinforcement learning, and time series prediction technologies together construct an end-to-end autonomous optimization system, driving the database system towards autonomy and intelligence.

8. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 1, characterized in that, The core mechanism of the storage layer adopts the Raft or Paxos protocol to ensure strong data consistency and high availability, providing strong consistency guarantees for upper-layer services.

9. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 8, characterized in that, The storage layer uses national cryptographic algorithms as the core encryption technology to ensure the confidentiality and compliance of static data storage.

10. The method for implementing a carrier-grade distributed database based on multimodal optimization according to claim 1, characterized in that, The operation and maintenance layer constructs an automated closed loop from monitoring and decision-making to execution. Anomaly detection can perceive the load in real time and automatically trigger expansion decisions. At the same time, all states are centrally monitored and manually intervened through a visual operation and maintenance platform.

Images