Big data read-write adaptive method based on decision algorithm and big data read-write system

Abstract

The invention discloses a big data read-write adaptive method based on a decision algorithm and a big data read-write system, and belongs to the technical field of intelligent decision of a data center, high-speed read-write operation of data is processed in real time in a service system, and a data read-write strategy and resource allocation are dynamically adjusted by analyzing a data access mode and a system load condition; data are preloaded through an intelligent prefetching strategy, and the storage and access paths of hotspot data are optimized, so that the waiting time of a user and the consumption of system resources are reduced. According to the method, data management can be simplified, data access efficiency can be improved, data consistency and security can be enhanced, the problem of multi-database management and query complexity faced by enterprises can be effectively solved, and more efficient and more consistent data access and management experience can be provided for the enterprises.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent decision-making in data centers, and in particular to a big data reading and writing adaptive method and a big data reading and writing system based on a decision algorithm. Background Art

[0002] With the increasing demand for big data and real-time data processing, enterprises are faced with the complexity of storing data in multiple different database systems. Traditionally, enterprises usually need to rely on multiple database technologies, such as Redis, MySQL, HBase, and HDFS, when processing different types of data. Each technology has its own unique advantages and applicable scenarios. However, with the increase in data volume and data type, managing and querying these scattered data sources has become increasingly challenging. Challenges brought by database diversity: 1) Installation and configuration complexity: Each database system has its own specific installation and configuration requirements, which not only increases the workload of deployment and maintenance, but also may introduce inconsistencies and security risks. 2) Data consistency and integration: Data consistency and integration issues between different databases often require additional development and management work, especially when it comes to cross-database transactions and queries.

[0003] Existing solutions have the following limitations:

[0004] 1. Distributed computing engines, such as Hadoop and Spark, can process large-scale data, but their performance for real-time data access and query is limited.

[0005] 2. Although data integration and ETL tools can help integrate different data sources, they do not solve the problem of real-time query and unified access to multiple databases. Summary of the invention

[0006] The technical task of the present invention is to address the above shortcomings and provide a big data reading and writing adaptive method and big data reading and writing system based on a decision algorithm, which can simplify data management, improve data access efficiency, enhance data consistency and security, and effectively solve the multi-database management and query complexity problems faced by enterprises, providing enterprises with a more efficient and consistent data access and management experience.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A big data read and write adaptive method based on decision-making algorithms, which processes high-speed data read and write operations in real time in business systems, and dynamically adjusts data read and write strategies and resource allocation by analyzing data access patterns and system load conditions;

[0009] Preload data through intelligent prefetching strategies to optimize the storage and access paths of hot data to reduce user waiting time and system resource consumption.

[0010] This method achieves cross-database query optimization by designing optimization algorithms and index structures, thereby improving the efficiency and response speed of cross-database queries; supports seamless integration of multiple data formats and data storage technologies, and ensures smooth migration and exchange of data between different database systems.

[0011] Further, data access pattern analysis includes:

[0012] Data access pattern recognition: By monitoring the data access patterns of users and applications, frequently accessed data and hot data are identified. These data usually have high read and write requirements, and memory cache or other high-speed storage optimization is prioritized to speed up data access.

[0013] Hotspot data management: Implement specific cache strategies for hotspot data, including using in-memory databases or cache services to reduce frequent access to disks or remote databases, thereby significantly improving the speed of data reading and writing;

[0014] Data access path optimization: Dynamically adjust the data reading and writing paths and algorithms based on the real-time monitored data access paths; for example, for complex queries, parallel processing or index acceleration may be preferred to improve query efficiency;

[0015] Intelligent prefetch strategy: To further optimize data access speed, implement a prefetch strategy to predict data that users may need and load it into cache or memory in advance to reduce user waiting time and system response delay.

[0016] Furthermore, load monitoring and dynamic adjustment include:

[0017] Real-time load monitoring: By monitoring system indicators, including CPU utilization, memory usage, and network bandwidth, the system load can be evaluated in real time;

[0018] Load classification strategy: Different data processing and scheduling strategies are adopted according to different load levels; for example, when the load is high, the priority and resource allocation of data processing tasks can be adjusted to ensure the timely completion of important business operations.

[0019] Dynamic resource allocation: Adjust resource allocation for data storage and processing tasks according to changes in load, including automatic expansion or contraction of computing resources, optimization of storage configuration, and other measures to cope with sudden increases or decreases in load;

[0020] Adaptive adjustment strategy: Continuously optimize the load adjustment strategy through learning and adaptation; including predicting future load trends based on historical data and models, adjusting system configuration in advance, and preventing potential performance problems.

[0021] Furthermore, the method supports real-time data synchronization and consistency maintenance among multiple database services, and ensures the timeliness and consistency of data updates through asynchronous data synchronization technology and transaction management technology; including:

[0022] Asynchronous data synchronization: Using asynchronous data synchronization technology, data update operations are synchronized from the primary database to the backup database or other related database services; this method can reduce the load on the primary database and reduce the delay of data synchronization;

[0023] Transaction consistency guarantee: When synchronizing data between multiple database services, transaction management technology is used to ensure the atomicity and consistency of data updates, including transaction commit and rollback mechanisms, as well as conflict resolution strategies, to avoid data inconsistency.

[0024] Distributed data synchronization management: For distributed database environments, data exchange and synchronization coordination between nodes are used to ensure data consistency and integrity of the overall system;

[0025] Automated monitoring and alarming: The built-in data synchronization monitoring and alarming mechanism monitors the status and performance indicators of data synchronization in real time. Once synchronization delays or errors are detected, automated remedial measures are taken immediately, such as resynchronization or notifying administrators for manual intervention.

[0026] Furthermore, the method provides a flexible parameter configuration function, allowing users to adjust the engine parameter settings according to specific business needs to achieve the best data management and operation effects; including:

[0027] Customized parameter settings: Users can flexibly configure various parameters of the engine according to specific business scenarios and needs, including cache size, data synchronization frequency, and query optimization parameters;

[0028] Default parameter optimization: By presetting default parameter settings, it is suitable for common business situations. Users can choose to directly adopt the default settings or make personalized adjustments;

[0029] Dynamic parameter adjustment: Supports dynamic parameter adjustment function, users can adjust parameters according to actual needs while the system is running without stopping or reconfiguring the system;

[0030] Parameter Impact Analysis: When users modify parameter settings, relevant parameter impact analysis reports are provided to help users evaluate the potential impact of modifications on system performance and efficiency, so as to make informed decisions.

[0031] Furthermore, it supports the ability to store and process large-scale data, including data sharding management, dynamic resource allocation, and incremental data processing, to cope with the ever-increasing business data and complex database environments;

[0032] It has horizontal expansion capabilities, supporting seamless system expansion under growing data volume and load conditions; by adding nodes or distributed architecture, it can handle larger-scale data storage and processing needs, ensuring system scalability and performance stability;

[0033] In order to process large amounts of data, data sharding and partition management are used to store data in multiple nodes or databases to achieve effective data management and distributed processing. This approach not only improves the system's parallel processing capabilities, but also optimizes data access and query efficiency;

[0034] In order to reduce the pressure on system resources, it supports incremental data processing and updating, that is, only processing the changed part of the data, without operating the entire data set every time. This method is particularly effective in processing real-time data and continuous updates, and can reduce the time and cost of data processing;

[0035] Adopt optimized storage solutions for different types of data, such as storing hot data in high-speed storage media (such as memory or solid-state drives) and storing cold data in lower-cost storage media (such as disks or cloud storage) to achieve a balance between storage cost and performance;

[0036] In order to effectively manage historical data and free up storage space, the engine implements automated data cleansing and archiving strategies. Based on the frequency of data access and retention policies, it automatically cleans up expired or no longer needed data, while retaining necessary historical data to support analysis and compliance requirements.

[0037] The present invention also claims a big data reading and writing system based on a decision algorithm, including a data access pattern analysis and optimization module, a load monitoring and dynamic adjustment module, an automated data synchronization module, a function encapsulation module, a parameter configuration module, and a large-scale data storage and processing module;

[0038] The system implements the above-mentioned big data reading and writing method based on decision algorithm.

[0039] Furthermore, the system is deployed as follows:

[0040] 1) Build web services to provide users with visual installation wizards and operation and maintenance functions;

[0041] 2) Prepare at least three resource servers for building data warehouses and installing database services such as Redis and MySQL, and support the expansion of more server resources at any time in the future; the specific memory, CPU and disk size are determined according to business needs;

[0042] 3) Install the web front-end and back-end services on a separate business server or one of the resource servers in step 2);

[0043] 4) Start the web service, check the service startup log to see if it is started successfully; access the web service through the browser;

[0044] 5) Create and complete account information, log in to the web service, and follow the wizard to install;

[0045] 6) Create a name for the read / write engine according to the wizard prompts, and create different read / write engines using different server resources;

[0046] 7) Configure the resource server according to the wizard prompts, and configure the connection information of at least three servers, including IP addresses and password-free login information;

[0047] 8) Select the database services to be installed, including Redis, MySQL, Hbase, Hive, HDFS, Yarn services and their respective versions;

[0048] 9) Configure the specific configuration items of each service, including disk path, memory size, and number of CPU cores;

[0049] 10) After selecting the services to be installed, start the installation;

[0050] 11) After the installation is complete, check whether each service can be accessed normally;

[0051] 12) Check whether the data service can be accessed through the unified read and write interface.

[0052] The present invention also claims a big data reading and writing device based on a decision algorithm, comprising: at least one memory and at least one processor;

[0053] The at least one memory is used to store a machine-readable program;

[0054] The at least one processor is used to call the machine-readable program to implement the above method.

[0055] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which implement the above method when executed by a processor.

[0056] Compared with the prior art, the big data reading and writing adaptive method and big data reading and writing system based on decision algorithm of the present invention have the following beneficial effects:

[0057] 1. Reduce operation and maintenance costs: By introducing a unified data engine, the workload of configuring, managing and maintaining multiple database systems can be reduced, and related labor costs and operating costs can be reduced.

[0058] 2. Improve data access efficiency: Users can access data in different database systems through a unified interface without having to worry about the underlying database technology, thereby improving the efficiency and accuracy of data query and operation.

[0059] 3. Enhance data consistency and security: The data engine can provide a unified security control and data consistency assurance mechanism to ensure the consistency and security of data across different systems and reduce data management risks.

[0060] 4. Improve decision-making efficiency: By integrating data from different database systems, enterprises can obtain a comprehensive and consistent data view more quickly, thereby supporting more accurate decision-making and analysis.

[0061] 5. Flexibility in technology upgrade and expansion: The architectural design of multi-database data engines should comply with the principles of standardization and modularization, making it more flexible and efficient when adding new database systems or upgrading existing systems.

[0062] The realization of effective benefits will bring significant advantages and value to the company's data management, operations and decision-making, helping the company to better cope with current data processing challenges and respond to possible changes and needs in the future more flexibly and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a deployment flow chart of a big data reading and writing method based on a decision algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0065] A big data read and write adaptive method based on decision-making algorithms, which processes high-speed data read and write operations in real time in business systems, and dynamically adjusts data read and write strategies and resource allocation by analyzing data access patterns and system load conditions to improve system response speed and stability;

[0066] Preload data through intelligent prefetching strategy to optimize the storage and access path of hot data to reduce user waiting time and system resource consumption;

[0067] Supports real-time data synchronization and consistency maintenance between multiple database services, and ensures the timeliness and consistency of data updates through asynchronous data synchronization technology and transaction management technology;

[0068] Provides flexible parameter configuration functions, allowing users to adjust parameter settings according to specific business needs to achieve optimal data management and operation effects;

[0069] The ability to support large-scale data storage and processing, including data sharding management, dynamic resource allocation, and incremental data processing, to cope with the ever-growing business data and complex database environments.

[0070] The specific implementation of this method is as follows:

[0071] 1. Intelligent analysis and optimization of data access patterns can improve data access efficiency and system response speed, including:

[0072] Data access pattern recognition: By monitoring the data access patterns of users and applications, frequently accessed data and hot data can be identified. These data usually have high read and write requirements, and memory cache or other high-speed storage optimization will be given priority to speed up data access.

[0073] Hot data management: Implement specific caching strategies for hot data, such as using in-memory databases or cache services, to reduce frequent access to disks or remote databases, thereby significantly improving the speed of data reading and writing.

[0074] Data access path optimization: Dynamically adjust the data reading and writing paths and algorithms based on the real-time monitored data access paths. For example, for complex queries, parallel processing or index acceleration may be preferred to improve query efficiency.

[0075] Intelligent prefetch strategy: To further optimize data access speed, a prefetch strategy may be implemented to predict the data that users may need and load it into cache or memory in advance, reducing user waiting time and system response delay.

[0076] 2. Load monitoring and dynamic adjustment to ensure that the system can operate stably and maintain high efficiency under various load conditions, including:

[0077] Real-time load monitoring: By monitoring system indicators such as CPU utilization, memory usage, network bandwidth, etc., the system load can be evaluated in real time;

[0078] Load classification strategy: Different data processing and scheduling strategies are adopted according to different load levels; for example, when the load is high, the priority and resource allocation of data processing tasks can be adjusted to ensure the timely completion of important business operations.

[0079] Dynamic resource allocation: Adjust resource allocation for data storage and processing tasks according to changes in load, including automatic expansion or contraction of computing resources, optimization of storage configuration, and other measures to cope with sudden increases or decreases in load;

[0080] Adaptive adjustment strategy: Continuously optimize the load adjustment strategy through learning and adaptation; for example, predict future load trends based on historical data and models, adjust the system configuration in advance, and prevent potential performance problems.

[0081] 3. Automated data synchronization mechanism to ensure data consistency and real-time performance between multiple database services, including:

[0082] Asynchronous data synchronization: Using asynchronous data synchronization technology, data update operations are synchronized from the primary database to the backup database or other related database services; this method can reduce the load on the primary database and reduce the delay of data synchronization;

[0083] Transaction consistency guarantee: When synchronizing data between multiple database services, transaction management technology is used to ensure the atomicity and consistency of data updates, including transaction commit and rollback mechanisms, as well as conflict resolution strategies, to avoid data inconsistency.

[0084] Distributed data synchronization management: For distributed database environments, data exchange and synchronization coordination between nodes are used to ensure data consistency and integrity of the overall system;

[0085] Automated monitoring and alarming: The built-in data synchronization monitoring and alarming mechanism monitors the status and performance indicators of data synchronization in real time. Once synchronization delays or errors are detected, automated remedial measures are taken immediately, such as resynchronization or notifying administrators for manual intervention.

[0086] 4. Transparent user experience and functional encapsulation, allowing users to focus on business needs rather than technical details; including

[0087] Simplified interface design: Provides easy-to-use interfaces and operation interfaces, allowing users to implement complex data management and query functions through a small number of operations without having to have an in-depth understanding of the underlying data structure and algorithms.

[0088] Hiding complexity: Complex data processing and algorithms are encapsulated internally, so users do not need to directly operate the database language or complex query statements, but can complete the required operations through high-level abstract interfaces.

[0089] Visual management interface: Provides an intuitive management interface through which users can monitor system operation status, configure parameters, and view real-time data synchronization, thus improving the visualization and friendliness of operations.

[0090] Intelligent suggestions and optimization: Based on data analysis capabilities, the system can provide users with intelligent suggestions and optimization solutions to help users better adjust system configurations and parameters to achieve optimal performance and resource utilization.

[0091] 5. Flexible parameter configuration function to adapt to different business needs and environmental changes; including:

[0092] Custom parameter settings: Users can flexibly configure various parameters of the engine according to specific business scenarios and needs, including cache size, data synchronization frequency, and query optimization parameters;

[0093] Default parameter optimization: By presetting default parameter settings, it is suitable for most common business situations. Users can choose to directly adopt the default settings or make personalized adjustments;

[0094] Dynamic parameter adjustment: Supports dynamic parameter adjustment function, users can adjust parameters according to actual needs while the system is running without stopping or reconfiguring the system;

[0095] Parameter Impact Analysis: When users modify parameter settings, relevant parameter impact analysis reports are provided to help users evaluate the potential impact of modifications on system performance and efficiency, so as to make informed decisions.

[0096] 6. Considering the rapid growth trend of modern enterprise data, we have designed technical solutions to cope with the challenges of large-scale data and data growth:

[0097] Horizontal expansion: It has horizontal expansion capabilities and supports seamless system expansion under the conditions of growing data volume and load. It can handle larger-scale data storage and processing needs by adding nodes or distributed architecture, ensuring the scalability and performance stability of the system.

[0098] Data sharding and partition management: For the processing of large amounts of data, data sharding and partition management are used to store data in multiple nodes or databases to achieve effective data management and distributed processing. This approach not only improves the system's parallel processing capabilities, but also optimizes data access and query efficiency;

[0099] Incremental data processing: In order to reduce the pressure on system resources, support incremental data processing and updating, that is, only process the part of the data that has changed, without having to operate the entire data set every time. This method is particularly effective in processing real-time data and continuous updates, and can reduce the time and cost of data processing;

[0100] Data storage optimization: Adopt optimized storage solutions for different types of data, such as storing hot data in high-speed storage media (such as memory or solid-state drives) and storing cold data in lower-cost storage media (such as disks or cloud storage) to achieve a balance between storage cost and performance;

[0101] Automated data cleansing and archiving: To effectively manage historical data and free up storage space, the engine implements automated data cleansing and archiving strategies. Based on the frequency of data access and retention policies, expired or no longer needed data is automatically cleaned up, while necessary historical data is retained to support analysis and compliance requirements.

[0102] This method is an innovative data management and processing solution that effectively solves the data management challenges faced by modern enterprises through its unique features and advanced technology. It not only improves the efficiency of data reading and writing and the system response speed, but also supports data synchronization and consistency maintenance in a multi-database environment, providing important support and guarantee for the development of enterprise data-driven.

[0103] By introducing a unified data engine, data management and operation and maintenance can be greatly simplified, reducing dependence on multiple database systems.

[0104] Users can access different types of data through a unified interface and query language without having to understand the underlying details of each database system, thereby improving data access efficiency.

[0105] Through the built-in data consistency and security control mechanism, the risk of data management can be reduced, the efficiency of data operations can be improved, and data consistency and security can be enhanced.

[0106] An embodiment of the present invention also provides a big data reading and writing system based on a decision algorithm, including a data access pattern analysis and optimization module, a load monitoring and dynamic adjustment module, an automatic data synchronization module, a function encapsulation module, a parameter configuration module and a large-scale data storage and processing module; the system implements the big data reading and writing method based on the decision algorithm described in the above embodiment.

[0107] The system is deployed as follows:

[0108] 1) Build web services to provide users with visual installation wizards and operation and maintenance functions;

[0109] 2) Prepare at least three resource servers for building data warehouses and installing database services such as Redis and MySQL, and support the expansion of more server resources at any time in the future; the specific memory, CPU and disk size are determined according to business needs;

[0110] 3) Install the web front-end and back-end services on a separate business server or one of the resource servers in step 2);

[0111] 4) Start the web service, check the service startup log to see if it is started successfully; access the web service through the browser;

[0112] 5) Create and complete account information, log in to the web service, and follow the wizard to install;

[0113] 6) Create a name for the read / write engine according to the wizard prompts, and create different read / write engines using different server resources;

[0114] 7) Configure the resource server according to the wizard prompts, and configure the connection information of at least three servers, including IP addresses and password-free login information;

[0115] 8) Select the database services to be installed, including Redis, MySQL, Hbase, Hive, HDFS, Yarn services and their respective versions;

[0116] 9) Configure the specific configuration items of each service, including disk path, memory size, and number of CPU cores;

[0117] 10) After selecting the services to be installed, start the installation;

[0118] 11) After the installation is complete, check whether each service can be accessed normally;

[0119] 12) Check whether the data service can be accessed through the unified read and write interface.

[0120] An embodiment of the present invention further provides a big data reading and writing device based on a decision algorithm, comprising: at least one memory and at least one processor;

[0121] The at least one memory is used to store a machine-readable program;

[0122] The at least one processor is used to call the machine-readable program to implement the big data reading and writing method based on the decision algorithm described in the above embodiment.

[0123] The embodiment of the present invention also provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by the processor, the big data reading and writing method based on the decision algorithm described in the above embodiment is implemented. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0124] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0125] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0126] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0127] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0128] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A big data reading and writing adaptive method based on decision algorithm, characterized in that: Process high-speed data read and write operations in real time in business systems, and dynamically adjust data read and write strategies and resource allocation by analyzing data access patterns and system load conditions; Preload data through intelligent prefetching strategies to optimize the storage and access paths of hot data to reduce user waiting time and system resource consumption.

2. According to the big data reading and writing adaptive method based on decision algorithm in claim 1, it is characterized in that: Data access pattern analysis includes: Data access pattern recognition: By monitoring the data access patterns of users and applications, frequently accessed data and hot data are identified, and memory cache or other high-speed storage optimization is prioritized for them; Hot data management: Implement specific cache strategies for hot data, including using in-memory databases or cache services to reduce frequent access to disks or remote databases; Data access path optimization: Dynamically adjust the data reading and writing paths and algorithms based on real-time monitored data access paths; Intelligent prefetching strategy: Implement prefetching strategy to predict data that users may need and load it into cache or memory in advance to reduce user waiting time and system response delay.

3. According to the big data reading and writing adaptive method based on decision algorithm in claim 1, it is characterized in that: Load monitoring and dynamic adjustment include: Real-time load monitoring: By monitoring system indicators, including CPU utilization, memory usage, and network bandwidth, the system load can be evaluated in real time; Load classification strategy: adopt different data processing and scheduling strategies according to different load levels; adjust the priority and resource allocation of data processing tasks when the load is high; Dynamic resource allocation: adjust resource allocation for data storage and processing tasks according to changes in load, including automatic expansion or contraction of computing resources and optimization of storage configuration; Adaptive adjustment strategy: Continuously optimize the load adjustment strategy through learning and adaptation; including predicting future load trends based on historical data and models, adjusting system configuration in advance, and preventing potential performance problems.

4. According to the big data reading and writing adaptive method based on decision algorithm in claim 1, it is characterized in that: Supports real-time data synchronization and consistency maintenance between multiple database services, and ensures the timeliness and consistency of data updates through asynchronous data synchronization technology and transaction management technology; including: Asynchronous data synchronization: Use asynchronous data synchronization technology to synchronize data update operations from the primary database to the backup database or other related database services; Transaction consistency guarantee: When synchronizing data between multiple database services, transaction management technology is used to ensure the atomicity and consistency of data updates, including transaction commit and rollback mechanisms, as well as conflict resolution strategies, to avoid data inconsistency. Distributed data synchronization management: For distributed database environments, data exchange and synchronization coordination between nodes are used to ensure data consistency and integrity of the overall system; Automated monitoring and alarming: Through the built-in data synchronization monitoring and alarm mechanism, the status and performance indicators of data synchronization are monitored in real time. Once synchronization delays or errors are found, automated remedial measures are taken immediately.

5. According to the big data reading and writing adaptive method based on decision algorithm in claim 1, it is characterized in that: Provides flexible parameter configuration functions, allowing users to adjust the engine parameter settings according to specific business needs, including: Customized parameter settings: Users can flexibly configure various parameters of the engine according to specific business scenarios and needs, including cache size, data synchronization frequency, and query optimization parameters; Default parameter optimization: By presetting default parameter settings, it is suitable for common business situations. Users can choose to directly adopt the default settings or make personalized adjustments; Dynamic parameter adjustment: Supports dynamic parameter adjustment function, users can adjust parameters according to actual needs while the system is running without stopping or reconfiguring the system; Parameter Impact Analysis: When users modify parameter settings, relevant parameter impact analysis reports are provided to help users evaluate the potential impact of modifications on system performance and efficiency.

6. A big data reading and writing adaptive method based on decision algorithm according to claim 1 or 5, characterized in that: Support for large-scale data storage and processing capabilities, including data sharding management, dynamic resource allocation, and incremental data processing to cope with growing business data and complex database environments; Support seamless system expansion under growing data volume and load conditions; handle larger-scale data storage and processing needs by adding nodes or distributed architecture; For the processing of large amounts of data, data sharding and partition management are used to store data in multiple nodes or databases; Supports incremental data processing and updating, that is, only processing the changed part of the data without operating the entire data set every time; Adopt optimized storage solutions for different types of data; Automatically clean up data that is expired or no longer needed, based on how often the data is accessed and retention policies, while retaining necessary historical data to support analytics and compliance needs.

7. A big data reading and writing system based on decision-making algorithm, characterized in that: It includes data access pattern analysis and optimization module, load monitoring and dynamic adjustment module, automatic data synchronization module, function encapsulation module, parameter configuration module and large-scale data storage and processing module; The system implements the big data reading and writing method based on the decision algorithm described in any one of claims 1 to 6.

8. The big data reading and writing system based on decision algorithm according to claim 1 is characterized in that: The system is deployed as follows: 1) Build web services to provide users with visual installation wizards and operation and maintenance functions; 2) Prepare at least three resource servers for building data warehouses and installing database services, and support the expansion of more server resources at any time in the future; 3) Install the web front-end and back-end services on a separate business server or one of the resource servers in step 2); 4) Start the web service and check the service startup log to see if it is started successfully; access the web service through a browser; 5) Create and complete account information, log in to the web service, and follow the wizard to install; 6) Create a name for the read / write engine according to the wizard prompts, and create different read / write engines using different server resources; 7) Configure the resource server according to the wizard prompts, and configure the connection information of at least three servers, including IP addresses and password-free login information; 8) Select the database services to be installed, including Redis, MySQL, Hbase, Hive, HDFS, Yarn services and their respective versions; 9) Configure the specific configuration items of each service, including disk path, memory size, and number of CPU cores; 10) After selecting the services to be installed, start the installation; 11) After the installation is complete, check whether each service can be accessed normally; 12) Check whether the data service can be accessed through the unified read and write interface.

9. A big data reading and writing device based on a decision algorithm, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 6.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images