Communication data storage system and method based on distributed architecture and intelligent management

Abstract

The invention provides a communication data storage system and method based on a distributed architecture and intelligent management, and the system comprises a distributed storage management module which is used for creating, configuring, monitoring and maintaining each node, dynamically storing communication data in each node, and distributing a read-write request, node faults are detected and processed; the data encryption and security module is used for performing fine-grained authority management on a user and an administrator and performing security management on stored communication data and backup data by adopting a multi-factor authentication mechanism; the automatic operation, maintenance and monitoring module is used for carrying out automatic operation, maintenance and monitoring; the green computing and energy management module is used for adjusting resource configuration of the system; the data backup and disaster recovery module is used for carrying out regular backup on the stored communication data, carrying out remote redundant storage on the backup data and carrying out disaster recovery on system faults; and the user interaction and management module is used for providing a friendly operation interface.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a communication data storage system and method based on a distributed architecture and intelligent management. Background Art

[0002] With the development of the mobile Internet and the Internet of Things (IoT), the volume of communication data has increased exponentially, posing higher requirements for data storage systems. Although existing communication data storage systems have achieved certain achievements, they also have the following defects:

[0003] 1. Insufficient data security and privacy protection, vulnerable to the risks of network attacks and privacy leakage;

[0004] 2. Poor scalability and extensibility, difficult to cope with the rapidly growing data volume and high-concurrency access requirements;

[0005] 3. Difficult to ensure data consistency and integrity, and synchronization delays and error recovery are common problems in a distributed environment;

[0006] 4. Low cost-effectiveness, high maintenance costs and low resource utilization increase the operation burden;

[0007] 5. Lack of flexibility and adaptability, difficult to quickly respond to changing technical environments and business requirements;

[0008] 6. Difficult to comply with regulations and regulatory requirements, especially facing complex regulatory requirements during cross-border operations;

[0009] 7. Insufficient interoperability and standardization, and difficult to integrate products from different manufacturers;

[0010] 8. Serious problems of high energy consumption and carbon emissions, data centers becoming big energy consumers and putting pressure on the environment. Summary of the Invention

[0011] The purpose of the present invention is to provide a communication data storage system and method based on a distributed architecture and intelligent management, aiming to solve the above problems in the prior art.

[0012] An embodiment of the present invention provides a communication data storage system based on a distributed architecture and intelligent management, including:

[0013] A distributed storage management module, connected to a data encryption and security module, an automated operation and maintenance and monitoring module, a green computing and energy management module, a data backup and disaster recovery module, and a user interaction and management module, for creating, configuring, monitoring, and maintaining each distributed storage node, dynamically storing communication data in each node, allocating read and write requests from the user interaction and management module, and detecting and handling node failures;

[0014] A data encryption and security module, connected to the distributed storage management module, data backup and disaster recovery module, and user interaction and management module, is used to perform fine-grained permission management on users and administrators accessing the system and adopt a multi-factor authentication mechanism, as well as perform security management on the stored communication data and backup data;

[0015] An automated operation and maintenance and monitoring module, connected to the distributed storage management module, green computing and energy management module, and user interaction and management module, is used to perform automated operation and maintenance and monitoring according to the status of each node, generate system performance optimization results, system logs, and audit reports;

[0016] A green computing and energy management module, connected to the distributed storage management module, automated operation and maintenance and monitoring module, and user interaction and management module, is used to adjust the resource configuration of the system according to the load information of each node and the system performance optimization results to obtain an energy-saving measure report;

[0017] A data backup and disaster recovery module, connected to the distributed storage management module, data encryption and security module, and user interaction and management module, is used to perform regular backups on the stored communication data, generate backup data, perform off-site redundant storage on the backup data, and perform disaster recovery on system failures according to a pre-defined emergency plan;

[0018] A user interaction and management module, connected to the distributed storage management module, data encryption and security module, automated operation and maintenance and monitoring module, green computing and energy management module, and data backup and disaster recovery module, is used to provide a friendly operation interface for users and administrators accessing the system.

[0019] An embodiment of the present invention provides a communication data storage method based on a distributed architecture and intelligent management, including:

[0020] Create, configure, monitor, and maintain each distributed storage node through the distributed storage management module, dynamically store communication data in each node, allocate read and write requests from the user interaction and management module, and detect and handle node failures;

[0021] Perform fine-grained permission management on users and administrators accessing the system and adopt a multi-factor authentication mechanism through the data encryption and security module, as well as perform security management on the stored communication data and backup data;

[0022] Perform automated operation and maintenance and monitoring according to the status of each node through the automated operation and maintenance and monitoring module, generate system performance optimization results, system logs, and audit reports;

[0023] The green computing and energy management module adjusts the resource allocation of the system according to the load information of each node and the system performance optimization result, and obtains an energy-saving measure report;

[0024] The data backup and disaster recovery module regularly backs up the stored communication data, generates backup data, redundantly stores the backup data in a different location, and performs disaster recovery on system failures according to a pre-established emergency plan;

[0025] The user interaction and management module provides a friendly operation interface for users and administrators accessing the system.

[0026] An embodiment of the present invention further provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the steps of the above communication data storage method based on a distributed architecture and intelligent management are implemented.

[0027] An embodiment of the present invention further provides a computer-readable storage medium, on which an implementation program for information transmission is stored. When the program is executed by a processor, the steps of the above communication data storage method based on a distributed architecture and intelligent management are implemented.

[0028] Adopting the embodiment of the present invention may include the following beneficial effects: The communication data storage system proposed in the embodiment of the present invention is a data storage system for large-scale communication networks. This system solves multiple problems existing in the existing system (such as data security, scalability, cost-effectiveness, etc.) through a distributed architecture, advanced encryption technology, automated operation and maintenance tools, and green computing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0030] Figure 1 It is a schematic diagram of a communication data storage system based on a distributed architecture and intelligent management according to an embodiment of the present invention;

[0031] Figure 2 It is a flowchart of a communication data storage method based on a distributed architecture and intelligent management according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will clearly and completely describe the technical solutions in one or more embodiments of this specification in conjunction with the accompanying drawings in one or more embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this document.

[0033] System Embodiment

[0034] According to an embodiment of the present invention, there is provided a communication data storage system based on a distributed architecture and intelligent management. Figure 1 It is a schematic diagram of the communication data storage system based on a distributed architecture and intelligent management according to an embodiment of the present invention, as Figure 1 shown. The communication data storage system based on a distributed architecture and intelligent management according to an embodiment of the present invention specifically includes:

[0035] A distributed storage management module 11, which is connected to a data encryption and security module, an automated operation and maintenance and monitoring module, a green computing and energy management module, a data backup and disaster recovery module, and a user interaction and management module, is used to create, configure, monitor, and maintain each distributed storage node, dynamically store communication data in each node, allocate read and write requests from the user interaction and management module, and detect and handle node failures;

[0036] A data encryption and security module 12, which is connected to the distributed storage management module, the data backup and disaster recovery module, and the user interaction and management module, is used to perform fine-grained permission management on users and administrators accessing the system and adopt a multi-factor authentication mechanism, and perform security management on the stored communication data and backup data. Specifically, it is used for:

[0037] Defining different roles for users and administrators accessing the system according to the organizational structure and business processes, setting corresponding operation permission lists for each role, authenticating the identities of users and administrators using several verification means, and encrypting the stored communication data and backup data end-to-end using corresponding encryption algorithms;

[0038] Among them, the operation permission list is dynamically adjusted according to different condition combinations;

[0039] The corresponding encryption algorithms include symmetric encryption, asymmetric encryption, and hybrid encryption;

[0040] The automated operation and maintenance and monitoring module 13, connected to the distributed storage management module, the green computing and energy management module, and the user interaction and management module, is used to perform automated operation and maintenance and monitoring according to the status of each node, and generate system performance optimization results, system logs, and audit reports;

[0041] The green computing and energy management module 14, connected to the distributed storage management module, the automated operation and maintenance and monitoring module, and the user interaction and management module, is used to adjust the resource configuration of the system according to the load information of each node and the system performance optimization results, and obtain an energy-saving measure report;

[0042] The data backup and disaster recovery module 15, connected to the distributed storage management module, the data encryption and security module, and the user interaction and management module, is used to regularly back up the stored communication data, generate backup data, perform off-site redundant storage on the backup data, and perform disaster recovery on system failures according to a pre-defined emergency plan;

[0043] The user interaction and management module 16, connected to the distributed storage management module, the data encryption and security module, the automated operation and maintenance and monitoring module, the green computing and energy management module, and the data backup and disaster recovery module, is used to provide a friendly operation interface for users and administrators accessing the system, specifically for:

[0044] Visually display the status, load information, system performance optimization results, system logs, audit reports, energy-saving measure reports, and system disaster recovery status of each node correspondingly;

[0045] The system further includes:

[0046] The interoperability and standardization module, connected to the data encryption and security module and the user interaction and management module, is used to check whether the system communication process follows the secure transmission standard, and provide open API interfaces;

[0047] The compliance and legal compliance module, connected to the automated operation and maintenance and monitoring module and the data backup and disaster recovery module, is used to conduct compliance reviews on system logs, audit reports, and the communication data backup process.

[0048] The following details the above technical solutions of the embodiments of the present invention in combination with the specific situation of the communication data storage system based on a distributed architecture and intelligent management of the embodiments of the present invention.

[0049] The communication data storage system proposed in the embodiments of the present invention includes multiple core modules, and each module is responsible for specific functions or tasks to ensure the efficient, secure, and reliable operation of the entire system. The main modules include:

[0050] 1. Distributed Storage Management Module

[0051] Node Management: Responsible for creating, configuring, monitoring, and maintaining each distributed storage node.

[0052] Load Balancing: Intelligent allocation of read and write requests to different nodes to ensure high availability and performance of the system.

[0053] Fault Tolerance and Recovery: Detect and handle node failures, and automatically perform data replication and migration to maintain service continuity.

[0054] 2. Data Encryption and Security Module

[0055] End-to-End Encryption Engine: Implement high-intensity encryption for data in transit to protect it from eavesdropping and tampering.

[0056] Fine-Grained Permission Control: Define user roles and permissions to ensure that only authorized personnel can access sensitive information.

[0057] Multi-Factor Authentication (MFA): Provide an additional security layer to increase account security.

[0058] 3. Automated Operations and Monitoring Module

[0059] Health Check and Self-Healing Mechanism: Regularly scan the system status and proactively detect and fix potential problems.

[0060] Performance Optimization: Use machine learning algorithms to analyze traffic patterns and dynamically adjust resource allocation.

[0061] Logging and Audit Trail: Completely record all operation logs to support post-event review and compliance checks.

[0062] 4. Green Computing and Energy Management Module

[0063] Energy-Saving Scheduling Strategy: Reasonably plan hardware usage according to the actual load situation to reduce unnecessary energy consumption.

[0064] Renewable Energy Integration: Prioritize the use of clean energy for power supply to reduce carbon emissions.

[0065] 5. Interoperability and Standardization Module

[0066] Standard Protocol Support: Follow industry norms and technical standards to improve compatibility with other systems.

[0067] Open API Interface: Provide a convenient access point for third-party developers to promote the construction of the ecosystem.

[0068] 6. Compliance and Legal Conformance Module

[0069] Policy Interpretation and Update: Continuously track changes in domestic and international laws and regulations to ensure that the system always complies with the latest requirements.

[0070] Data Governance Framework: Establish internal rules and regulations to guide various activities within the data life cycle.

[0071] 7. User Interaction and Management Module

[0072] Web Interface / CLI Tool: Provide a friendly operation interface for administrators and ordinary users.

[0073] API Service: A program interface for developers, facilitating integration into other applications.

[0074] 8. Data Backup and Disaster Recovery Module

[0075] Regular Backup Plan: Set up automated full or incremental backup tasks.

[0076] Off-site Redundant Storage: Store copies of important data in different geographical locations to prevent data loss caused by regional disasters.

[0077] Quick Recovery Process: Develop a detailed emergency response plan to quickly resume normal operations in the event of a major accident.

[0078] The following are the connection methods between the various modules in the embodiments of the present invention:

[0079] 1. Connection of the Distributed Storage Management Module to Other Modules

[0080] With the Data Encryption and Security Module: During data writing or reading, the Distributed Storage Management Module needs to call the encryption / decryption services provided by the Data Encryption and Security Module to ensure that all data stored on the nodes is encrypted.

[0081] With the Automated Operation and Maintenance and Monitoring Module: Share node status information, such as health status, performance metrics, etc., so that the Automated Operation and Maintenance and Monitoring Module can make timely decisions (for example, divert traffic from a faulty node).

[0082] With the Green Computing and Energy Management Module: Provide information about the load of each node to help the Green Computing and Energy Management Module formulate energy-saving scheduling strategies.

[0083] 2. Connection of the Data Encryption and Security Module to Other Modules

[0084] With the User Interaction and Management Module: Provide authentication and permission control functions for the Web interface / CLI tool and API service to ensure that only authorized users can perform specific operations.

[0085] Interoperability and Standardization Module: Adhere to industry-standard security protocols (such as TLS) to ensure secure communication with other systems.

[0086] Data Backup and Disaster Recovery Module: Ensure that backup data is also protected by strong encryption and can be correctly decrypted during recovery.

[0087] 3. Connections between the Automated Operations and Monitoring Module and Other Modules

[0088] Distributed Storage Management Module: As mentioned before, receive node status information from this module for intelligent operations decision-making.

[0089] Green Computing and Energy Management Module: Adjust the usage of hardware resources according to the performance optimization results, indirectly affecting energy consumption.

[0090] Compliance and Legal Compliance Module: Regularly generate system logs and audit reports for compliance checks.

[0091] 4. Connections between the Green Computing and Energy Management Module and Other Modules

[0092] Distributed Storage Management Module: Obtain node load information to guide the implementation of energy-saving measures.

[0093] Automated Operations and Monitoring Module: Adjust resource allocation based on performance optimization suggestions to achieve energy conservation and emission reduction effects.

[0094] 5. Connections between the Interoperability and Standardization Module and Other Modules

[0095] User Interaction and Management Module: Allow third-party applications to access through open API interfaces and also support the standard protocols of internal management tools.

[0096] Data Encryption and Security Module: Ensure that the security transmission standards are followed during communication.

[0097] 6. Connections between the Compliance and Legal Compliance Module and Other Modules

[0098] Automated Operations and Monitoring Module: Collect system logs and operation records as the basis for compliance audits.

[0099] Data Backup and Disaster Recovery Module: Ensure that the backup process complies with relevant regulatory requirements and can handle potential data retention period regulations.

[0100] 7. Connections between the User Interaction and Management Module and Other Modules

[0101] With all other modules: As the human - machine interaction entry, it directly or indirectly interacts with each module. Whether it is configuration management, status viewing, or executing specific tasks, they all need to be completed through this module.

[0102] 8. Connection between the data backup and disaster recovery module and other modules

[0103] With the distributed storage management module: It relies on the storage infrastructure provided by it for data backup.

[0104] With the data encryption and security module: Ensure the security and integrity of the backup data.

[0105] With the compliance and legal compliance module: Comply with regulations regarding data retention and privacy protection.

[0106] That is, the various modules in the entire system do not exist in isolation, but are closely connected and cooperate with each other. Each module has a clear functional positioning. In actual operation, they will continuously exchange information and services according to business requirements to jointly ensure the high - efficiency, stability, and security of the system.

[0107] In summary, the embodiments of the present invention aim to provide a more secure, efficient, economical, and environmentally friendly communication data storage system, specifically including:

[0108] 1. Distributed architecture:

[0109] 1.1 Adopt a decentralized node layout to ensure the high availability and fault - tolerance of the system.

[0110] 1.1.1 Distributed architecture selection

[0111] Use a distributed file system (DFS): Such as HDFS of Hadoop, Ceph, etc. These systems disperse the load by splitting files into blocks and storing them on multiple nodes, and provide a redundancy mechanism to improve fault - tolerance.

[0112] Based on blockchain technology: Utilize the characteristics of blockchain, such as immutability and trustlessness, to build a more secure and reliable decentralized storage network. For example, IPFS (InterPlanetary File System) combines the characteristics of P2P technology and content - based addressing, enabling files to be efficiently distributed globally.

[0113] 1.1.2 Data distribution strategy

[0114] Hash partitioning: Calculate the hash of the data according to specific rules (such as file name, user ID, etc.) and distribute it to different nodes for storage. This can not only evenly disperse the data but also speed up the search speed.

[0115] Consistent Hashing Algorithm: When new nodes are added or old nodes leave, consistent hashing can minimize the amount of data migration and maintain the stability and efficiency of the system.

[0116] 1.1.3 Redundancy and Replication Mechanism

[0117] Multi-copy Preservation: Create multiple copies for each piece of data and distribute them across nodes in different geographical locations to ensure data availability even if some nodes fail.

[0118] Preferably, the embodiments of the present invention can adopt Erasure Coding: Compared with the simple multi-copy method, Erasure Coding can provide a higher redundancy at the same storage cost while reducing space occupancy. Among them, Erasure Coding (erasure coding) is a data protection method that divides data into multiple segments, generates redundant information for these segments, and then distributes them across different storage nodes. This method can significantly reduce the required storage space while ensuring data reliability and persistence, and has higher storage efficiency compared to the traditional multi-copy replication method.

[0119] 1.2 Support horizontal scaling simultaneously, and the number of storage nodes can be dynamically increased or decreased according to actual needs to adapt to the ever-changing data scale.

[0120] 1.2.1 Dynamic Resource Management

[0121] Automatically Discover New Nodes: Develop or integrate a service discovery mechanism that allows the system to automatically detect newly added nodes and incorporate them into the cluster to participate in task processing.

[0122] Elastic Scaling Framework: Adopt container orchestration tools such as Kubernetes, which can automatically adjust the number of running Pods according to real-time traffic changes, thereby achieving seamless horizontal scaling.

[0123] 1.2.2 Performance Optimization Measures

[0124] Read-Write Separation: For read-intensive applications, the pressure on the main database can be shared by increasing the number of read-only replicas; for write-intensive scenarios, methods such as introducing message queues or asynchronous logging can be considered to relieve the pressure.

[0125] Cache Acceleration: Deploy a distributed cache layer (such as Redis, Memcached), cache hot data in memory to reduce the disk I / O frequency and improve the response speed.

[0126] 1.2.3 Continuous Integration and Delivery (CI / CD)

[0127] Version Control and Release Process: Establish a sound code library management and continuous integration environment to ensure that each update can be quickly and stably pushed to the production environment without affecting the continuity of existing services.

[0128] Blue-Green Deployment / Rolling Update: Complete software upgrades without downtime through blue-green deployment or rolling update to ensure that the service is always online.

[0129] 2. Enhance Encryption Technology and Access Control Mechanisms:

[0130] 2.1 Implement end-to-end encryption to ensure the security of data in transit and at rest. End-to-end encryption (E2EE) means that during the process of data being encrypted at the sender and decrypted at the receiver, intermediate nodes cannot read or tamper with the data content.

[0131] 2.1.1 Select Appropriate Encryption Algorithms

[0132] Symmetric Encryption: Use the same key for encryption and decryption, which is suitable for the rapid processing of large amounts of data. Among them, AES (Advanced Encryption Standard) is a commonly used symmetric encryption algorithm.

[0133] Asymmetric Encryption: Use a pair of public and private keys for encryption and decryption, which has higher security but slower speed. Among them, RSA and ECC (Elliptic Curve Cryptography) are common asymmetric encryption algorithms.

[0134] Hybrid Encryption: Combine the advantages of symmetric and asymmetric encryption and is usually used in practical applications. For example, use asymmetric encryption to exchange the session key required for symmetric encryption, and then use symmetric encryption to transmit data.

[0135] 2.1.2 Key Management

[0136] Key Generation and Distribution: Ensure that each user has an independent key pair and complete the initial distribution of keys through a secure channel.

[0137] Key Storage: Securely store the private key on the user's device or use a Hardware Security Module (HSM) to manage and protect the key.

[0138] Key Rotation: Regularly update keys to reduce the risk of long-term use and establish a mechanism for destroying old keys.

[0139] 2.1.3 Encryption Process

[0140] Data Preparation: Preprocess the data to be transmitted on the client side, such as compression, segmentation, etc.

[0141] Encryption operation: Encrypt the data using the selected encryption algorithm to generate ciphertext.

[0142] Additional information: Add necessary metadata such as timestamps, sender IDs, etc. to each encrypted message for subsequent verification and auditing.

[0143] Transport encryption: Ensure the security of the entire communication link. The TLS / SSL protocol can be used to protect the data integrity at the transport layer.

[0144] 2.1.4 Decryption process

[0145] Authentication: The receiver first verifies the legitimacy of the sender's identity.

[0146] Key acquisition: Obtain the decryption key according to the pre-agreed method.

[0147] Decryption operation: Decrypt the received message using the correct key to restore the original data.

[0148] 2.2 Introduce fine-grained permission management and multi-factor authentication mechanisms to prevent unauthorized access. Fine-grained permission control means precisely allocating permissions according to the user's role, responsibilities, and specific operation requirements, thus implementing the principle of least privilege.

[0149] 2.2.1 Define roles and permissions

[0150] Role division: Define different roles according to the organizational structure and business processes, such as administrators, ordinary users, auditors, etc.

[0151] Permission matrix: Set a detailed list of operation permissions for each role, specifying which resources can be accessed and which actions can be performed.

[0152] 2.2.2 Attribute-based access control (ABAC)

[0153] Attribute definition: In addition to the traditional user identity, more dimensions of attributes can be considered, such as department, project, task type, etc.

[0154] Policy formulation: Create flexible access control policies that allow dynamic adjustment of permissions according to multiple condition combinations. For example, "Only members belonging to a specific project can view the documents related to that project."

[0155] 2.2.3 Multi-factor authentication (MFA)

[0156] Enhanced security: Require users to prove their identity through multiple verification means, such as password + SMS verification code, fingerprint recognition, etc.

[0157] Prevent unauthorized access: Even if the password is leaked, it is difficult to be illegally invaded.

[0158] 2.2.4 Audit and Monitoring

[0159] Logging: Comprehensively record the logs of all key operations, including login attempts, file access, modifications, etc.

[0160] Anomaly Detection: Utilize machine learning algorithms to analyze user behavior patterns and promptly detect and respond to potential security threats.

[0161] Report Generation: Regularly generate security reports to help management understand the security status of the system and take corresponding measures.

[0162] 2.3 Integration and Testing

[0163] System Integration: Seamlessly integrate the above functional modules into the communication data storage system to ensure the coordinated operation of each part.

[0164] Compatibility Testing: Verify whether the newly added security features affect the performance and user experience of the existing system.

[0165] Security Assessment: Invite a third-party organization or expert team to conduct a comprehensive security review of the entire system, identify potential vulnerabilities and repair them.

[0166] 3. Automated Operation and Maintenance Tools and Intelligent Monitoring Modules:

[0167] A. Automatically detect and repair potential faults using machine learning algorithms to improve the self-healing ability and stability of the system. Specifically, the following machine learning methods can be adopted:

[0168] (1) Supervised Learning

[0169] Classification Algorithms: Such as Support Vector Machine (SVM), Random Forest, and Gradient Boosting Trees. These algorithms can be used to classify new observations into predefined categories, such as the difference between normal operations and fault states.

[0170] Regression Models: Such as linear regression, ridge regression, or Lasso regression. They can be used to estimate the change trend of continuous variables, such as whether an increase in temperature will cause hardware overheating.

[0171] (2) Unsupervised Learning

[0172] Clustering Analysis: Algorithms such as K-means and DBSCAN can help discover unknown data distribution patterns, thereby identifying outliers or abnormal behaviors.

[0173] Principal Component Analysis (PCA): Used for dimensionality reduction, removing noise while retaining the most important feature information, which helps simplify the subsequent fault diagnosis process.

[0174] Autoencoders: A neural network structure that can detect outliers by reconstructing input data. That is, when the output is significantly different from the input, it is considered an abnormal situation.

[0175] (3) Reinforcement Learning

[0176] Q-Learning / DQN (Deep Q-Networks): Suitable for situations where online policy adjustment is required, such as selecting the optimal action according to the current environmental conditions to avoid impending failures.

[0177] PPO (Proximal Policy Optimization): Very effective for problems with dynamic changes in complex environments. It can continuously optimize the decision-making process while ensuring performance stability.

[0178] (4) Deep Learning

[0179] Convolutional Neural Network (CNN): If there are a large number of image-based logs or monitoring video materials, then CNN can be used to extract key visual features therein to assist in fault location.

[0180] Recurrent Neural Network (RNN) and its variants LSTM / GRU: Particularly suitable for processing time series data, such as the historical records of server performance metrics, to predict future health conditions.

[0181] (5) Ensemble Learning

[0182] Bagging&Boosting: For example, advanced ensemble frameworks such as XGBoost and LightGBM, which combine multiple weak classifiers to form a stronger overall prediction ability and improve the accuracy of fault detection.

[0183] (6) Graph Neural Networks (GNNs)

[0184] Graph Neural Networks: When it comes to complex network topology relationships, GNN can capture potential risk factors from the connection patterns between nodes, which is crucial for analyzing the fault propagation paths in distributed systems.

[0185] Specifically, which of the above algorithm types to choose should be determined according to the type of stored communication data. The implementation steps are as follows:

[0186] (1) Data collection: First, it is necessary to ensure that there is sufficient historical data as training samples, including but not limited to key performance indicators such as CPU usage, memory occupancy, disk I / O speed, and network traffic.

[0187] (2) Feature engineering: Preprocess the original data to extract useful feature vectors.

[0188] (3) Model training: Select a suitable machine learning algorithm to build a prediction model and train it using the labeled dataset.

[0189] (4) Validation and evaluation: Test the generalization ability and accuracy of the model through means such as cross-validation, and adjust the parameters and retrain if necessary.

[0190] (5) Deployment and go live: Once the model has been fully verified, it can be integrated into the existing operation and maintenance platform to monitor the system status in real time and trigger corresponding warning mechanisms.

[0191] (6) Feedback loop: Continuously collect newly generated data, regularly update the model parameters, and maintain its ability to adapt to the latest changes.

[0192] B. Optimize resource allocation and reduce energy consumption through real-time performance monitoring and predictive analysis.

[0193] The design steps of the automated operation and maintenance tool and intelligent monitoring module in the embodiments of the present invention are as follows:

[0194] ① Architecture design

[0195] A. Architecture of the automated operation and maintenance tool

[0196] Modular design: Divide the operation and maintenance functions into independent modules (such as deployment, configuration management, backup and recovery, etc.) for easy maintenance and expansion.

[0197] API interface integration: Provide RESTful API or other standard protocol interfaces to support interaction with other tools and services.

[0198] Script language support: Allow the use of script languages such as Python and Shell to write custom tasks to increase flexibility.

[0199] CI / CD pipeline integration: Combine with the continuous integration / continuous delivery (CI / CD) pipeline to achieve automatic testing and deployment after code changes.

[0200] B. Architecture of the intelligent monitoring system

[0201] Multi-source data collection: Collect information from multiple levels such as hardware sensors, operating system logs, and application metrics.

[0202] Real-time Processing Engine: Utilize stream processing frameworks (such as Apache Kafka, Apache Flink) to perform rapid analysis on real-time data.

[0203] Machine Learning Model: Train predictive maintenance models to identify potential failure modes; simultaneously adopt anomaly detection algorithms to capture unexpected behaviors.

[0204] Visualization Dashboard: Construct an easy-to-understand data display interface to help administrators intuitively grasp the system status.

[0205] ② Function Realization

[0206] A. Functions of Automated Operation and Maintenance Tools

[0207] Health Check: Regularly scan the status of all nodes in the cluster, including key metrics such as disk space, memory usage, and CPU utilization.

[0208] Dynamic Scaling: Automatically adjust the number of instances according to preset rules or traffic conditions to ensure service quality and cost control.

[0209] Fault Self-Healing: Once a problem is detected in a certain component, immediately trigger the repair program and attempt to restart the service or migrate tasks to other healthy nodes.

[0210] Configuration Synchronization: Centralize the management of configuration files and automatically distribute them to each node after updates to maintain consistency.

[0211] Backup and Recovery: Execute full or incremental backup operations according to the plan and quickly restore historical versions when necessary.

[0212] B. Functions of Intelligent Monitoring System

[0213] Threshold Alarm: Set reasonable upper and lower limits for important performance parameters, and issue warning notifications to relevant personnel immediately when the range is exceeded.

[0214] Trend Prediction: Predict future resource requirements based on historical data and make advance plans.

[0215] Correlation Analysis: Combine data points from multiple sources to find the root cause of problems and assist in troubleshooting.

[0216] Root Cause Location: Find the root cause of the problem through in-depth diagnosis to improve the speed of problem-solving.

[0217] Capacity Planning: Evaluate whether the existing infrastructure can meet future growth requirements and guide capacity expansion decisions.

[0218] ③ Technology Selection

[0219] Container Orchestration Platform: Kubernetes is used to manage containerized applications and services.

[0220] Log collection and analysis tools: ELK Stack (Elasticsearch, Logstash, Kibana) or the Prometheus + Grafana combination can effectively collect, index, and visualize various types of logs.

[0221] Alert service platforms: PagerDuty, Alertmanager, etc. can be used as media for alert notifications.

[0222] Machine learning frameworks: TensorFlow, PyTorch are suitable for building complex prediction models.

[0223] Preferably, the module can also include: ① Testing and verification

[0224] Unit testing: Write test cases for each individual functional module to ensure its correctness.

[0225] Integration testing: Simulate various scenarios in a real environment to verify whether the collaboration between different parts is smooth.

[0226] Load testing: Apply pressure beyond normal loads to test the system's extreme performance and stability.

[0227] User acceptance testing: Invite users to participate in the trial and obtain feedback for further product improvement.

[0228] ② Continuous iteration and optimization

[0229] Performance tuning: Continuously adjust parameter settings based on the actual performance during operation to pursue the best performance.

[0230] Security hardening: Regularly review the security of code logic and dependency libraries and patch known vulnerabilities.

[0231] 4. Green computing and sustainable development:

[0232] A. Use energy-efficient hardware components, such as low-power processors and solid-state drives (SSDs), to reduce power consumption. Specifically, it can include: Low-power processors: Adopt CPUs with lower power consumption but high performance, such as server chips based on the ARM architecture. Solid-state drives (SSDs): Compared with traditional mechanical hard drives, SSDs have faster speeds and lower energy consumption. High-efficiency power supplies: Use power devices with higher conversion efficiency to reduce power losses. Natural cooling technology: Use methods such as natural cold air or liquid cooling to replace traditional air-conditioning refrigeration to reduce energy consumption. Preferably, the layout and management of the data center can also be optimized, including: Site selection considerations: Build the data center in a cool climate to better utilize the external environment for natural cooling. Modular construction: Flexibly adjust the scale of the data center according to actual needs to avoid unnecessary resource waste. Virtualization technology: Improve the utilization rate of physical servers through virtual machines and reduce the number of idle hardware. Intelligent scheduling algorithms: Dynamically allocate computing resources based on the workload to minimize energy consumption while ensuring task completion.

[0233] B. Promote the use of renewable energy for power supply to reduce the carbon footprint. For example: Solar photovoltaic panels: Install solar panels in suitable locations to power some facilities. Or Wind turbines: Consider introducing small-scale wind power projects in combination with geographical location advantages.

[0234] 5. Industry standard construction and enhanced interoperability:

[0235] Actively participate in formulating unified data storage and processing standards to promote the seamless integration of products from different manufacturers.

[0236] Develop open API interfaces to simplify the docking process with other systems.

[0237] 6. Compliance and audit trail functions:

[0238] Build a data management module that complies with international and domestic laws and regulations to ensure legality and compliance.

[0239] Provide detailed log records and audit reports for subsequent review.

[0240] 7. Flexible deployment and support for multiple application scenarios:

[0241] Design it into a modular structure, allowing users to select different function combinations according to their own needs.

[0242] Support cloud deployment, on-premises deployment, and hybrid modes to meet diverse workload requirements.

[0243] Specific implementation examples of the communication data storage system proposed in the embodiments of the present invention are as follows:

[0244] 1. Initial configuration:

[0245] Install the necessary software packages and service components.

[0246] Set initial parameters, including cluster scale, storage capacity, security policies, etc.

[0247] 2. Daily operation and maintenance:

[0248] Conduct regular health checks to ensure that all nodes are running properly.

[0249] Adjust the system configuration according to the monitoring data to optimize the performance.

[0250] 3. Data operations:

[0251] Users can upload, download, and query communication data through the Web interface or command-line tools.

[0252] The system automatically performs backup tasks and periodically verifies the validity of the backup files.

[0253] 4. Exception handling:

[0254] When an abnormal situation is detected, the system will trigger an alarm to notify the administrator and attempt to automatically solve the problem; if the problem cannot be automatically solved, detailed diagnostic information will be provided for manual intervention.

[0255] Method embodiments

[0256] According to an embodiment of the present invention, a communication data storage method based on a distributed architecture and intelligent management is provided. Figure 2 It is a flowchart of the communication data storage method based on a distributed architecture and intelligent management according to an embodiment of the present invention, as Figure 2 shown. The communication data storage method based on a distributed architecture and intelligent management according to an embodiment of the present invention specifically includes:

[0257] Step S201, create, configure, monitor, and maintain each distributed storage node through a distributed storage management module, dynamically store communication data in each node, allocate read and write requests from the user interaction and management module, and detect and handle node failures;

[0258] Step S202, perform fine-grained permission management on users and administrators accessing the system through the data encryption and security module and adopt a multi-factor authentication mechanism, and perform security management on the stored communication data and backup data, specifically including:

[0259] The data encryption and security module defines different roles for the users and administrators accessing the system according to the organizational structure and business processes, sets corresponding operation permission lists for each role, adopts several verification means to authenticate the users and administrators, and uses the corresponding encryption algorithms to perform end-to-end encryption on the stored communication data and backup data;

[0260] Among them, the operation permission list is dynamically adjusted according to different condition combinations;

[0261] The corresponding encryption algorithms include symmetric encryption, asymmetric encryption, and hybrid encryption;

[0262] Step S203: The automated operation and maintenance and monitoring module performs automated operation and maintenance and monitoring according to the status of each node, and generates system performance optimization results, system logs, and audit reports;

[0263] Step S204: The green computing and energy management module adjusts the resource configuration of the system according to the load information of each node and the system performance optimization results, and obtains an energy-saving measure report;

[0264] Step S205: The data backup and disaster recovery module regularly backs up the stored communication data to generate backup data, performs off-site redundant storage on the backup data, and performs disaster recovery on system failures according to the pre-established emergency plan;

[0265] Step S206: The user interaction and management module provides a friendly operation interface for the users and administrators accessing the system, specifically including:

[0266] The user interaction and management module performs corresponding visual displays on the status of each node, load information, system performance optimization results, system logs, audit reports, energy-saving measure reports, and system disaster recovery situations;

[0267] The method further includes:

[0268] The interoperability and standardization module checks whether the system communication process follows the secure transmission standard and provides an open API interface;

[0269] The compliance and legal compliance module conducts compliance reviews on system logs, audit reports, and the communication data backup process.

[0270] The embodiment of the present invention is a method embodiment corresponding to the above system embodiment. The specific operations of each step can be understood with reference to the description of the system embodiment and will not be elaborated here.

[0271] Device Embodiment 1

[0272] An embodiment of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the computer program is executed by the processor, the steps described in the method embodiment are implemented.

[0273] Second Embodiment of the Device

[0274] An embodiment of the present invention provides a computer-readable storage medium, on which an implementation program for information transmission is stored, and when the program is executed by a processor, the steps described in the method embodiment are implemented.

[0275] The computer-readable storage medium described in this embodiment includes, but is not limited to: ROM, RAM, magnetic disk, optical disc, etc.

[0276] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A communication data storage system based on a distributed architecture and intelligent management, characterized in that Including: A distributed storage management module, connected to a data encryption and security module, an automated operation and maintenance and monitoring module, a green computing and energy management module, a data backup and disaster recovery module, and a user interaction and management module, for creating, configuring, monitoring, and maintaining each distributed storage node, dynamically storing communication data in each node, allocating read and write requests from the user interaction and management module, and detecting and handling node failures; A data encryption and security module, connected to the distributed storage management module, the data backup and disaster recovery module, and the user interaction and management module, for performing fine-grained permission management on users and administrators accessing the system and adopting a multi-factor authentication mechanism, and for securely managing stored communication data and backup data; An automated operation and maintenance and monitoring module, connected to the distributed storage management module, the green computing and energy management module, and the user interaction and management module, for performing automated operation and maintenance and monitoring based on the status of each node, generating system performance optimization results, system logs, and audit reports; A green computing and energy management module, connected to the distributed storage management module, the automated operation and maintenance and monitoring module, and the user interaction and management module, for adjusting the resource configuration of the system according to the load information of each node and the system performance optimization results to obtain an energy-saving measure report; A data backup and disaster recovery module, connected to the distributed storage management module, the data encryption and security module, and the user interaction and management module, for regularly backing up stored communication data, generating backup data, and storing the backup data redundantly at a remote location, and for performing disaster recovery on system failures according to a pre-defined emergency plan; A user interaction and management module, connected to the distributed storage management module, the data encryption and security module, the automated operation and maintenance and monitoring module, the green computing and energy management module, and the data backup and disaster recovery module, for providing a friendly operation interface for users and administrators accessing the system.

2. The system according to claim 1, characterized in that, The system further includes: An interoperability and standardization module, connected to the data encryption and security module and the user interaction and management module, for checking whether the system communication process complies with security transmission standards and for providing an open API interface; A compliance and legal compliance module, connected to the automated operation and maintenance and monitoring module and the data backup and disaster recovery module, for performing compliance reviews on system logs, audit reports, and the communication data backup process.

3. The system according to claim 1, characterized in that, The data encryption and security module is specifically used for: Defining different roles for users and administrators accessing the system according to the organizational structure and business processes, setting corresponding operation permission lists for each role, authenticating the identities of users and administrators using a number of verification means, and end-to-end encrypting the stored communication data and backup data using corresponding encryption algorithms; Wherein, the operation permission list is dynamically adjusted according to different condition combinations; The corresponding encryption algorithms include symmetric encryption, asymmetric encryption, and hybrid encryption.

4. The system according to claim 1, wherein The user interaction and management module is specifically used for: Visually display the status, load information, system performance optimization results, system logs, audit reports, energy-saving measures reports, and system disaster recovery status of each node accordingly.

5. A communication data storage method based on a distributed architecture and intelligent management, characterized in that Including: Create, configure, monitor, and maintain each distributed storage node through the distributed storage management module, dynamically store communication data in each node, allocate read and write requests from the user interaction and management module, and detect and handle node failures; Perform fine-grained permission management on users and administrators accessing the system through the data encryption and security module and adopt a multi-factor authentication mechanism, and perform security management on the stored communication data and backup data; Automatically perform operation and maintenance and monitoring based on the status of each node through the automated operation and maintenance and monitoring module, and generate system performance optimization results, system logs, and audit reports; Adjust the resource configuration of the system according to the load information of each node and the system performance optimization results through the green computing and energy management module to obtain an energy-saving measures report; Regularly back up the stored communication data through the data backup and disaster recovery module to generate backup data, perform off-site redundant storage on the backup data, and perform disaster recovery on system failures according to a pre-established emergency plan; Provide a friendly operation interface for users and administrators accessing the system through the user interaction and management module.

6. The method according to claim 5, characterized in that, The method further includes: Check whether the security transmission standard is followed during the system communication process through the interoperability and standardization module, and provide an open API interface; Conduct a compliance review on system logs, audit reports, and the communication data backup process through the compliance and legal compliance module.

7. The method according to claim 5, wherein Performing fine-grained permission management on users and administrators accessing the system through the data encryption and security module and adopting a multi-factor authentication mechanism, and performing security management on the stored communication data and backup data specifically includes: Define different roles for users and administrators accessing the system according to the organizational structure and business processes through the data encryption and security module, set corresponding operation permission lists for each role, authenticate the identities of users and administrators using several verification means, and perform end-to-end encryption on the stored communication data and backup data using corresponding encryption algorithms; Among them, the operation permission list is dynamically adjusted according to different condition combinations; The corresponding encryption algorithms include symmetric encryption, asymmetric encryption, and hybrid encryption.

8. The method according to claim 5, characterized in that, Providing a friendly operation interface for users and administrators accessing the system through the user interaction and management module specifically includes: Visually display the status, load information, system performance optimization results, system logs, audit reports, energy-saving measures reports, and system disaster recovery status of each node accordingly through the user interaction and management module.

9. An electronic device, characterized in that, Including: A memory, a processor, and a computer program stored on the memory and executable on the processor, where when the computer program is executed by the processor, it implements the steps of the communication data storage method based on a distributed architecture and intelligent management as described in any one of claims 5-8.

10. A computer-readable storage medium, characterized in that, An implementation program for information transmission is stored on the computer-readable storage medium. When the program is executed by a processor, the steps of the communication data storage method based on a distributed architecture and intelligent management as described in any one of claims 5-8 are implemented.

Images