Data management method, system, equipment and medium for integrated data medium

Abstract

The invention discloses a data management method, system, equipment and medium for an integrated data medium, and relates to the technical field of data management, and the method comprises the following steps: starting a data import task, obtaining enterprise resources and customer relationship data according to a set mode and parameters, and storing the data in a distributed file system HDFS in a block manner in a directory and file form, processing the required data by adopting a parallel processing mode to generate a final result; and learning a data correlation mode of the enterprise resource and customer relationship from the final result, and obtaining an association relationship between the enterprise resource and customer relationship data. According to the method, the data in the distributed file system is read and processed in a parallel processing mode, the final result is generated through aggregation of the processed key value pairs, high data processing efficiency is achieved through parallel processing, and the accuracy and efficiency of data retrieval by a user are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a data management method, system, device and medium for an integrated data middle platform. Background Art

[0002] With the rapid development of corporate business and the advancement of digital transformation, data has become a key factor in corporate decision-making and operations.

[0003] Typically, enterprises use a data management middle platform to manage and operate their internal data. The data management middle platform collects, models, analyzes, and applies multi-source heterogeneous data inside and outside the enterprise, thereby achieving management of enterprise data.

[0004] In traditional data management, the data center independently stores and processes data, and there is a lack of correlation between data, which leads to data silos between different businesses during data management, difficulty in data sharing, and low data processing efficiency. When faced with massive amounts of data, traditional methods are unable to cope with the real-time requirements. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a data management method, system, device and medium for an integrated data middle platform, so as to solve the problems of lack of correlation between data in the prior art, resulting in data islands between different businesses during data management, difficulty in data sharing, low data processing efficiency, and the inability of traditional methods to meet real-time requirements when faced with massive amounts of data.

[0006] The present invention specifically provides the following technical solutions: A data management method for an integrated data center comprises the following steps: Initiate a data import task in the data center, obtain data on enterprise resources and customer relationships according to the set method and parameters, and store the data in the form of directories and files in blocks in the distributed file system HDFS; The required data is processed in parallel, including: in the Map phase, the data blocks in HDFS are read in the form of directories and files, the records in each data block are cleaned and key-value pairs are processed, and the processed key-value pairs are sent to the Reduce phase; in the Reduce phase, the key-value pairs from the Map phase are received, and the key-value pairs are aggregated to generate the final result; The data correlation patterns of enterprise resources and customer relationships are learned from the final results to obtain the correlation relationship between enterprise resources and customer relationship data.

[0007] Preferably, before the required data is processed in a parallel processing manner, the following steps are included: Convert the data formats of enterprise resources and customer relationship data from different sources, and fill in missing enterprise resources and customer relationship data by deletion or interpolation to obtain preprocessed data; Checking the preprocessed data for duplicate data and outliers, deleting duplicates and processing outliers to obtain cleaned data; the outliers include extreme values ​​and data that do not conform to business rules; The distributed file system HDFS is used to store clean data, and the programming model MapReduce is used to divide the data processing task into two stages, Map and Reduce, to process the data in parallel.

[0008] Preferably, the parallel processing method further includes: Use Spark's Resilient Distributed Dataset (RDD) or DataFrame / Dataset API to read data directly from HDFS, process the data in parallel in memory, and write the processed data to HDFS.

[0009] Preferably, after generating the final result, the method further includes: Extract data from the final result, send the extracted data to the target location, and perform secondary cleaning on the data; Group and aggregate the data after secondary cleaning, and sort the grouping and aggregation results; Select the sorting results to obtain the required columns, and write the required column data back to HDFS to obtain the associated data.

[0010] Preferably, learning the correlation pattern between enterprise resources and customer relationship data from the final result to obtain the correlation relationship between enterprise resources and customer relationship data includes: Using a machine learning model or a deep learning model to learn features from the final results; the features represent key information or patterns in the enterprise resources and customer relationship data; The features are used to identify patterns in enterprise resources and customer relationship data, and obtain the associations between enterprise resources and customer relationship data; the patterns are regularity, periodicity, trend or anomaly in enterprise resources and customer relationship data, and the associations are correlation, causality or dependency between different enterprise resources and customer relationship data indicators.

[0011] Preferably, after obtaining the association relationship between the enterprise resources and the customer relationship data, the method further includes: Input the association relationship into the application, monitor the data source status and availability including the association relationship through the application, and record the transmission log and error log of the data source in real time; When users search, the execution status and results of data processing tasks are monitored, and when data processing tasks are abnormal, an alarm is issued and relevant personnel are notified to handle the problem; Set data quality standards and detection mechanisms to regularly or in real time check whether data quality meets the requirements, and when there are problems with data quality, issue an alarm and notify relevant personnel to handle it.

[0012] Preferably, when the user searches, the user request is also distributed to multiple servers connected to the data center, and when distributing, optimization selection is performed based on factors such as server load, geographical location, and user request type.

[0013] The present invention provides a data management system of an integrated data center, comprising: The data acquisition module is used to start the data import task in the data center, obtain the data of enterprise resources and customer relationships according to the set method and parameters, and store the data in the form of directories and files in blocks in the distributed file system HDFS; The parallel processing module is used to process the required data in a parallel processing manner, including: in the Map phase, reading the data blocks in the HDFS in the form of directories and files, performing cleaning and key-value pair processing on the records in each data block, and sending the processed key-value pairs to the Reduce phase; in the Reduce phase, receiving the key-value pairs from the Map phase, aggregating them according to the key-value pairs, and generating the final results; The application module is used to learn the data correlation pattern of enterprise resources and customer relationships from the final result, and obtain the correlation relationship between enterprise resources and customer relationship data.

[0014] The present invention provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the data management method of the above-mentioned integrated data center.

[0015] The present invention provides a storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the data management method of the above-mentioned integrated data center are implemented.

[0016] Compared with the prior art, the present invention has the following significant advantages: The present invention obtains relevant data through data import tasks and setting parameters, and stores the relevant data in a distributed file system. The distributed storage of relevant data facilitates data sharing during subsequent parallel processing. In order to improve data management efficiency, a parallel processing method is adopted to read and process the data in the distributed file system, and the final result is generated by aggregating the processed key-value pairs, thereby realizing decentralized storage and sharing of data through distributed files, realizing classified storage of data from different channels, solving the problem of data islands, and obtaining higher data processing efficiency through parallel processing of data in the distributed file system, thereby achieving comprehensive management and sharing of massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart is provided for a data management method of an integrated data center of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] The data center used in the present invention usually adopts a layered architecture, and each layer of application is built in a microservice manner. This architecture can support flexible expansion and customization to meet the needs of different industries and scenarios.

[0020] The main levels of the data center are: Customer business systems: such as enterprise resource planning systems, customer relationship management systems, and various types of IoT data collection devices, are the sources of data.

[0021] Data transmission network: responsible for uploading data from customer business systems to the data storage system. Data transmission can be carried out through public cloud, private cloud, hybrid cloud, and self-built proprietary cloud.

[0022] Unified data collection platform: supports offline data migration, real-time data migration, and on-demand reporting to ensure data integrity and accuracy.

[0023] Data storage layer: Use a reasonable technical architecture to store different types of data, such as relational databases, non-relational databases NoSQL (Not Only SQL) or data lakes.

[0024] Data processing layer: includes multiple steps such as data cleaning, conversion, aggregation, and analysis to ensure the quality and accuracy of data. Data service layer: provides data services to upper-layer applications through application programming interfaces (APIs) or software development kits (SDKs) to meet business needs. The data center consists of multiple decoupled subsystems, each of which has specific functions. These subsystems include: Data storage framework: responsible for data storage and management, ensuring data security and availability. Data collection framework: responsible for collecting data from various data sources, ensuring data integrity and accuracy. Data processing framework: responsible for data cleaning, conversion, aggregation and analysis, improving data quality and value. Data governance framework: responsible for data normalization and standardization management, ensuring data consistency and accuracy. Data security framework: responsible for data security, including data encryption, access control and other security measures. Data operation framework: responsible for data monitoring, scheduling and optimization and other operational work, ensuring efficient use of data and stable operation.

[0025] Based on the above layered architecture, the present invention proposes a data management method for an integrated data center. Figure 1 As shown, the specific steps include: Step S1: Start the data import task in the data center, obtain the data of enterprise resources and customer relationships according to the set method and parameters, and store the data in the form of directories and files in blocks in the distributed file system HDFS.

[0026] Use real-time, batch and streaming access methods to access data sources including relational databases, non-relational databases and API interfaces. Introduce efficient data cleaning and verification mechanisms to ensure data quality and improve access efficiency by at least 30%. This is the data access layer. Parameters include source data format and target table structure.

[0027] Step S2: Process the required data in a parallel processing manner, including: in the Map phase, read the data blocks in HDFS in the form of directories and files, perform cleaning and key-value pair processing on the records in each data block, and send the processed key-value pairs to the Reduce phase; in the Reduce phase, receive the key-value pairs from the Map phase, aggregate them according to the key-value pairs, and generate the final result.

[0028] Use distributed computing frameworks (such as Hadoop, Spark) and advanced data processing algorithms to achieve efficient data processing. Support complex processing processes such as data cleaning, conversion, integration, and modeling. The processing speed is at least 50% higher than traditional methods, and can process massive amounts of data at the PB level.

[0029] Collect the raw data that needs to be processed from multiple sources (such as databases, spreadsheets, text files, or the Web). The raw data here is enterprise resources and customer relationship data.

[0030] Before using parallel processing to process the required data, including: The data formats of enterprise resources and customer relationship data from different sources are converted, and missing enterprise resources and customer relationship data are filled by deletion or interpolation to obtain preprocessed data.

[0031] Check the preprocessed data for duplicate data and outliers, delete duplicates and process outliers to obtain cleaned data; outliers include extreme values ​​and data that do not comply with business rules, and duplicates are completely identical records in the check data, unify data standards, and improve data consistency and comparability.

[0032] The distributed system infrastructure Hadoop distributed file system HDFS is used to store and clean data, and the programming model MapReduce is used to divide the data processing task into two stages, Map and Reduce, to process the data in parallel. Specifically: 1. Use HDFS to store data: Hadoop's distributed file system (HDFS) is ideal for storing large-scale data sets because it is highly fault-tolerant and scalable.

[0033] Data upload: First, upload the original data to HDFS. You can use Hadoop's command line tool hdfs dfs-put to upload data.

[0034] Data organization: In HDFS, data is usually organized in the form of directories and files. Make sure the data is organized in a way that is suitable for subsequent processing.

[0035] 2. Data cleaning: Data cleaning usually includes operations such as removing duplicates, processing missing values, and converting data types. Although MapReduce can be used for data cleaning, Spark is usually more suitable for such tasks due to its in-memory computing characteristics.

[0036] Use Spark function Spark for cleaning: You can use Spark's data frame DataFrame or Dataset API to load data in HDFS, and then use Spark SQL or DataFrame conversion operations (such as filter(), drop(), na.fill(), withColumn(), cast(), etc.) to clean the data.

[0037] 3. Use MapReduce for parallel processing: Map phase: In this phase, the Map function reads the data blocks (splits) in HDFS, performs cleaning and key-value pair processing on the records in each data block, and the processing results (key-value pairs) are sent to the Reduce phase.

[0038] Reduce phase: The Reduce function receives the key-value pairs from the Map phase, aggregates or further processes them, and generates the final result.

[0039] The required data is processed in parallel, and also includes: Use Spark's Resilient Distributed Dataset (RDD) or DataFrame / Dataset API to read data directly from HDFS, process the data in parallel in memory, and write the processed data to HDFS. Specifically: 4. Use Spark’s RDD / DataFrame / Dataset API for parallel processing: Spark provides more advanced APIs than MapReduce, such as RDD (Resilient Distributed Dataset), DataFrame, and Dataset, all of which can process data in memory, greatly improving the processing speed.

[0040] RDD: RDD is the most basic data structure in Spark. It is an immutable collection of elements that can be operated on in parallel.

[0041] DataFrame: DataFrame is an advanced encapsulation of RDD, which provides SQL-like query functions and supports more optimized execution plans.

[0042] Dataset: Dataset was introduced in Spark 1.6, which combines the flexibility of RDD and the optimized execution engine of DataFrame. The Dataset API provides a type-safe way to operate on data.

[0043] 5. Integrate Hadoop and Spark: Data reading: Spark can read data directly from HDFS without first transferring the data to other storage systems.

[0044] Data writing: The processed data can be written back to HDFS for subsequent analysis or storage.

[0045] 6.Performance optimization: Resource allocation: Properly configure the resources of Hadoop and Spark clusters (such as CPU, memory, and disk space).

[0046] Data partitioning: Optimize data partitioning strategies to improve the efficiency of parallel processing.

[0047] Cache: Cache frequently accessed datasets in memory to reduce disk I / O operations. D or DataFrame / Dataset API to process data directly in memory.

[0048] Data conversion (using Spark SQL or Spark DataFrame API to perform data conversion operations to improve conversion efficiency).

[0049] After generating the final result, it also includes: Extract data from multiple data sources (final results), send the extracted data to the target location, and perform secondary cleaning on the data. Perform association processing on the secondary cleansed data according to the new data organization logic. Association processing includes: Use aggregation functions to group and aggregate the data after secondary cleaning, and use sorting functions to sort the grouping and aggregation results.

[0050] Use the selection function to select the sorting results to obtain the required columns, and write the required column data back to HDFS to obtain the associated data.

[0051] In Spark, you can easily associate data using the DataFrame or Dataset API; specifically: Data processing after association: Data Aggregation: Use groupBy() and agg() functions to group and aggregate data.

[0052] Data sorting: Use the orderBy() function to sort the results.

[0053] Data selection: Use the select() function to select the required columns.

[0054] Data writing: Write the processed data back to HDFS or other storage systems.

[0055] Performance is an important consideration when doing association processing. Here are some ways to optimize association performance: Ensure the affinity key is partitioned: If the dataset is large, consider partitioning the data based on the affinity key to reduce the amount of data in the shuffle operation.

[0056] Use broadcast variables: If one of the datasets is small, consider broadcasting it to all nodes to reduce network transmission and memory consumption.

[0057] Adjust the degree of parallelism: Adjust the degree of parallelism of the Spark job based on the cluster resources and data size to make full use of the cluster resources.

[0058] The association between analysis results is achieved through the association logic in the analysis model.

[0059] Business logic: Associate analysis results based on business rules and business relationships between data. For example, the association of sales data with inventory data can be achieved through business logic to analyze the impact of inventory changes on sales.

[0060] Statistical logic: Use statistical methods to correlate analysis results. For example, through statistical methods such as correlation analysis and regression analysis, find out the correlation and causal relationship between different data indicators.

[0061] Machine learning logic: In machine learning models, algorithms automatically discover patterns of associations between data. These patterns may appear as correlations between features, clustering results, or parameter relationships in a predictive model.

[0062] Visualization: To facilitate the understanding and application of analysis results, the relevant results are usually presented through visualization. For example, charts, dashboards or reports are used to show the relationship and trend between different data indicators.

[0063] Data Modeling: Identify the problem: Identify the problem to be solved, including its description, background, purpose, and limitations.

[0064] Collect data (already completed in the previous step): Ensure the quality and completeness of the data.

[0065] Build a model: Based on the characteristics of the problem, select an appropriate mathematical model to describe the problem.

[0066] Analytical model: Analyze the established mathematical model, determine the parameters and assumptions of the model, and test and optimize the model.

[0067] Data verification and publication: Data verification: Use data visualization tools, data statistical tools, etc. to verify the converted and modeled data to ensure the accuracy and reliability of the data.

[0068] Data publishing: The processed data is published to the operating environment for use by users or systems.

[0069] Performance optimization and monitoring (using the load balancing and task allocation strategies of the distributed computing framework to ensure maximum utilization of all nodes).

[0070] Solving the model: Use machine learning libraries such as Spark MLlib to solve the model.

[0071] Step S3: Learn the data correlation patterns of enterprise resources and customer relationships from the final results, and obtain the correlation relationship between enterprise resources and customer relationship data.

[0072] Based on advanced analysis algorithms and models such as machine learning and deep learning, it provides a variety of analysis methods, including real-time analysis, offline analysis, predictive analysis, etc., to meet different business needs. The accuracy of analysis results is improved by at least 20%, providing strong support for corporate decision-making. Specifically: Learn the relevant patterns of enterprise resources and customer relationship data from the final results, and obtain the association relationship between enterprise resources and customer relationship data, including: Use machine learning models or deep learning models to learn features from data; features represent key information or patterns in the data.

[0073] Identify patterns in data through features and obtain the correlation between data; patterns are regularity, periodicity, trend or anomaly in data, and correlation is the correlation, causality or dependency between different data indicators.

[0074] Before using a machine learning model or deep learning model to learn features from data, it also includes: Select machine learning models or deep learning models based on business needs and data characteristics, such as classification, clustering, prediction, etc.

[0075] Train machine learning models or deep learning models using labeled data (for supervised learning) or unlabeled data (for unsupervised learning).

[0076] Evaluate the performance of machine learning or deep learning models through cross validation, confusion matrix, precision, and recall metrics, and adjust accordingly.

[0077] In the data management method of the integrated data center, machine learning models or deep learning models are used to identify patterns and associations from data, mainly following the following pattern: Identification of patterns and relationships: Feature learning: During the training process, a machine learning or deep learning model automatically learns feature representations in the data. These features may represent key information or patterns in the data.

[0078] Pattern recognition: The model uses learned feature representations to identify patterns in the data. These patterns may appear as regularities, periodicities, trends, or anomalies in the data.

[0079] Relationship discovery: In addition to pattern recognition, the model can also discover relationships between data. These relationships may be manifested as correlation, causality, or dependency between different data indicators. By discovering relationships, we can better understand the business logic and mechanism behind the data.

[0080] Results evaluation and application: Evaluation: Evaluate the patterns and relationships identified by the model. This usually involves using a test dataset to verify the accuracy and generalization ability of the model.

[0081] Application of results: Apply the evaluation results to actual business scenarios. For example, optimize the recommendation system based on the identified user behavior patterns; formulate marketing strategies based on the association relationship discovery, etc.

[0082] Continuous Optimization: Model optimization: Continuously optimize machine learning or deep learning models based on actual application results and changes in business needs. This may include adjusting model parameters, improving algorithms, or introducing new data sources.

[0083] Data feedback: New data in business practice will be continuously fed back to the data center, forming a closed-loop data management and optimization process. Through data feedback, model performance can be continuously optimized and the accuracy and effectiveness of data analysis can be improved.

[0084] Pattern recognition: Using machine learning models or deep learning models can automatically identify complex patterns and associations from data, helping to discover hidden information and regularities.

[0085] Anomaly Detection: Use unsupervised learning models to detect anomalies and fraud in data, providing important support for business risk control.

[0086] Predictive analysis: Predictive models can predict future trends and results based on historical data, providing support for corporate strategic planning and decision-making.

[0087] Data Visualization and Reporting: Data visualization: Visualize data analysis results through charts, images, etc. to help users understand data and insights more intuitively.

[0088] Generate reports: Generate detailed data analysis reports based on business needs, including data analysis results, insights, and recommendations.

[0089] The application provides a variety of application methods such as API, SDK, and data services. It is equipped with a variety of application scenario templates to lower the user threshold. It supports rapid application development and deployment.

[0090] The unified data management platform manages the entire data center in a unified manner, including data access, processing, analysis, and application monitoring and scheduling. It provides functions such as data quality monitoring, data security monitoring, and system performance monitoring. It ensures the accuracy and reliability of data and improves the stability and performance of the system.

[0091] After obtaining the association between enterprise resources and customer relationship data, it also includes: The association relationship is input into the application, and the status and availability of the data source including the association relationship are monitored through the application, and the transmission log and error log of the data source are recorded in real time.

[0092] When users search, the execution status and results of data processing tasks are monitored, and when abnormalities occur in data processing tasks, alarms are issued and relevant personnel are notified to handle them.

[0093] Set data quality standards and detection mechanisms to regularly or in real time check whether data quality meets the requirements, and when there are problems with data quality, issue an alarm and notify relevant personnel to handle it.

[0094] The data schedule is: Task definition: Define data processing tasks and processes based on business needs and data characteristics. Determine task dependencies and trigger conditions to ensure that tasks can be executed in the correct order and time.

[0095] Task scheduling execution: Use the task scheduling system to schedule and execute tasks according to the defined tasks and processes.

[0096] Monitor the execution status and results of tasks to ensure that tasks are executed correctly and efficiently.

[0097] Monitor the status and results of task scheduling in real time, including key indicators such as task execution time and success rate.

[0098] When task scheduling is abnormal or fails, an alarm will be issued in time and relevant personnel will be notified to handle it.

[0099] It adopts multi-level security protection measures, including data encryption, access control, security audit, etc. It provides a friendly user interface and operation guide to facilitate users to quickly get started and use it. Rich online help and customer service support ensure that users get timely help during use.

[0100] Among them, horizontal expansion: The horizontal expansion of the database mainly relies on technologies such as sharding, master-slave replication and clustering. These methods allow the database to distribute data among multiple servers or nodes, thereby improving the processing power and fault tolerance of the system. For example, sharding can achieve horizontal splitting of data by hashing the user ID and distributing the data to different database instances. Master-slave replication achieves read-write separation by copying data from the master database to the slave database, thereby reducing the burden on the master database and improving read performance.

[0101] Vertical expansion: Vertical expansion mainly involves improving the performance of a single database instance, such as upgrading hardware, optimizing query statements, adjusting database configuration, etc. Although this has little to do with modular design and loose coupling, it is one of the important means to ensure database performance.

[0102] Modular design and loosely coupled system architecture provide good support for horizontal and vertical expansion. By defining clear module boundaries and interfaces, the system can easily add or replace modules to meet different business needs and performance challenges. At the same time, the loosely coupled design reduces the dependencies between modules and reduces the impact of system changes on overall stability.

[0103] The following operations need to be performed in the data management system security policy: Data encryption, including: Transmission encryption: Use SSL / TLS and other protocols to encrypt the data transmission process to ensure that the data is not intercepted or tampered with during transmission.

[0104] Storage encryption: Encrypt data stored on the server, including using symmetric encryption algorithms such as AES or asymmetric encryption algorithms such as RSA to protect the security of data when it is stored at rest.

[0105] Access control, including: Identity authentication: Ensure that the identity of users accessing the data system is authentic and reliable through methods such as username and password, two-factor authentication or multi-factor authentication.

[0106] Permission management: Models such as role-based access control (RBAC) or attribute-based access control (ABAC) assign different data access and operation permissions to different users or user groups.

[0107] Data Isolation: Use database isolation, network isolation or physical isolation to isolate data of different sensitivity levels or data of different businesses to prevent data leakage or abuse.

[0108] Data backup and recovery: Back up data regularly, including full backup and incremental backup, to deal with the risk of data loss or damage. At the same time, establish a data recovery mechanism to ensure rapid recovery in the event of data loss or damage.

[0109] Security Audit and Monitoring: Audit and monitor data access and operation behaviors, record and analyze data access logs, and promptly discover and handle security incidents.

[0110] Compliance Management: Comply with relevant laws, regulations and industry standards, such as GDPR, HIPAA, etc., to ensure that the data processing process meets compliance requirements.

[0111] Permission management: Role definition and assignment: Define clear user roles, such as administrator, data analyst, data scientist, etc., and assign corresponding data access and operation permissions to each role.

[0112] Permission refinement: Refine data access permissions, such as read-only permission, read-write permission, delete permission, etc., to ensure that users can only access and operate the data they need.

[0113] Permission change management: Establish an authority change management process to review and record the addition, deletion, and modification of authority to ensure the legality and traceability of authority changes.

[0114] Permission audit: Regularly audit the allocation and use of permissions to ensure the correctness and legality of permission allocation, and promptly identify and address issues such as permission abuse.

[0115] Permission inheritance and sharing: When necessary, permissions can be inherited or shared between roles or user groups to improve the flexibility and efficiency of permission management.

[0116] At the same time, the realization of this goal involves many aspects. The following is a detailed explanation of these aspects: 1. Dynamic resource scheduling: Dynamic resource scheduling is one of the key technologies to cope with business growth and data volume growth. Based on cloud computing technology, it can dynamically adjust computing resources, storage resources, and network resources according to actual business needs. The implementation of this process usually relies on automated scripts or cloud platform management tools.

[0117] Real-time monitoring: The system collects key performance indicators (KPIs) such as CPU usage, memory usage, network bandwidth, etc. through real-time monitoring tools to understand the current system load.

[0118] Resource assessment: Based on real-time monitoring data, the system evaluates current resource usage and determines whether resources need to be increased or decreased.

[0119] Automatic adjustment: When the system finds that resources are insufficient or excessive, the resource adjustment mechanism is automatically triggered. For example, in a high-concurrency scenario, if the CPU usage continues to rise and approaches the threshold, the system can automatically add more server instances or improve the specifications of existing instances.

[0120] 2. Load balancing technology: Load balancing technology distributes user requests to multiple server instances to achieve balanced processing of requests and avoid overloading a single server.

[0121] When users search, the user requests are distributed to multiple servers connected to the data center, and during the distribution, the optimization selection is made based on factors such as server load, geographical location, and user request type. Specifically: Intelligent distribution: The load balancer intelligently distributes user requests to different server instances based on preset algorithms (such as polling, least number of connections, source address hashing, etc.). These algorithms can be optimized based on factors such as server load, geographic location, and user request type.

[0122] Health check: The load balancer regularly performs health checks on the backend servers to ensure that only healthy servers can receive user requests. This helps to promptly detect and isolate faulty servers to prevent them from affecting the overall system.

[0123] Session persistence: For requests that need to maintain session status (such as requests after a user logs in), the load balancer can use a session persistence mechanism (such as cookie-based session persistence) to ensure that these requests are distributed to the same server for processing.

[0124] 3. Security policy and permission management: In high-concurrency scenarios, security policies and permission management are particularly important. They can prevent common security threats such as distributed denial of service (DDoS) attacks and structured query language (SQL) injections, and ensure that only authorized users can access system resources.

[0125] Firewall and security group: By configuring firewall and security group rules, you can limit the access of external traffic to the system. Only traffic that meets specific rules can be allowed to enter the system.

[0126] Access control list: Set different access rights for different users or user groups to ensure that they can only access system resources within their permission range. This can be achieved through mechanisms such as role-based access control or attribute-based access control.

[0127] Data encryption and transmission security: Sensitive data is encrypted and protected using security protocols such as HTTPS during transmission. This prevents data from being intercepted or tampered with during transmission.

[0128] Security auditing and logging: Audit and log system access and operations to track and investigate security incidents when they occur. This helps to detect and handle potential security threats in a timely manner.

[0129] Based on the above method, the present invention provides a data management system of an integrated data middle platform, including: a data acquisition module, a parallel processing module and an application module.

[0130] Among them, the data acquisition module is used to start the data import task in the data center, obtain the data of enterprise resources and customer relationships according to the set method and parameters, and store the data in the distributed file system HDFS in the form of directories and files; the parallel processing module is used to process the required data in a parallel processing manner, including: in the mapping Map stage, read the data blocks in HDFS in the form of directories and files, perform cleaning and key-value pair processing on the records in each data block, and send the processed key-value pairs to the Reduce stage; in the specification Reduce stage, receive the key-value pairs from the Map stage, aggregate them according to the key-value pairs, and generate the final result; the application module is used to learn the data-related patterns of enterprise resources and customer relationships from the final results, and obtain the association between enterprise resources and customer relationship data.

[0131] The present invention also provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a data management method of an integrated data center.

[0132] According to the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.

[0133] The present invention also provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of a data management method of an integrated data center are implemented.

[0134] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0135] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. For technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A data management method for an integrated data center, characterized in that: include: Initiate a data import task in the data center, obtain data on enterprise resources and customer relationships according to the set method and parameters, and store the data in the form of directories and files in blocks in the distributed file system HDFS; The required data is processed in parallel, including: in the Map phase, the data blocks in HDFS are read in the form of directories and files, the records in each data block are cleaned and key-value pairs are processed, and the processed key-value pairs are sent to the Reduce phase; in the Reduce phase, the key-value pairs from the Map phase are received, and the key-value pairs are aggregated to generate the final result; The data correlation patterns of enterprise resources and customer relationships are learned from the final results to obtain the correlation relationship between enterprise resources and customer relationship data.

2. The data management method of an integrated data center as claimed in claim 1, characterized in that: Before the required data is processed in parallel, the method includes: Convert the data formats of enterprise resources and customer relationship data from different sources, and fill in missing enterprise resources and customer relationship data by deletion or interpolation to obtain preprocessed data; Checking the preprocessed data for duplicate data and outliers, deleting duplicates and processing outliers to obtain cleaned data; the outliers include extreme values ​​and data that do not conform to business rules; The distributed file system HDFS is used to store clean data, and the programming model MapReduce is used to divide the data processing task into two stages, Map and Reduce, to process the data in parallel.

3. The data management method of an integrated data center as claimed in claim 1, characterized in that: The method of processing the required data in a parallel processing manner further includes: Use Spark's Resilient Distributed Dataset (RDD) or DataFrame / Dataset API to read data directly from HDFS, process the data in parallel in memory, and write the processed data to HDFS.

4. The data management method of an integrated data center as claimed in claim 1, characterized in that: After generating the final result, the following steps are also included: Extract data from the final result, send the extracted data to the target location, and perform secondary cleaning on the data; Group and aggregate the data after secondary cleaning, and sort the grouping and aggregation results; Select the sorting results to obtain the required columns, and write the required column data back to HDFS to obtain the associated data.

5. The data management method of an integrated data center as claimed in claim 1, characterized in that: Learning the relevant patterns of enterprise resources and customer relationship data from the final result, and obtaining the association relationship between the enterprise resources and customer relationship data, including: Using a machine learning model or a deep learning model to learn features from the final results; the features represent key information or patterns in the enterprise resources and customer relationship data; The features are used to identify patterns in enterprise resources and customer relationship data, and obtain the associations between enterprise resources and customer relationship data; the patterns are regularity, periodicity, trend or anomaly in enterprise resources and customer relationship data, and the associations are correlation, causality or dependency between different enterprise resources and customer relationship data indicators.

6. The data management method of an integrated data center as claimed in claim 1, characterized in that: It is characterized in that After obtaining the association relationship between the enterprise resources and the customer relationship data, the following steps are also included: Input the association relationship into the application, monitor the data source status and availability including the association relationship through the application, and record the transmission log and error log of the data source in real time; When users search, the execution status and results of data processing tasks are monitored, and when data processing tasks are abnormal, an alarm is issued and relevant personnel are notified to handle the problem; Set data quality standards and detection mechanisms to regularly or in real time check whether data quality meets the requirements, and when there are problems with data quality, issue an alarm and notify relevant personnel to handle it.

7. The data management method of an integrated data center as claimed in claim 6, characterized in that: When the user searches, the user request is also distributed to multiple servers connected to the data center, and when distributing, optimization selection is performed based on factors such as server load, geographical location, and user request type.

8. A data management system for an integrated data center, characterized in that: include: The data acquisition module is used to start the data import task in the data center, obtain the data of enterprise resources and customer relationships according to the set method and parameters, and store the data in the form of directories and files in blocks in the distributed file system HDFS; The parallel processing module is used to process the required data in a parallel processing manner, including: in the Map phase, reading the data blocks in the HDFS in the form of directories and files, performing cleaning and key-value pair processing on the records in each data block, and sending the processed key-value pairs to the Reduce phase; in the Reduce phase, receiving the key-value pairs from the Map phase, aggregating them according to the key-value pairs, and generating the final results; The application module is used to learn the data correlation pattern of enterprise resources and customer relationships from the final result, and obtain the correlation relationship between enterprise resources and customer relationship data.

9. A computer device, characterized in that: It includes a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a data management method of an integrated data center as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a data management method of an integrated data center as described in any one of claims 1 to 7 are implemented.

Images