Data management system and method and server

Abstract

The invention provides a data management system, a data management method and a server, and relates to the technical field of data management.In the process that the data management system carries out data management on different service systems, various levels and units contained in the system are utilized, and the data management efficiency is improved. According to the invention, a simple and effective mode can be utilized to break through data barriers of business systems in an enterprise, unified specification of the data is realized, ambiguity and repeatability of the data are avoided, and decision-making efficiency and operation efficiency of the enterprise are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data management system, method and server. Background Art

[0002] In the era of big data, enterprises will involve many types of business systems, each of which corresponds to specific data. Due to the data barriers between different business systems and the data walls in each department of the business system, the data between different departments and different business systems are ambiguous and repetitive, which makes it difficult for enterprises to achieve comprehensive data management, reducing the decision-making efficiency and operational effectiveness of enterprises. Although the data management methods in the existing technology can solve this problem to a certain extent, there are problems such as complex data management process and difficult data synchronization management. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a data management system, method and server. The system, through various built-in levels and units, can use a simple and effective way to break through the data barriers of various business systems within the enterprise, achieve unified standardization of data, avoid ambiguity and duplication of data, and improve the decision-making efficiency and operational efficiency of the enterprise.

[0004] In a first aspect, an embodiment of the present invention provides a data management system, the system comprising:

[0005] The data source layer is used to obtain the data source and its corresponding data type parameters, data flow parameters and data volume;

[0006] The data processing layer is used to determine the data processing strategy based on the data type parameter, data flow parameter and data volume, and use the data processing strategy to process the data source data request to obtain the output data of the data source layer;

[0007] The data storage layer is used to save the output data to the preset data warehouse and build a data interaction channel based on the data architecture parameters of the data warehouse;

[0008] The data service layer is used to construct data query services, data analysis services, and data report generation services corresponding to the data source according to the first interaction end of the data interaction channel;

[0009] The application layer is used to construct access services corresponding to the data query service, data analysis service, and data report generation service according to the second interaction end of the data interaction channel;

[0010] The data security layer is used to determine the control strategy, encryption strategy and desensitization strategy between the first interaction terminal and the second interaction terminal in the data interaction channel, and use the control strategy, encryption strategy and desensitization strategy to process the interaction data between the first interaction terminal and the second interaction terminal;

[0011] The monitoring and optimization layer is used to monitor the operating status of interactive data in the data source layer, data processing layer, data storage layer, data service layer, application layer and data security layer.

[0012] Optionally, the data source layer includes:

[0013] A log management unit, used to obtain a first data source corresponding to the log management process and a first data type parameter, a first data flow direction parameter and a first data volume corresponding to the log management process;

[0014] A department management unit, used for acquiring a second data source corresponding to the department management and a second data type parameter, a second data flow direction parameter, and a second data volume corresponding to the department management;

[0015] A user management unit, used for obtaining a third data source corresponding to user management and its corresponding third data type parameter, third data flow direction parameter and third data volume;

[0016] A role management unit, used for acquiring a fourth data source corresponding to the role management and a fourth data type parameter, a fourth data flow direction parameter and a fourth data volume corresponding to the role management;

[0017] The module management unit is used to obtain the fifth data source corresponding to the module management and its corresponding fifth data type parameter, fifth data flow direction parameter and fifth data volume.

[0018] Optional, data processing layer, including:

[0019] A data extraction unit is used to determine the corresponding data source according to the data type parameter and the data flow direction parameter, determine the data processing strategy corresponding to the data source according to the data volume, and obtain the extracted data of the data source after processing the data request of the data source according to the data processing strategy;

[0020] A data cleaning unit, used to determine a data cleaning rule based on a format parameter corresponding to the extracted data, and to process the extracted data using the data cleaning rule to obtain cleaned data corresponding to the extracted data;

[0021] The data loading unit is used to determine a data conversion rule based on the conversion parameters corresponding to the cleaned data, and perform data conversion on the cleaned data using the data conversion rule to obtain output data.

[0022] Optionally, the data storage layer includes:

[0023] An operation data storage unit is used to save the output data into a data warehouse, and to construct a first data interaction channel corresponding to the data query based on the first data architecture parameter corresponding to the query requirement of the output data;

[0024] The data warehouse unit is used to obtain the second data architecture parameters corresponding to the output data in the operation data storage unit for summarization, aggregation and analysis, and determine the second data interaction channel corresponding to the data summarization, data aggregation and data analysis of the output data based on the second data architecture parameters;

[0025] The data mart unit is used to obtain the third data architecture parameters corresponding to the visualization processing of the output data in the operation data storage unit, and determine the third data interaction channel corresponding to the visualization processing of the output data based on the third data architecture parameters.

[0026] Optionally, the data service layer includes:

[0027] A data query service unit, used to construct a data query service according to a data query service interface corresponding to the application layer and based on a data query interaction terminal corresponding to the application layer in a data interaction channel;

[0028] A data analysis service unit, used to build a data analysis service based on the data analysis service interface corresponding to the application layer and on the data analysis interaction end corresponding to the application layer in the data interaction channel;

[0029] The data report generation service unit is used to generate a data report service interface corresponding to the application layer and to build a data report generation service based on the data report generation interaction terminal corresponding to the application layer in the data interaction channel.

[0030] Optional, application layer, including:

[0031] A data query access unit, configured to construct a first access service corresponding to the data query service based on a data query access interface corresponding to the data query service unit and on a data query interaction end corresponding to the data query service in a data interaction channel;

[0032] A data analysis access unit, configured to construct a second access service corresponding to the data analysis service according to a data analysis access interface corresponding to the data analysis service unit and based on a data analysis interaction terminal corresponding to the data analysis service in a data interaction channel;

[0033] The data report generation access unit is used to construct a third access service corresponding to the data report generation service according to the data report access interface corresponding to the data report generation service unit and based on the data report interaction terminal corresponding to the data report generation service in the data interaction channel.

[0034] Optional, data security layer, including:

[0035] An access control unit, used to generate a control strategy according to the role access control parameters between the first interaction terminal and the second interaction terminal, and perform access interaction processing on the role access control interaction data between the first interaction terminal and the second interaction terminal using the control strategy;

[0036] An encryption unit, used to generate an encryption strategy according to the encryption control parameters between the first interaction terminal and the second interaction terminal, and use the encryption strategy to encrypt the encrypted interaction data between the first interaction terminal and the second interaction terminal;

[0037] The desensitization unit is used to generate a desensitization strategy according to the desensitization control parameters between the first interaction terminal and the second interaction terminal, and use the desensitization strategy to perform desensitization processing on the desensitized interaction data between the first interaction terminal and the second interaction terminal.

[0038] Optional, monitoring and optimization layer, including:

[0039] A visualization unit is used to obtain operation status parameters corresponding to interactive data in the data source layer, data processing layer, data storage layer, data service layer, application layer and data security layer by using a service monitoring tool, and monitor the interactive data based on the operation status parameters;

[0040] The log analysis unit is used to obtain the log corresponding to the interaction data, generate analysis parameters corresponding to the log based on the operation status parameters, and analyze and process the log using the analysis parameters.

[0041] In a second aspect, an embodiment of the present invention provides a data management method, which is applied to the data management system mentioned in the first aspect, and the method includes:

[0042] Use the data source layer to obtain the data source and its corresponding data type parameters, data flow parameters and data volume;

[0043] Using the data processing layer, determine the data processing strategy based on the data type parameter, data flow parameter and data volume, and use the data processing strategy to process the data source data request to obtain the output data of the data source layer;

[0044] Using the data storage layer, the output data is saved in the preset data warehouse, and a data interaction channel is built based on the data architecture parameters of the data warehouse;

[0045] Using the data service layer, build data query services, data analysis services, and data report generation services corresponding to the data source according to the first interactive end of the data interactive channel;

[0046] Using the application layer, building access services corresponding to the data query service, data analysis service, and data report generation service according to the second interaction end of the data interaction channel;

[0047] Using the data security layer, determine the control strategy, encryption strategy and desensitization strategy between the first interaction terminal and the second interaction terminal in the data interaction channel, and use the control strategy, encryption strategy and desensitization strategy to process the interaction data between the first interaction terminal and the second interaction terminal;

[0048] The monitoring and optimization layer is used to monitor the operating status of interactive data in the data source layer, data processing layer, data storage layer, data service layer, application layer and data security layer.

[0049] In a third aspect, an embodiment of the present invention further provides a server, including a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the data management method provided in the second aspect.

[0050] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to implement the steps of the data management method provided in the second aspect.

[0051] The embodiments of the present invention provide a data management system, method and server, wherein the data management system includes: a data source layer, a data processing layer, a data storage layer, a data service layer, an application layer, a data security layer, and a monitoring and optimization layer. Among them, the data source layer is used to obtain the data source and its corresponding data type parameters, data flow parameters and data volume; the data processing layer is used to determine the data processing strategy based on the data type parameters, data flow parameters and data volume, and use the data processing strategy to process the data source data request to obtain the output data of the data source layer; the data storage layer is used to save the output data to the preset data warehouse, and build a data interaction channel based on the data architecture parameters of the data warehouse; the data service layer is used to build the data query service, data analysis service and data report generation service corresponding to the data source according to the first interaction end of the data interaction channel; the application layer is used to build the access service corresponding to the data query service, data analysis service and data report generation service according to the second interaction end of the data interaction channel; the data security layer is used to determine the control strategy, encryption strategy and desensitization strategy between the first interaction end and the second interaction end in the data interaction channel, and use the control strategy, encryption strategy and desensitization strategy to process the interaction data between the first interaction end and the second interaction end; the monitoring and optimization layer is used to monitor the running status of the interaction data in the data source layer, data processing layer, data storage layer, data service layer, application layer and data security layer. In the process of data management for different business systems, by utilizing the various levels and units contained in the system, the data barriers of various business systems within the enterprise can be broken down in a simple and effective way, achieving unified data standardization, avoiding data ambiguity and duplication, and improving the enterprise's decision-making and operational efficiency.

[0052] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1A schematic diagram of the structure of a data management system provided by an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of the structure of a data source layer in a data management system provided by an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the structure of a data processing layer in a data management system provided by an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of the structure of a data storage layer in a data management system provided by an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the structure of a data service layer in a data management system provided by an embodiment of the present invention;

[0060] Figure 6 A schematic diagram of the structure of an application layer in a data management system provided by an embodiment of the present invention;

[0061] Figure 7 A schematic diagram of the structure of a data security layer in a data management system provided by an embodiment of the present invention;

[0062] Figure 8 A schematic diagram of the structure of a monitoring and optimization layer in a data management system provided by an embodiment of the present invention;

[0063] Fig. 9 A schematic diagram of the structure of another data management system provided by an embodiment of the present invention;

[0064] Fig.10 A flow chart of a data management method provided by an embodiment of the present invention;

[0065] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present invention.

[0066] icon:

[0067] 10-data source layer; 20-data processing layer; 30-data storage layer; 40-data service layer; 50-application layer; 60-data security layer; 70-monitoring and optimization layer;

[0068] 11-log management unit; 12-department management unit; 13-user management unit; 14-role management unit; 15-module management unit;

[0069] 21-data extraction unit; 22-data cleaning unit; 23-data loading unit;

[0070] 31-operation data storage unit; 32-data warehouse unit; 33-data mart unit;

[0071] 41-data query service unit; 42-data analysis service unit; 43-data report generation service unit;

[0072] 51-data query access unit; 52-data analysis access unit; 53-data report generation access unit;

[0073] 61-access control unit; 62-encryption unit; 63-desensitization unit;

[0074] 71- visualization unit; 72- log analysis unit;

[0075] 101 - processor; 102 - memory; 103 - bus; 104 - communication interface. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Enterprises in the era of big data will involve many types of business systems, each of which corresponds to specific data. Due to the data barriers between different business systems and the data wall problem in each department of the business system, there are ambiguity and duplication of data between different departments and different business systems. This makes it difficult for enterprises to achieve comprehensive management of data, reducing the decision-making efficiency and operational effectiveness of enterprises. Although the data management methods in the prior art can solve this problem to a certain extent, there are problems such as complex data management process and difficult data synchronization management. Based on this, the present invention provides a data management system, method and server. Through the various levels and units built into the data management system, it is possible to use a simple and effective method to break through the data barriers of various business systems within the enterprise, achieve unified data standardization, avoid data ambiguity and duplication, and improve the decision-making efficiency and operational efficiency of the enterprise.

[0078] To facilitate understanding of this embodiment, a data management system disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the system includes:

[0079] The data source layer 10 is used to obtain the data source and its corresponding data type parameters, data flow parameters and data volume. The data source layer 10 is specifically connected to the data processing layer 20 and can be implemented using relevant adapters or connectors. The data source layer 10 involves various business systems, log systems, external APIs, etc. as data sources, and obtains the data type parameters, data flow parameters and data volume corresponding to the data source. In actual scenarios, the data source can clarify the data requirements and expected goals based on the needs of the business system, thereby determining the relevant data source, and determining the specific data type, data volume and data flow direction in combination with the data source.

[0080] The data processing layer 20 is used to determine the data processing strategy based on the data type parameter, data flow parameter and data volume, and obtain the output data of the data source layer after processing the data request of the data source using the data processing strategy. The data processing layer is specifically responsible for extracting data from the data source layer 10, processing the data using the generated data processing strategy, and transmitting the obtained output data to the data storage layer 30. Specifically, the data processing layer 20 can realize the processes of extracting, cleaning, converting and loading the data in the data source.

[0081] The data storage layer 30 is used to save the output data in a preset data warehouse and build a data interaction channel based on the data architecture parameters of the data warehouse. The data interaction channel involves a first interaction terminal and a second interaction terminal, the first interaction terminal is set in the data service layer 40, and the second interaction terminal is set in the application layer 50. The data storage layer 30 can save, summarize and analyze the output data, and build a data interaction channel between the data service layer 40 and the application layer 50 by saving the output data in the data warehouse.

[0082] The data service layer 40 is used to construct data query services, data analysis services, and data report generation services corresponding to the data source according to the first interaction end of the data interaction channel. The data service layer 40 mainly provides API interfaces corresponding to the data query, data analysis, and data report generation processes, thereby realizing interaction with the data storage layer 30. Specifically, the data service layer 40 corresponds to the back end of the data management system, and is used to undertake data query services, data analysis services, and data table generation services corresponding to the data source.

[0083] The application layer 50 is used to construct access services corresponding to the data query service, data analysis service, and data report generation service according to the second interactive end of the data interaction channel. The application layer 50 is the front end of the user's corresponding data management system. Through the second interactive end, it constructs access services corresponding to the data query service, data analysis service, and data report generation service in the data service layer 40, so as to provide users with access to the data service layer 40 through the front-end related applications, so as to obtain the corresponding data reports and analysis results.

[0084] The data security layer 60 is used to determine the control strategy, encryption strategy and desensitization strategy between the first interaction terminal and the second interaction terminal in the data interaction channel, and use the control strategy, encryption strategy and desensitization strategy to process the interaction data between the first interaction terminal and the second interaction terminal;

[0085] The data security layer 60 runs through all levels of the data management system. It processes the interactive data in the data interaction channel by obtaining the control strategy, encryption strategy and desensitization strategy between the first interactive end and the second interactive end, thereby providing access control, encryption, desensitization and other security functions during the data access process.

[0086] The monitoring and optimization layer 70 is used to monitor the operation status of the interactive data in the data source layer, data processing layer, data storage layer, data service layer, application layer and data security layer. The monitoring and optimization layer 70 monitors each layer of the entire data management system and monitors the interactive data corresponding to the operation status of each layer from the perspective of performance, health status and security.

[0087] Optionally, the data source layer 10, such as Figure 2 As shown, including:

[0088] The log management unit 11 is used to obtain a first data source corresponding to the log management process and a first data type parameter, a first data flow direction parameter and a first data volume corresponding to the first data source;

[0089] The department management unit 12 is used to obtain a second data source corresponding to the department management and its corresponding second data type parameter, second data flow direction parameter and second data volume;

[0090] A user management unit 13, used for acquiring a third data source corresponding to user management and its corresponding third data type parameter, third data flow direction parameter and third data volume;

[0091] The role management unit 14 is used to obtain a fourth data source corresponding to the role management and its corresponding fourth data type parameter, fourth data flow direction parameter and fourth data volume;

[0092] The module management unit 15 is used to obtain the fifth data source corresponding to the module management and its corresponding fifth data type parameter, fifth data flow direction parameter and fifth data volume.

[0093] The log management unit 11 in the data source layer 10 corresponds to the data source managed by logs, the department management unit 12 corresponds to the data source managed by departments, the user management unit 13 corresponds to the data source managed by users, the role management unit 14 corresponds to the data source managed by roles, and the module management unit 15 corresponds to the data source managed by modules. Different data sources can set data type parameters, data flow parameters, and data volume according to different data standards and naming conventions to ensure data consistency and comparability between different departments. Adapters or connectors can be used between the data source layer 10 and the data processing layer 20 to extract data. Specifically, asynchronous data transmission can be achieved through message queues (such as Kafka) to ensure the reliability and efficiency of data transmission.

[0094] Optionally, the data processing layer 20, such as Figure 3 As shown, including:

[0095] The data extraction unit 21 is used to determine the corresponding data source according to the data type parameter and the data flow direction parameter, determine the data processing strategy corresponding to the data source according to the data volume, and obtain the extracted data of the data source after performing data request processing on the data source according to the data processing strategy;

[0096] A data cleaning unit 22 is used to determine a data cleaning rule based on the format parameters corresponding to the extracted data, and process the extracted data using the data cleaning rule to obtain cleaned data corresponding to the extracted data;

[0097] The data loading unit 23 is used to determine a data conversion rule based on the conversion parameters corresponding to the cleaned data, and perform data conversion on the cleaned data using the data conversion rule to obtain output data.

[0098] Specifically, the data processing layer 20 can be constructed using ETL tools to obtain a data extraction unit 21, a data cleaning unit 22, and a data loading unit 23. Specifically, the data extraction unit 21 is responsible for extracting data from various data sources in real time or in batches. These data may include sales records, customer information, financial statements, etc. The data cleaning unit 22 is used to check and clean the data quality to ensure data accuracy. The data loading unit 23 is used to convert the data into a unified format and perform necessary data processing, such as data mapping, aggregation, etc. From a business perspective, the data extraction unit 21 can ensure the real-time nature of business operations. For example, once sales data is generated in the business system, it is immediately extracted and transmitted to the data warehouse. Before the data arrives at the data warehouse, the data cleaning unit 22 and the data loading unit 23 clean and convert it to meet the storage standards and formats of the data warehouse. At the business level, this means eliminating data redundancy, correcting erroneous data, unifying the metrics of different business systems, and ensuring data consistency and accuracy.

[0099] Optionally, the data storage layer 30, such as Figure 4 As shown, including:

[0100] The operation data storage unit 31 is used to save the output data to the data warehouse, and to construct a first data interaction channel corresponding to the data query based on the first data architecture parameter corresponding to the query requirement of the output data;

[0101] The data warehouse unit 32 is used to obtain the second data architecture parameters corresponding to the output data in the operation data storage unit for summarization, aggregation and analysis, and determine the second data interaction channel corresponding to the data summarization, data aggregation and data analysis of the output data based on the second data architecture parameters;

[0102] The data mart unit 33 is used to obtain the third data architecture parameters corresponding to the visualization processing of the output data in the operation data storage unit, and determine the third data interaction channel corresponding to the visualization processing of the output data based on the third data architecture parameters.

[0103] The operational data storage unit 31 can use a relational database to store near-real-time raw data as the primary level of the data warehouse to store near-real-time raw data. The data in the operational data storage unit 31 is usually not summarized to maintain high data fidelity. At the business level, the operational data storage unit 31 can support business users' query needs for raw data, for example, to conduct detailed transaction queries or generate daily reports. The data warehouse unit 32 can use data warehouse technology (such as Snowflake, Redshift) to summarize and analyze data. The data warehouse unit 32 further processes and integrates the data, including data summarization, aggregation and analysis. At the business level, it supports complex analytical queries, such as monthly sales analysis, customer purchase behavior analysis, etc., to help business departments and management gain in-depth business insights. The data mart unit 33 can establish a data mart for specific business needs and use NoSQL databases or search engines (such as Elasticsearch) to provide fast queries. The data mart unit 33 can provide customized data views and reports for specific business applications or departments, and these data views are optimized to support fast queries and decisions. At the business level, the data mart unit 33 enables users in a specific business field to quickly obtain the required data.

[0104] Optionally, the data service layer 40, such as Figure 5 As shown, including:

[0105] The data query service unit 41 is used to construct a data query service according to the data query service interface corresponding to the application layer and based on the data query interaction terminal corresponding to the application layer in the data interaction channel;

[0106] The data analysis service unit 42 is used to build a data analysis service according to the data analysis service interface corresponding to the application layer and based on the data analysis interaction terminal corresponding to the application layer in the data interaction channel;

[0107] The data report generation service unit 43 is used to build a data report generation service according to the data report generation service interface corresponding to the application layer and based on the data report generation interaction terminal corresponding to the application layer in the data interaction channel.

[0108] The data service layer 40 constructs data query services, data analysis services, and data report generation services through the data query service unit 41, the data analysis service unit 42, and the data report generation service unit 43, so that the data warehouse provides the data in the data warehouse to the front-end application or other business system through the data service layer 40, thereby realizing data sharing and reuse. Figure 6 As shown, including:

[0109] The data query access unit 51 is used to construct a first access service corresponding to the data query service based on the data query access interface corresponding to the data query service unit and the data query interaction end corresponding to the data query service in the data interaction channel;

[0110] The data analysis access unit 52 is used to construct a second access service corresponding to the data analysis service according to the data analysis access interface corresponding to the data analysis service unit and based on the data analysis interaction terminal corresponding to the data analysis service in the data interaction channel;

[0111] The data report generation access unit 53 is used to construct a third access service corresponding to the data report generation service according to the data report access interface corresponding to the data report generation service unit and based on the data report interaction terminal corresponding to the data report generation service in the data interaction channel.

[0112] The application layer 50 is connected to the data query service unit 41, data analysis service unit 42 and data report generation service unit 43 in the data service layer 40 through the data query access unit 51, data analysis access unit 52 and data report generation access unit 53, and specifically provides data services through RESTful API or GraphQL API to realize data query, analysis and report generation. At the business level, the above units can support cross-system business process integration, such as supply chain management, customer relationship management, etc., and improve the automation and intelligence level of business processes.

[0113] Optionally, a data security layer 60, such as Figure 7 As shown, including:

[0114] An access control unit 61, configured to generate a control strategy according to the role access control parameters between the first interaction terminal and the second interaction terminal, and perform access interaction processing on the role access control interaction data between the first interaction terminal and the second interaction terminal using the control strategy;

[0115] The encryption unit 62 is used to generate an encryption policy according to the encryption control parameters between the first interaction terminal and the second interaction terminal, and use the encryption policy to encrypt the encrypted interaction data between the first interaction terminal and the second interaction terminal;

[0116] The desensitization unit 63 is used to generate a desensitization strategy according to the desensitization control parameters between the first interactive terminal and the second interactive terminal, and perform desensitization processing on the desensitized interaction data between the first interactive terminal and the second interactive terminal using the desensitization strategy.

[0117] The data security layer 60 can implement TLS / SSL encrypted transmission through the access control unit 61, encryption unit 62 and desensitization unit 63 to ensure data transmission security, and use RBAC (role-based access control) for fine-grained permission management; in addition, data desensitization and anonymization technology are also applied to protect personal privacy. Through the above units, data security and compliance measures can be implemented at the data warehouse level to ensure data security and compliance with relevant laws and regulations. At the business level, this means encrypting, access controlling and auditing sensitive data to protect customer information and corporate assets.

[0118] Optionally, a monitoring and optimization layer 70, such as Figure 8 As shown, including:

[0119] The visualization unit 71 is used to obtain the operation status parameters corresponding to the interactive data in the data source layer, the data processing layer, the data storage layer, the data service layer, the application layer and the data security layer by using the service monitoring tool, and monitor the interactive data based on the operation status parameters;

[0120] The log analysis unit 72 is used to obtain the log corresponding to the interaction data, generate analysis parameters corresponding to the log based on the operation status parameters, and analyze and process the log using the analysis parameters.

[0121] The monitoring and optimization layer 70 can provide decision makers with intuitive business indicator display and monitoring through built-in visualization units 71 and log analysis units 72 through visualization tools such as cockpits. At the business level, key performance indicators (KPIs) such as sales, inventory levels, customer satisfaction, etc. can be monitored in real time and respond quickly to market changes. Specifically, tools such as Prometheus and Grafana can be used for system monitoring and performance visualization; in addition, through the implementation of log management, ELK Stack (Elasticsearch, Logstash, Kibana) is used for log collection and analysis.

[0122] like Fig. 9 The structural diagram of another data management system shown in the figure, wherein the system management targeted by the data source layer mainly includes: log management, department management, user management, role management, module management; through the data processing layer, data storage layer, data service layer, application layer, data security layer, monitoring and optimization layer, approval management, report management, workbench, application management, announcement management, agency management and cockpit functions can be realized; the workbench also interacts with report relations and application management, and also determines the system list and jump, report list and jump.

[0123] The above data management system can improve data management efficiency, reduce manual intervention and lower the error rate through ETL (Extract-Transform-Load) automated process and data standardization processing;

[0124] Enhance decision-making support capabilities: The construction of data warehouses and data lakes provides a comprehensive data view for decision-making, and data visualization tools intuitively display key indicators;

[0125] Optimize business processes: Single sign-on and business system integration simplify user operations and improve work efficiency.

[0126] By improving data security: Multi-layered data security measures, including access control, data encryption and desensitization, protect data from unauthorized access.

[0127] Promote the comprehensive digital transformation of enterprises: The construction of the data middle platform promotes data integration and analysis, provides enterprises with data-driven insights, and supports business innovation and automation.

[0128] Expand the application scenarios of the data middle platform: The openness of data services allows reuse and expansion in multiple business scenarios, such as supply chain optimization, customer relationship management, etc.

[0129] It can be seen from the data management system described in the above embodiments that in the process of data management of different business systems, by utilizing the various levels and units contained in the data management system, it is possible to break through the data barriers of various business systems within the enterprise in a simple and effective way, achieve unified data standardization, avoid data ambiguity and duplication, and improve the decision-making efficiency and operational efficiency of the enterprise.

[0130] The embodiment of the present invention also provides a data management method, which is applied to the data management system mentioned in the above embodiment, such as Fig.10 As shown, the method includes:

[0131] Step S1001, using the data source layer to obtain the data source and its corresponding data type parameters, data flow parameters and data volume;

[0132] Step S1002, using the data processing layer, determining a data processing strategy based on data type parameters, data flow parameters and data volume, and using the data processing strategy to process data requests on the data source to obtain output data of the data source layer;

[0133] Step S1003, using the data storage layer, saving the output data to a preset data warehouse, and building a data interaction channel based on the data architecture parameters of the data warehouse;

[0134] Step S1004, using the data service layer to construct data query services, data analysis services, and data report generation services corresponding to the data source according to the first interaction end of the data interaction channel;

[0135] Step S1005, using the application layer, constructing access services corresponding to the data query service, the data analysis service, and the data report generation service according to the second interaction end of the data interaction channel;

[0136] Step S1006, using the data security layer, determining a control strategy, an encryption strategy, and a desensitization strategy between the first interaction terminal and the second interaction terminal in the data interaction channel, and using the control strategy, the encryption strategy, and the desensitization strategy to process the interaction data between the first interaction terminal and the second interaction terminal;

[0137] Step S1007, using the monitoring and optimization layer, monitors the operating status of the interactive data in the data source layer, data processing layer, data storage layer, data service layer, application layer and data security layer.

[0138] In actual scenarios, this method can achieve the following process based on the data management system:

[0139] Demand analysis: Communicate with each business department to clarify data needs and expected goals; determine data sources, data types, data volumes, and data flows.

[0140] Data source identification and access: Identify all data sources that need to be integrated; design data extraction strategies, including batch extraction and real-time extraction.

[0141] ETL tool selection and configuration: Select an ETL tool that suits the size and needs of the enterprise; configure the ETL tool and set rules for data extraction, transformation, and loading.

[0142] Data extraction: Write ETL scripts to connect to data sources and extract required data; test the data extraction process to ensure data accuracy and completeness.

[0143] Data cleaning: Identify data quality issues such as duplication, errors, missing data, etc.; develop data cleaning rules and scripts to automate the data cleaning process.

[0144] Data transformation: Design data models to convert data into a format suitable for analysis; apply data transformation logic such as data mapping, aggregation, merging, etc.

[0145] Data loading: Load cleaned and transformed data into the data warehouse; ensure that the data update mechanism is synchronized with the business system.

[0146] Data warehouse architecture design: Design the architecture of the data warehouse, including operational data storage, data warehouse and data mart; determine the data storage plan and select the appropriate database system.

[0147] Building a data management cloud platform: Select a cloud service provider to build a data management cloud platform; configure cloud platform resources such as computing, storage, and network.

[0148] Data asset management: Implement metadata management to record the structure, source, and conversion history of data; design data catalogs to facilitate users to discover and understand data.

[0149] Data security strategy formulation: formulate data access control strategies, implement user identity authentication and authorization; implement data encryption and desensitization to protect sensitive data.

[0150] Data service interface development: develop RESTful API or GraphQL API to provide data services; design authentication and authorization mechanisms for data services.

[0151] Data intelligence service system development: Integrate machine learning platforms, develop predictive models and analysis tools; train models, and deploy intelligent service systems.

[0152] Business report and dashboard development: determine report requirements and design report templates; develop report generation tools and integrate them into the data center.

[0153] Testing and quality assurance: Perform unit testing, integration testing, and performance testing on each component of the data center to ensure that the stability, reliability, and performance of the data center meet the requirements.

[0154] Deployment and launch: Verify the operation of the data center in the test environment; deploy the data center to the production environment and launch it online.

[0155] Monitoring and optimization: Establish a monitoring system to monitor the performance and status of the data center in real time; optimize the data center based on monitoring results and user feedback.

[0156] User training and documentation: Train users on the use of the data center; write user manuals and developer documentation.

[0157] Maintenance and support: Provide technical support to solve problems encountered by users when using the data center; regularly update the data center, fix defects, and add new features.

[0158] Feedback loop: Establish a feedback mechanism to collect user opinions and suggestions; continuously improve the functions and performance of the data center based on feedback.

[0159] It can be seen from the data management method mentioned in the above embodiment that in the process of data management of different business systems, this method can break through the data barriers of various business systems within the enterprise in a simple and effective way by utilizing various levels and units contained in the data management system, thereby achieving unified standardization of data, avoiding ambiguity and duplication of data, and improving the decision-making efficiency and operational efficiency of the enterprise.

[0160] The data management method provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned data management system embodiment. For the sake of brief description, for parts not mentioned in the method embodiment, reference may be made to the corresponding contents in the aforementioned data management system embodiment.

[0161] This embodiment also provides a server, the structural diagram of which is as follows: Fig.11 As shown, the device includes a processor 101 and a memory 102; wherein the memory 102 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the steps of the above-mentioned data management method.

[0162] Fig.11 The server shown further includes a bus 103 and a communication interface 104 , and the processor 101 , the communication interface 104 and the memory 102 are connected via the bus 103 .

[0163] The memory 102 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The bus 103 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0164] The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and send the encapsulated IPv4 message or IPv4 message to the user terminal through the network interface.

[0165] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 102, and the processor 101 reads the information in the memory 102 and completes the steps of the method of the above embodiment in combination with its hardware.

[0166] An embodiment of the present invention further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data management method in the above embodiment are executed.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0170] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0171] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data management system, characterized in that: The system comprises: The data source layer is used to obtain the data source and its corresponding data type parameters, data flow parameters and data volume; A data processing layer, used to determine a data processing strategy based on the data type parameter, the data flow direction parameter and the data volume, and obtain output data of the data source layer after processing the data request of the data source using the data processing strategy; A data storage layer, used to save the output data to a preset data warehouse and build a data interaction channel based on the data architecture parameters of the data warehouse; A data service layer, used to construct a data query service, a data analysis service, and a data report generation service corresponding to the data source according to the first interaction end of the data interaction channel; An application layer, used to construct access services corresponding to the data query service, the data analysis service, and the data report generation service according to the second interaction end of the data interaction channel; A data security layer, used to determine a control strategy, an encryption strategy and a desensitization strategy between the first interaction end and the second interaction end in the data interaction channel, and to process the interaction data between the first interaction end and the second interaction end using the control strategy, the encryption strategy and the desensitization strategy; The monitoring and optimization layer is used to monitor the operating status of the interactive data in the data source layer, the data processing layer, the data storage layer, the data service layer, the application layer and the data security layer.

2. The data management system according to claim 1, characterized in that: The data source layer includes: A log management unit, used to obtain a first data source corresponding to the log management process and a first data type parameter, a first data flow direction parameter and a first data volume corresponding to the log management process; A department management unit, used for acquiring a second data source corresponding to the department management and a second data type parameter, a second data flow direction parameter, and a second data volume corresponding to the department management; A user management unit, used for obtaining a third data source corresponding to user management and its corresponding third data type parameter, third data flow direction parameter and third data volume; A role management unit, used for acquiring a fourth data source corresponding to the role management and a fourth data type parameter, a fourth data flow direction parameter and a fourth data volume corresponding to the role management; The module management unit is used to obtain the fifth data source corresponding to the module management and its corresponding fifth data type parameter, fifth data flow direction parameter and fifth data volume.

3. The data management system according to claim 1, characterized in that: The data processing layer includes: A data extraction unit, configured to determine the corresponding data source according to the data type parameter and the data flow direction parameter, determine the data processing strategy corresponding to the data source using the data volume, and obtain extracted data from the data source after performing data request processing on the data source using the data processing strategy; A data cleaning unit, configured to determine a data cleaning rule based on a format parameter corresponding to the extracted data, and process the extracted data using the data cleaning rule to obtain cleaned data corresponding to the extracted data; The data loading unit is used to determine a data conversion rule based on the conversion parameters corresponding to the cleaned data, and perform data conversion on the cleaned data using the data conversion rule to obtain the output data.

4. The data management system according to claim 1, characterized in that: The data storage layer comprises: An operation data storage unit is used to save the output data to the data warehouse, and to construct a first data interaction channel corresponding to the data query based on the first data architecture parameter corresponding to the query requirement of the output data; a data warehouse unit, used to obtain a second data architecture parameter corresponding to the output data in the operation data storage unit when summarizing, aggregating, and analyzing the output data, and determine a second data interaction channel corresponding to the data summarizing, data aggregating, and data analyzing the output data based on the second data architecture parameter; The data mart unit is used to obtain the third data architecture parameters corresponding to the visualization processing of the output data in the operation data storage unit, and determine the third data interaction channel corresponding to the visualization processing of the output data based on the third data architecture parameters.

5. The data management system according to claim 1, characterized in that: The data service layer includes: A data query service unit, configured to construct the data query service according to the data query service interface corresponding to the application layer and based on the data query interaction end corresponding to the application layer in the data interaction channel; A data analysis service unit, configured to construct the data analysis service according to the data analysis service interface corresponding to the application layer and based on the data analysis interaction terminal corresponding to the application layer in the data interaction channel; The data report generation service unit is used to build the data report generation service according to the data report generation service interface corresponding to the application layer and based on the data report generation interaction terminal corresponding to the application layer in the data interaction channel.

6. The data management system according to claim 5, characterized in that: The application layer includes: A data query access unit, configured to construct a first access service corresponding to the data query service according to a data query access interface corresponding to the data query service unit and based on a data query interaction end corresponding to the data query service in the data interaction channel; A data analysis access unit, configured to construct a second access service corresponding to the data analysis service according to a data analysis access interface corresponding to the data analysis service unit and based on a data analysis interaction terminal corresponding to the data analysis service in the data interaction channel; The data report generation access unit is used to construct a third access service corresponding to the data report generation service according to the data report access interface corresponding to the data report generation service unit and based on the data report interaction terminal corresponding to the data report generation service in the data interaction channel.

7. The data management system according to claim 1, characterized in that: The data security layer includes: An access control unit, configured to generate the control strategy according to the role access control parameters between the first interaction terminal and the second interaction terminal, and perform access interaction processing on the role access control interaction data between the first interaction terminal and the second interaction terminal using the control strategy; an encryption unit, configured to generate the encryption policy according to the encryption control parameter between the first interaction terminal and the second interaction terminal, and encrypt the encrypted interaction data between the first interaction terminal and the second interaction terminal using the encryption policy; A desensitization unit is used to generate the desensitization strategy according to the desensitization control parameters between the first interaction end and the second interaction end, and use the desensitization strategy to desensitize the desensitized interaction data between the first interaction end and the second interaction end.

8. The data management system according to claim 1, characterized in that: The monitoring and optimization layer includes: A visualization unit, configured to obtain, by using a service monitoring tool, operating status parameters corresponding to the interactive data in the data source layer, the data processing layer, the data storage layer, the data service layer, the application layer, and the data security layer, and monitor the interactive data based on the operating status parameters; The log analysis unit is used to obtain the log corresponding to the interaction data, generate analysis parameters corresponding to the log based on the operation status parameters, and analyze and process the log using the analysis parameters.

9. A data management method, characterized in that: The method is applied to the data management system according to any one of claims 1 to 8, and the method comprises: Using the data source layer to obtain the data source and its corresponding data type parameters, data flow parameters and data volume; Determine a data processing strategy based on the data type parameter, the data flow direction parameter and the data volume by using the data processing layer, and obtain output data of the data source layer after processing the data request by using the data processing strategy; Using the data storage layer, the output data is saved in a preset data warehouse, and a data interaction channel is constructed based on the data architecture parameters of the data warehouse; Using the data service layer, building data query services, data analysis services, and data report generation services corresponding to the data source according to the first interaction end of the data interaction channel; Using the application layer, building access services corresponding to the data query service, the data analysis service, and the data report generation service according to the second interaction end of the data interaction channel; Determine, by using the data security layer, a control strategy, an encryption strategy, and a desensitization strategy between the first interaction end and the second interaction end in the data interaction channel, and process the interaction data between the first interaction end and the second interaction end by using the control strategy, the encryption strategy, and the desensitization strategy; The monitoring and optimization layer is used to monitor the operating status of the interactive data in the data source layer, the data processing layer, the data storage layer, the data service layer, the application layer and the data security layer.

10. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the data management method according to claim 9.

Images