Enterprise standardized data computing platform

Abstract

The invention discloses an enterprise standardized data computing platform which comprises a standardized data integration and model construction module which integrates data cleaning, data conversion and data loading, converts non-standardized data from different systems into a unified standard format and loads the non-standardized data into a preset data model; the high-performance memory data processing engine adopts memory data processing to realize complex calculation and real-time analysis; through an intelligent caching mechanism, the data processing efficiency is optimized; the full-scene privacy calculation module is integrated with a privacy calculation technology, so that the platform has full-scene secret state calculation capability, and wide-area data circulation and cooperation among multiple mechanisms are realized on the premise of ensuring that data are not leaked; the method has the advantages that the privacy computing technology is deeply integrated into an enterprise data computing platform, efficient circulation and cooperation of data in a secret state are achieved, the data application scene is greatly expanded, and meanwhile the safety and privacy of the data are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of data computing, and in particular relates to an enterprise standardized data computing platform. Background Art

[0002] The process of data standardization is actually to implement data standards on the data integration platform, and load the data generated by each system into the data model of the integration platform through cleaning and conversion to achieve data standardization.

[0003] Data standards are the foundation of data asset management and are the process of accurately defining data assets. For an enterprise with a large amount of data assets, or an enterprise that wants to realize data asset transactions, building data standards is a must.

[0004] A unified data computing platform with publication number CN118277471A is disclosed, which includes a data model unit, a memory data processing unit, an API interface unit and a modular component unit. The data model unit is used to build a visual data computing platform, unify and compatibly realize data from different data sources, and as a data base, ensure the data unification and stability of different front-end systems, and can ensure mutual migration between different front-end systems, and support rapid iteration of business needs; the memory data processing unit replaces the traditional database access method by using memory to load data, thereby ensuring high performance and rapid response of data during processing, and uses the data model unit to realize complex calculation and data processing tasks; the API interface unit includes a unified input and output parameter design for developers to integrate and use the data model unit; the modular component unit is responsible for processing specific data processing or computing tasks, and the modular component unit is independently updated to ensure the flexibility and scalability of the platform to adapt to rapidly changing business needs.

[0005] Existing technologies, such as the unified data computing platform with publication number CN118277471A, have achieved unified management and efficient processing of data to a certain extent, but they still have shortcomings in terms of complex and changing business needs, data security, and cross-institutional data circulation. Summary of the invention

[0006] The purpose of the present invention is to provide an enterprise standardized data computing platform that can realize standardized data processing and efficient computing, and ensure high security, high performance and high availability of data circulation.

[0007] To achieve the above objectives, the present invention provides the following technical solutions: an enterprise standardized data computing platform, comprising

[0008] Standardized data integration and model building module: This module integrates data cleaning, data conversion and data loading, converts non-standardized data from different systems into a unified standard format, and loads it into the preset data model;

[0009] High-performance memory data processing engine: Use memory data processing to achieve complex calculations and real-time analysis; optimize data processing efficiency through intelligent caching mechanism;

[0010] Full-scenario privacy computing module: Integrates privacy computing technology to enable the platform to have full-scenario confidential computing capabilities, and realize wide-area data circulation and collaboration among multiple institutions without leaking data;

[0011] Comprehensive early warning and monitoring mechanism: The platform has a built-in early warning and monitoring mechanism that can monitor key indicators in the data processing process in real time, promptly discover and deal with potential problems, and ensure the stable operation of the system and data security.

[0012] As a preferred technical solution of the present invention, the privacy computing technology includes homomorphic encryption and secure multi-party computing.

[0013] As a preferred technical solution of the present invention, the preset data model includes a data warehouse model, a relational data model, and a NoSQL data model.

[0014] As a preferred technical solution of the present invention, the following principles are followed when constructing the preset data model: business demand orientation; standardization and normalization; scalability and flexibility; performance optimization.

[0015] As a preferred technical solution of the present invention, the data cleaning includes removing duplicate data; processing missing values; and correcting erroneous data.

[0016] As a preferred technical solution of the present invention, the data conversion includes defining a standard format; data mapping; converting data types; data standardization; and data aggregation and splitting.

[0017] As a preferred technical solution of the present invention, the data loading includes selecting a data model; formulating a data loading strategy; executing data loading; and verifying data quality.

[0018] As a preferred technical solution of the present invention, the intelligent cache mechanism includes page cache; database cache; object-level cache; and query result cache.

[0019] As a preferred technical solution of the present invention, the early warning and monitoring mechanism includes real-time monitoring of data processing; emergency response and processing; and monitoring and early warning optimization.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] Deeply integrating privacy computing technology into the enterprise data computing platform has achieved efficient data circulation and collaboration in a confidential state, greatly expanding data application scenarios while ensuring data security and privacy;

[0022] Through the modular component architecture, the present invention provides extremely high flexibility and scalability, enabling enterprises to quickly adjust system configurations according to their own business needs, reducing operation and maintenance costs and improving system response speed;

[0023] The innovative early warning and monitoring mechanism can monitor the entire data processing chain in real time, discover and solve problems in advance, avoid potential data risks, and improve the company's data governance capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Example 1

[0027] See also Figure 1 , which is the first embodiment of the present invention, and provides an enterprise standardized data computing platform, including

[0028] Standardized data integration and model building module: This module integrates advanced data cleaning, conversion and loading technologies, and can automatically convert non-standardized data from different systems into a unified standard format and load it into the preset data model. At the same time, this module supports dynamic adjustment of data models to adapt to rapid changes in business needs;

[0029] High-performance memory data processing engine: Different from traditional database access methods, this invention adopts memory data processing technology to greatly improve data processing speed and support complex calculations and real-time analysis; through intelligent caching mechanism, it further optimizes data processing efficiency and ensures that the system still maintains excellent performance in high-concurrency scenarios;

[0030] Full-scenario privacy computing module: Integrates privacy computing technologies such as homomorphic encryption and secure multi-party computing, enabling the platform to have full-scenario confidential computing capabilities; Under the premise of ensuring that data is not leaked, it enables wide-area data circulation and collaboration among multiple institutions, meeting the needs of large-scale data opening and sharing;

[0031] Modular component architecture: The platform adopts a modular design, and each component is independently developed, updated and maintained, which greatly improves the flexibility and scalability of the system. Developers can quickly combine or replace components according to specific business needs to adapt to the rapidly changing market environment.

[0032] Comprehensive early warning and monitoring mechanism: The platform has a built-in comprehensive early warning and monitoring mechanism that can monitor key indicators in the data processing process in real time, promptly discover and handle potential problems, and ensure the stable operation of the system and data security.

[0033] In this embodiment, preferably, the preset data model includes a data warehouse model, a relational data model, and a NoSQL data model;

[0034] Data warehouse model: It is a common form of preset data model, which is used to support complex data analysis and reporting requirements; data warehouse models usually include star model, snowflake model, etc. These models organize data through dimension tables and fact tables to efficiently support query and analysis;

[0035] Relational data model: Another common default data model, it is based on relational algebra and relational database theory. In the relational data model, data is organized into tables, and tables are related to each other through relationships such as foreign keys. The relational data model supports complex queries and transaction processing and is the basis of many database management systems.

[0036] NoSQL data model: NoSQL data models include key-value storage, column storage, document storage, and graph databases, each of which has different characteristics and applicable scenarios; NoSQL data models are usually used to process large-scale, unstructured or semi-structured data.

[0037] In this embodiment, preferably, the following principles are followed when constructing the preset data model:

[0038] Business demand orientation: The data model should be designed closely around business needs to ensure that it can meet the needs of business analysis and decision-making;

[0039] Standardization and normalization: Data models should follow certain standards and specifications to ensure data consistency and interoperability;

[0040] Scalability and flexibility: The data model should have a certain degree of scalability and flexibility to adapt to the needs of future business development and data growth;

[0041] Performance optimization: When designing a data model, you need to consider factors such as query performance and data loading efficiency to ensure that the data model can efficiently support business operations and analysis.

[0042] In this embodiment, preferably, data cleaning includes

[0043] De-duplicate data: Identify and delete or merge duplicate records through database query technology or ETL tools;

[0044] Handling missing values: Based on business rules, use methods such as mean, median, mode, etc. to fill missing values, or directly treat missing values ​​as special values;

[0045] Correcting erroneous data: Identifying and correcting formatting errors, logical errors, or outliers requires the use of regular expressions, data validation rules, or machine learning algorithms.

[0046] In this embodiment, preferably, data conversion includes

[0047] Define standard formats: Define unified data structures, types, encoding rules, and naming conventions based on business requirements and data characteristics;

[0048] Data mapping: Establish the mapping relationship between source data fields and target data fields to ensure that the data can correspond correctly during the conversion process;

[0049] Convert data types: Use ETL tools or programming scripts to convert fields of different data types into a unified data type, such as converting a string date to a date type;

[0050] Data standardization: Apply data standardization methods, such as data normalization, standardized coding, etc., to ensure that the data values ​​are within a consistent dimension and range;

[0051] Data aggregation and splitting: Aggregate or split data according to business needs to meet different analysis requirements.

[0052] In this embodiment, preferably, data loading includes

[0053] Select a data model: Select an appropriate data model, such as a relational data model, a NoSQL data model, or a business-specific model, based on business requirements and data characteristics;

[0054] Data loading strategy: formulate data loading strategy, including loading order, concurrency control, error handling, etc.;

[0055] Perform data loading: Use ETL tools or database management tools to load the transformed data into the target system;

[0056] Verify data quality: After data loading is completed, perform quality verification on the loaded data to ensure the accuracy and completeness of the data.

[0057] Example 2

[0058] See also Figure 1 , which is the second embodiment of the present invention, is based on the previous embodiment, except that:

[0059] Smart caching mechanisms include

[0060] Page caching: Cache dynamically generated page content. When the same page is requested again, the content can be returned directly from the cache, thereby reducing the overhead of database queries and page rendering.

[0061] Database cache: Store frequently accessed data in the cache to reduce the number of direct queries to the database. This can be achieved through an in-memory database or cache middleware. In high-performance in-memory data processing engines, this cache mechanism is particularly important and can significantly improve data access speed.

[0062] Object-level caching: Cache frequently used objects or data in memory to improve read and write speeds; this caching mechanism is usually managed by a caching framework and can be optimized for specific objects to reduce memory usage and improve performance;

[0063] Query result cache: The results of database queries are cached. When the same query request is encountered next time, the cached results are directly returned. This can not only reduce the query pressure on the database, but also significantly improve the query response speed.

[0064] Early warning and monitoring mechanisms include

[0065] Real-time monitoring of data processing

[0066] Key indicator monitoring: Real-time monitoring of key indicators in the data processing process, such as system load, response time, throughput, data flow, etc., to detect anomalies in a timely manner;

[0067] Log collection and analysis: Collect system logs, application logs, database logs, etc., and identify potential problems or abnormal behaviors through log analysis;

[0068] Early warning system

[0069] Preset thresholds: Preset thresholds for various indicators based on business needs and system performance requirements; once an indicator exceeds or falls below the preset threshold, the system will automatically trigger an alert;

[0070] Intelligent analysis: Use machine learning, big data analysis and other technologies to conduct intelligent analysis of monitoring data and predict potential risks or problems;

[0071] Alarm strategy: allows users to set refined alarm trigger conditions according to actual needs, such as alarm level (serious, warning, prompt, etc.), alarm content, alarm duration, alarm shielding period, etc.;

[0072] Emergency response and handling

[0073] Automatic response: After the warning is triggered, the system can automatically execute preset emergency response measures, such as restarting services, switching backup data, etc.

[0074] Manual intervention: For issues that require manual processing, the system will promptly notify relevant personnel to ensure that the problem is resolved in a timely manner;

[0075] Monitoring and early warning optimization

[0076] Monitoring indicator optimization: Continuously optimize the selection of monitoring indicators and the setting of thresholds based on changes in business needs and improvements in system performance;

[0077] Early warning strategy optimization: Based on historical data and actual operation conditions, the early warning strategy is continuously optimized to improve the accuracy and effectiveness of early warnings.

[0078] Although the embodiments of the present invention have been shown and described, as detailed above, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An enterprise standardized data computing platform, characterized by: include Standardized data integration and model building module: This module integrates data cleaning, data conversion and data loading, converts non-standardized data from different systems into a unified standard format, and loads it into the preset data model; High-performance memory data processing engine: Use memory data processing to achieve complex calculations and real-time analysis; optimize data processing efficiency through intelligent caching mechanism; Full-scenario privacy computing module: Integrates privacy computing technology to enable the platform to have full-scenario confidential computing capabilities, and realize wide-area data circulation and collaboration among multiple institutions without leaking data; Comprehensive early warning and monitoring mechanism: The platform has a built-in early warning and monitoring mechanism that can monitor key indicators in the data processing process in real time, promptly discover and deal with potential problems, and ensure the stable operation of the system and data security.

2. The enterprise standardized data computing platform according to claim 1, characterized in that: The privacy computing technology includes homomorphic encryption and secure multi-party computing.

3. The enterprise standardized data computing platform according to claim 1, characterized in that: The preset data models include data warehouse models, relational data models, and NoSQL data models.

4. The enterprise standardized data computing platform according to claim 1, characterized in that: When building a preset data model, follow the following principles: business demand-oriented; Standardization and normalization; scalability and flexibility; performance optimization.

5. The enterprise standardized data computing platform according to claim 1, characterized in that: The data cleaning includes removing duplicate data; processing missing values; and correcting erroneous data.

6. The enterprise standardized data computing platform according to claim 1, characterized in that: The data conversion includes defining a standard format; data mapping; converting data types; data standardization; Data aggregation and splitting.

7. The enterprise standardized data computing platform according to claim 1, characterized in that: The data loading includes selecting a data model; formulating a data loading strategy; executing data loading; and verifying data quality.

8. The enterprise standardized data computing platform according to claim 1, characterized in that: The intelligent cache mechanism includes page cache; database cache; object-level cache; and query result cache.

9. The enterprise standardized data computing platform according to claim 1, characterized in that: Early warning and monitoring mechanisms include real-time monitoring of data processing; emergency response and processing; Monitoring and early warning optimization.

Images