Big data management method, system, electronic device and storage medium

Abstract

The present application discloses a big data management method, system, electronic device and storage medium. Among them, the big data management system includes: a data integration module, which is used to obtain the business data of each data source in multiple data sources; a data storage module, which is deployed with a data asset center, and the data storage module is used to integrate the business data of each data source according to the data type of the business data of each data source and store it in a distributed storage space, and the data asset center builds data assets based on the business data in the distributed storage space; a data calculation module, which is deployed with multiple data calculation engines, which are used to call the corresponding data calculation engine to process the data assets according to user needs, so as to provide data services to users. The embodiment of the present application can provide users with a variety of data application services through data integration, data storage, and data calculation, and has a business data integration capability with high reliability, high security, strong fault tolerance and ease of use.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a big data management method, system, electronic device and storage medium. Background Art

[0002] In related technologies, various information systems of enterprises are mostly purchased or built independently, and cannot achieve information interconnection, resulting in multiple data islands within the enterprise. The development of the Internet and mobile Internet has brought many new business models, and the data generated cannot be interoperable with the data under the traditional model, further exacerbating the problem of data islands. Data scattered in various islands cannot well support the business decisions of enterprises and cannot well respond to rapidly changing business needs. Summary of the invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desired to provide a big data management method, system, electronic device and storage medium, which can provide users with a variety of data application services through data integration, data storage and data calculation, and have high reliability, high security, strong fault tolerance and easy-to-use business data integration capabilities.

[0004] In a first aspect, an embodiment of the present application provides a big data management system, including:

[0005] A data integration module, used to obtain business data from each of multiple data sources;

[0006] A data storage module, in which a data asset center is deployed. The data storage module is used to integrate the business data of each data source and store it in a distributed storage space according to the data type of the business data of each data source. The data asset center builds data assets based on the business data in the distributed storage space;

[0007] The data computing module is deployed with multiple data computing engines, which are used to call corresponding data computing engines to process the data assets according to user needs to provide data services for users.

[0008] In a second aspect, an embodiment of the present application provides a big data management method, including:

[0009] Obtain business data from each of multiple data sources;

[0010] According to the data type of the business data of each data source, the business data of each data source is integrated and stored in a distributed storage space, and data assets are constructed based on the business data in the distributed storage space;

[0011] The data assets are processed according to user needs to provide data services to users.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the big data management method as described in the second aspect above is implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program is used to implement the big data management method described in the second aspect above.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product on which a computer program is stored, and the computer program is used to implement the big data management method as described in the second aspect above.

[0015] The big data management method, system, electronic device and storage medium provided in the embodiments of the present application obtain business data of each data source from multiple data sources through a data integration module, integrate the business data of each data source and store it in a distributed storage space through a data storage module, and finally process the data assets using a data computing module to provide data services to users. The embodiments of the present invention can provide users with a variety of data application services through data integration, data storage, and data computing, and have a business data integration capability with high reliability, high security, strong fault tolerance and ease of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0017] Figure 1 This is a structural block diagram of a big data management system according to an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a big data management system according to an embodiment of the present application;

[0019] Figure 3 A flowchart of a big data management method according to an embodiment of the present application;

[0020] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant disclosure, rather than to limit the disclosure. It should also be noted that, for ease of description, only the parts related to the disclosure are shown in the accompanying drawings.

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] Figure 1 is a structural block diagram of a big data management system according to an embodiment of the present application. Figure 1 As shown, according to an embodiment of the present application, a big data management system includes: a data integration module 110, a data storage module 120 and a data calculation module 130, wherein:

[0024] The data integration module 110 is used to obtain the business data of each of the multiple data sources. The data storage module 120 is deployed with a data asset center 121, which is used to integrate the business data of each data source and store it in a distributed storage space according to the data type of the business data of each data source. The data asset center 121 builds data assets based on the business data in the distributed storage space. The data computing module 130 is deployed with multiple data computing engines, which are used to call the corresponding data computing engine to process the data assets according to user needs to provide data services to users.

[0025] In one embodiment of the present invention, the data type of the business data includes structured business data, semi-structured business data and unstructured business data. Figure 2 As shown, the data integration module 110 uses visual data integration and synchronization tools to obtain structured business data and semi-structured business data of each of the multiple data sources, and the data integration module 110 uses a web crawler to obtain unstructured business data of each of the multiple data sources. The distributed storage space of the data storage module 120 includes a distributed file system HDFS and a distributed file system Ceph, and the data asset center 121 builds a data lake based on the distributed file system HDFS and the distributed file system Ceph, wherein the data lake is part of the data assets. A data warehouse is also deployed in the data storage module 120, and the data warehouse is built based on a data model and the data lake, wherein the data warehouse is part of the data assets.

[0026] Specifically, combined Figure 2As shown, the data storage module 120 supports distributed file systems such as HDFS and Ceph, and supports business storage requirements of relational databases, various Sql databases, feature vector databases, etc.

[0027] The data integration module 110 supports the collection and aggregation of different types of master data from Internet public data sources, various business systems, third-party subsystems, IoT platforms, AI platforms, etc., and supports the collection of structured data, semi-structured data and unstructured data.

[0028] The data computing module 130 supports computing engines such as Hive, MapReduce and other batch computing, Spark memory computing, ClickHouse OLAP engine, Flink streaming computing, etc., so as to meet the data computing needs of business scenarios in various fields.

[0029] Data Asset Center 121 can build a data lake, connect and integrate various information systems, break down data silos, build a data asset map, and provide one-stop data asset construction, management, and sharing capabilities.

[0030] The business data processing flow of the big data management system of the embodiment of the present invention is as follows:

[0031] The data integration module 110 realizes data synchronization, data exchange, data migration, data transmission, file import and export, and data aggregation between various business systems on the Internet, the same local area network, different network domains, and multiple hierarchical domains, and realizes the integration of multiple data sources. Among them, the data integration methods are roughly divided into three categories: full collection, subscription incremental collection, and single query collection. The supported data source types are: databases (relational databases, NoSQL databases, columnar databases, distributed parallel processing databases, etc.), search engines (ElasticSearch, etc.), big data platforms (HDFS, Hive, HBase, MongoDB, Impala, etc.), service interfaces (REST API, WebService interface, socket interface, etc.), message queues (kafka, RabbitMQ, etc.), file servers (ftp, sftp, etc.), and many other data source types.

[0032] Through data integration, the data of equipment and business systems can be aggregated together. The distributed file system HDFS is used to store structured data and semi-structured data, and the object storage Ceph is used to store unstructured data (pictures, files, audio, video). By building a basic data lake with HDFS and Ceph, data from various sources and types of data can be stored, and data sharing in various application scenarios can be supported. Based on the initial source data in the data lake, a data warehouse can be built through data calculation and data modeling to meet the needs of business operation analysis and data analysis.

[0033] Data computing related to big data can be divided into offline computing, real-time computing, in-memory computing, and OLAP (On-Line Analytical Processing). Among them, big data computing engines such as Flink support stream computing. The data computing engine of the embodiment of the present invention can support two computing engines: MapReduce, Spark, and Flink. Resource scheduling relies on YARN to execute, realizing batch and stream integration. Offline computing is implemented using the MapReduce computing engine, real-time computing is implemented using the Flink computing engine, in-memory computing is implemented using the Spark computing engine, and OLAP is implemented using the ClickHouse column database.

[0034] Data processing is performed using two architectures: stream processing and batch processing. The stream processing part is responsible for real-time data processing, and the batch processing part is responsible for the processing of large amounts of historical data. The batch processing service regularly extracts data into the distributed file system HDFS, and then passes it to the distributed computing database Hive for processing. The accurate results after processing are provided by the ClickHouse real-time query engine. The real-time processing service obtains data from Kafka in real time and passes it to the Flink real-time computing engine for real-time processing. The processing results will be stored in the ClickHouse database that supports real-time reading and writing. The data warehouse construction model adopts the bottom-up data warehouse modeling method proposed by Inmon. The logical layering of the data warehouse is the original data layer ODS, the detailed data warehouse layer DWD, the summary data warehouse layer DWS, the application data layer ADS, and the dimension layer DIM. The data lake and the data warehouse together constitute the overall data assets. By connecting the data of each business system and all products, the data only needs to be processed once, which can realize data reuse and sharing, abandon the original chimney-style development and storage of each business, and reduce development costs and manpower and material resources.

[0035] In one embodiment of the present invention, it also includes: an interface module, wherein the interface module provides a data service interface and an application service interface, wherein the data service interface is used to provide data sharing services for users, and the application service interface is used to provide data application services for users. In addition, the big data management system also includes: a data governance module for managing the data assets; a data operation and maintenance module for providing data operation and maintenance services for the big data management system.

[0036] Specifically, building data assets requires management and maintenance from beginning to end. Data governance is implemented from beginning to end and in the middle of the process of building the entire big data platform. Data governance refers to the process and practice of enterprise data management, including metadata management, data quality management, data cleaning and conversion, data desensitization and encryption, data lifecycle management, etc. Metadata management is to establish big data category information, improve the description of data assets through metadata management, and at the same time, in the process of building data assets, pay attention to the refinement of the business meaning of data assets, and process and organize data into unambiguous data assets. Data quality management is like the company's regular maintenance of physical assets, regularly purifying big data, providing full-link data quality audit monitoring, quality screen, quality score, monitoring history and other functions, and monitoring abnormal data based on basic rules and custom rules. Data cleaning and conversion is to provide a variety of conversion and cleaning functions for field-level data of each data source to obtain clean data. Data desensitization and encryption are security measures for data. Data desensitization is to deform data through desensitization rules for certain sensitive information to achieve reliable protection of sensitive privacy data. Data encryption is the process of converting data into unrecognizable data through keys and encryption algorithms, and the original value can only be restored through decryption. Data lifecycle management is the archiving of big data and the deletion of certain data when it is no longer necessary to keep it.

[0037] Based on data assets, data services and data applications can be provided. Data services are provided through a unified data access portal, shielding the complexity of heterogeneous data while controlling the security of data access and data content. Data services are achieved by building various thematic data API interfaces to share data, and API interfaces are also provided to support BI reports to display the data results required for calculation. Data applications include intelligent applications such as intelligent prediction, knowledge graph, recommendation system, vector search, etc. Intelligent prediction is to predict the results of specified indicators within a certain period of time in the future based on historical business data and AI perception data. Knowledge graph is to build a knowledge graph in related fields based on business data, and provide visual presentation and retrieval. The recommendation system recommends the most relevant business products to users based on business data, historical data and historical behavior data. Vector retrieval realizes the creation and maintenance of TB-level / 500,000+ vector libraries, and queries similar vector lists in seconds. Platform operation and maintenance exists in the entire life cycle of big data platform construction: providing monitoring and operation services for hardware and software equipment components and application services, realizing alarm services, and providing cluster expansion services to ensure the demand for cluster servers for rapid business growth.

[0038] According to the big data management system of the embodiment of the present invention, the business data of each data source in multiple data sources is obtained through the data integration module, the business data of each data source is integrated and stored in the distributed storage space through the data storage module, and finally the data assets can be processed by the data calculation module to provide data services for users. The system can provide users with a variety of data application services through data integration, data storage, and data calculation, and has a business data integration capability with high reliability, high security, strong fault tolerance, and ease of use.

[0039] On the other hand, Figure 3 As shown, the embodiment of the present application provides a big data management method, comprising the following steps:

[0040] S301: Acquire business data of each data source among multiple data sources;

[0041] S302: Integrate the business data of each data source and store them in a distributed storage space according to the data type of the business data of each data source, and construct data assets based on the business data in the distributed storage space;

[0042] S303: Process the data assets according to user needs to provide data services to users.

[0043] In one embodiment of the present application, the data type of the business data includes structured business data, semi-structured business data and unstructured business data. The data integration module uses a visual data integration and synchronization tool to obtain structured business data and semi-structured business data from each of the multiple data sources. The data integration module uses a web crawler to obtain unstructured business data from each of the multiple data sources.

[0044] In one embodiment of the present application, the distributed storage space of the data storage module includes a distributed file system HDFS and a distributed file system Ceph, and the data asset center builds a data lake based on the distributed file system HDFS and the distributed file system Ceph, wherein the data lake is part of the data assets.

[0045] In one embodiment of the present application, a data warehouse is also deployed in the data storage module, and the data warehouse is constructed based on the data model and the data lake, wherein the data warehouse is a part of the data assets.

[0046] In one embodiment of the present application, it also includes:

[0047] An interface module, wherein the interface module provides a data service interface and an application service interface, wherein the data service interface is used to provide data sharing services for users, and the application service interface is used to provide data application services for users.

[0048] In one embodiment of the present application, it also includes:

[0049] A data governance module, used to manage the data assets;

[0050] The data operation and maintenance module is used to provide data operation and maintenance services to the big data management system.

[0051] According to the big data management method of the embodiment of the present invention, the business data of each data source in multiple data sources is obtained through the data integration module, the business data of each data source is integrated and stored in the distributed storage space through the data storage module, and finally the data assets can be processed by the data calculation module to provide data services for users. The method can provide users with a variety of data application services through data integration, data storage, and data calculation, and has a business data integration capability with high reliability, high security, strong fault tolerance, and ease of use.

[0052] It should be noted that the specific implementation method of the big data management method of the embodiment of the present application is similar to the specific implementation method of the big data management system of the embodiment of the present application. Please refer to the description of the method part for details, and no further details will be given here.

[0053] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0054] like Figure 4 As shown, the electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from the storage part 602 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0055] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0056] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, the above-mentioned functions defined in the electronic device of the present application are executed.

[0057] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electronic device, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction-executing electronic device, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction-executing electronic device, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0058] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the processing receiving device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based electronic device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0059] The units or modules involved in the embodiments of the present application may be implemented by software or hardware. The units or modules described may also be provided in a processor, and the processor is used to implement a big data management method when executing the program: obtaining business data of each of a plurality of data sources; integrating the business data of each data source and storing it in a distributed storage space according to the data type of the business data of each data source, and constructing data assets based on the business data in the distributed storage space; processing the data assets according to user needs to provide data services to users.

[0060] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the big data management method described in the present application: obtaining business data of each data source from multiple data sources; according to the data type of the business data of each data source, integrating the business data of each data source and storing it in a distributed storage space, and constructing data assets based on the business data in the distributed storage space; processing the data assets according to user needs to provide data services to users.

[0061] As another aspect, the present application also provides a computer program product, which may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device. The above computer program product stores one or more programs, and when the above programs are used by one or more processors to execute the big data management method described in the present application: obtaining business data of each data source from multiple data sources; according to the data type of the business data of each data source, integrating the business data of each data source and storing it in a distributed storage space, and constructing data assets based on the business data in the distributed storage space; processing the data assets according to user needs to provide data services to users.

[0062] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A big data management system, characterized in that: include: A data integration module, used to obtain business data from each of multiple data sources; The data types of the business data include structured business data, semi-structured business data and unstructured business data; A data storage module, in which a data asset center is deployed. The data storage module is used to integrate the business data of each data source and store it in a distributed storage space according to the data type of the business data of each data source. The data asset center builds data assets based on the business data in the distributed storage space; The distributed storage space of the data storage module includes a distributed file system HDFS and a distributed file system Ceph, and the data asset center builds a data lake based on the distributed file system HDFS and the distributed file system Ceph, wherein the data lake is a part of the data assets; The data computing module is deployed with multiple data computing engines, which are used to call corresponding data computing engines to process the data assets according to user needs to provide data services for users.

2. The big data management system according to claim 1, characterized in that: The data integration module uses a visual data integration and synchronization tool to obtain structured business data and semi-structured business data from each of the multiple data sources, and the data integration module uses a web crawler to obtain unstructured business data from each of the multiple data sources.

3. The big data management system according to claim 1, characterized in that: A data warehouse is also deployed in the data storage module. The data warehouse is constructed based on the data model and the data lake, wherein the data warehouse is a part of the data assets.

4. The big data management system according to claim 1, characterized in that: Also includes: An interface module, wherein the interface module provides a data service interface and an application service interface, wherein the data service interface is used to provide data sharing services for users, and the application service interface is used to provide data application services for users.

5. The big data management system according to claim 1, characterized in that: Also includes: A data governance module, used to manage the data assets; The data operation and maintenance module is used to provide data operation and maintenance services to the big data management system.

6. A big data management method, wherein the big data management system according to any one of claims 1 to 5 is used to execute the big data management method, characterized in that: include: Obtain business data from each of multiple data sources; According to the data type of the business data of each data source, the business data of each data source is integrated and stored in a distributed storage space, and data assets are constructed based on the business data in the distributed storage space; The data assets are processed according to user needs to provide data services to users.

7. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the big data management method according to claim 6 when executing the program.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, it is used to implement the big data management method according to claim 6.

9. A computer program product, characterized in that A computer program is stored thereon, and when the computer program is executed, it is used to implement the big data management method according to claim 6.

Images