Data analysis method and system and computer program product

Abstract

The invention particularly relates to a data analysis method and system. The method comprises the steps of extracting original data corresponding to a target service from a target data source according to a pre-configured data extraction strategy of the target service, loading the original data to an operation data storage layer of a data warehouse, and storing the original data; operating a data storage layer to output the original data to a detail data layer; the detailed data layer performs data preprocessing on the original data to obtain basic data and outputs the basic data to the summarized data layer; the summarizing data layer performs aggregation processing on the basic data according to a current data processing task corresponding to the target business, obtains structured data corresponding to the target business, and outputs the structured data to the application data service layer; and the application data service layer performs data analysis on the structured data according to the service configuration information corresponding to the current data processing task, and outputs a corresponding data analysis result. According to the scheme, efficient and accurate data management and analysis can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data analysis method, a data analysis system, a storage medium, a computer program product, and an electronic device. Background Art

[0002] With the rapid development of the Internet and computer technology, the amount of transaction data in industries such as finance and e-commerce has increased significantly; the corresponding demand for data analysis has also gradually increased. Taking the financial industry as an example, it is necessary to accurately analyze massive amounts of transaction data to provide data support for risk assessment and decision-making of financial institutions. However, traditional data analysis methods are often limited by performance bottlenecks when faced with large-scale data, complex query requirements and high-throughput scenarios, making it difficult to achieve efficient expansion, thus posing a challenge to business advancement.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides a data analysis method and system, a storage medium, a computer program product, and an electronic device, which can effectively overcome the defects existing in the prior art.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to a first aspect of the present disclosure, a data analysis method is provided, the method comprising:

[0007] According to the data extraction strategy preconfigured for the target business, the original data corresponding to the target business is extracted from the target data source, and the original data is loaded into the operational data storage layer of the data warehouse and stored;

[0008] The operation data storage layer outputs the raw data to the detailed data layer; the detailed data layer performs data preprocessing on the raw data to obtain basic data, and outputs the basic data to the summary data layer;

[0009] The summary data layer aggregates the basic data according to the current data processing task corresponding to the target business, obtains the structured data corresponding to the target business, and outputs it to the application data service layer;

[0010] The application data service layer performs data analysis on the structured data according to the business configuration information corresponding to the current data processing task, and outputs corresponding data analysis results.

[0011] In some exemplary embodiments, the method further comprises:

[0012] A dimensional data mart layer is constructed based on the data detail layer, and public data corresponding to the target business is stored in the dimensional data mart layer; wherein the public data can be reused in the application data service layer.

[0013] In some exemplary embodiments, the method further comprises:

[0014] Using Hive data warehouse tool to build the operational data storage layer based on Hadoop distributed structure;

[0015] The detailed data layer, summary data layer, and dimension data mart layer are constructed based on the Doris database.

[0016] In some exemplary embodiments, the original data includes non-real-time data and real-time data corresponding to the target service;

[0017] The extracting the original data corresponding to the target business from the target data source according to the data extraction strategy preconfigured for the target business includes:

[0018] Calling the ETL Cloud tool to obtain the non-real-time data from the target data source according to the data extraction strategy; and

[0019] The Flink CDC tool is called to extract the real-time data according to the data extraction strategy; the target data source includes business data.

[0020] In some exemplary embodiments, the data extraction strategy corresponding to the non-real-time data includes customized: data source directory, data conversion configuration information, data loading configuration information;

[0021] The method further comprises:

[0022] In response to an application creation operation inputted in the graphical user interface, an ELT process corresponding to the target service is created;

[0023] Configure a data source directory according to the input data source type and data source configuration information, so as to extract data according to the data source directory;

[0024] At least one data processing node is configured according to the data requirements corresponding to the target business, so as to determine the data conversion configuration information according to the data processing node; wherein the data processing node includes: at least one of data cleaning, data conversion, data filtering, formatting, and field configuration;

[0025] The data loading configuration information is determined according to the output component information and the data output path information.

[0026] In some exemplary embodiments, the method further comprises:

[0027] Open the Binlog log file in the source database and add corresponding configuration information to configure the Binlog log file as a data source, and extract the original data in real time from the data source;

[0028] Using the Flink CDC tool, establish a first mapping table corresponding to the target data table of the database in the data warehouse and a second mapping table corresponding to the source database; and establish a mapping relationship between the first mapping table and the second mapping table;

[0029] Performing data cleaning and / or conversion processing on the extracted raw data based on the data conversion configuration information of the Flink CDC tool;

[0030] The original data processed by data cleaning and / or conversion is synchronously processed based on the mapping relationship.

[0031] In some exemplary embodiments, the method further comprises:

[0032] Receive data query requests;

[0033] Convert the data query request into a structured query statement and decompose it to obtain multiple sub-query statements;

[0034] The multiple sub-query statements are distributed to the distributed storage nodes corresponding to the data warehouse, so that each of the distributed storage nodes processes the sub-query statements.

[0035] In some exemplary embodiments, the detailed data layer performs data preprocessing on the raw data to obtain basic data, including:

[0036] Constructing a corresponding data table model according to the target field corresponding to the data analysis object of the target business and the data type corresponding to the target field;

[0037] The original data is preprocessed based on the data table model to obtain the basic data.

[0038] According to a second aspect of the present disclosure, there is provided a data analysis system, comprising:

[0039] A data extraction module is used to extract the original data corresponding to the target business from the target data source according to the data extraction strategy preconfigured for the target business, and load the original data into the operational data storage layer of the data warehouse for storage;

[0040] A data warehouse, used for the operational data storage layer to output the original data to the detailed data layer; the detailed data layer performs data preprocessing on the original data to obtain basic data, and outputs the basic data to the summary data layer; the summary data layer aggregates the basic data according to the current data processing task corresponding to the target business, obtains the structured data corresponding to the target business, and outputs it to the application data service layer; the application data service layer performs data analysis on the structured data according to the business configuration information corresponding to the current data processing task, and outputs the corresponding data analysis results.

[0041] According to a third aspect of the present disclosure, there is provided a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned data analysis method is implemented.

[0042] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0043] Processor; and

[0044] A memory, configured to store executable instructions of the processor;

[0045] Wherein, the processor is configured to implement the above-mentioned data analysis method by executing the executable instructions.

[0046] According to a fifth aspect of the present disclosure, there is provided a computer program product on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned data analysis method is implemented.

[0047] The data analysis method provided by the embodiment of the present disclosure can realize the automatic extraction, conversion and storage of business data by pre-configuring data extraction strategies, and can realize the real-time capture of real-time data, and use the operational data storage layer, detailed data layer, summary data layer, and application data service layer of the data warehouse to implement the hierarchical management strategy of data, establish a strict quality control system, ensure the accuracy and integrity of data, effectively prevent data pollution and leakage, and enhance the credibility of data assets. In this way, efficient and accurate management and analysis of data can be achieved.

[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0050] Figure 1 A schematic diagram schematically illustrates a data analysis method according to an exemplary embodiment of the present disclosure;

[0051] Figure 2 A schematic diagram schematically illustrates a system architecture corresponding to a data analysis method of an exemplary embodiment of the present disclosure;

[0052] Figure 3 A schematic diagram schematically illustrates a data warehouse in an exemplary embodiment of the present disclosure;

[0053] Figure 4 A schematic diagram schematically illustrates a data analysis system according to an exemplary embodiment of the present disclosure;

[0054] Figure 5 The diagram schematically shows the components of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0056] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0057] In the related technologies, for traditional data analysis solutions, traditional relational databases are often limited by performance bottlenecks when facing large-scale data, complex query requirements and high-throughput scenarios, making it difficult to achieve efficient expansion, thus posing a challenge to business advancement. The shortcomings are: 1) Manual coding and inefficiency: During the system construction process, users are forced to perform manual coding to achieve data extraction, conversion, loading and storage, resulting in complex processes, steep learning curves, and unfriendly to non-professionals. This process is not only time-consuming, but also prone to inefficiency due to differences in personal experience, making it difficult to build an efficient and smooth business closed loop. 2) Database performance bottleneck: Although traditional databases perform well in query and insert operations, they often face performance degradation, low throughput and insufficient data real-time when processing massive amounts of capital data. This is undoubtedly a major limitation for capital systems that rely on real-time, large-scale data analysis, affecting the optimization and experience of business scenarios. 3) Complexity and risk: In terms of data analysis, existing databases usually require the writing of complex data statistics statements and scripts, which not only increases the technical threshold, but also increases the risk of errors, which may have a negative impact on system stability. In addition, when the system is expanded later, it faces the limitations of the database architecture, resulting in high maintenance costs and difficulty in expansion. 4) Lack of data security and permission management: The lack of a strict data security and permission management mechanism poses the risk of data pollution and leakage, which not only threatens business security, but also violates the legal and regulatory requirements for data protection.

[0058] In view of the shortcomings and deficiencies of the prior art, this example embodiment provides a data analysis method based on the OLAP (online analytical processing) architecture and a distributed computing engine, thereby constructing a data analysis solution with excellent performance and easy operation. Figure 1 As shown, the data analysis method may specifically include:

[0059] Step S11, extracting the original data corresponding to the target business from the target data source according to the pre-configured data extraction strategy of the target business, and loading the original data into the operational data storage layer of the data warehouse and storing it;

[0060] Step S12, the operation data storage layer outputs the original data to the detailed data layer; the detailed data layer performs data preprocessing on the original data to obtain basic data, and outputs the basic data to the summary data layer;

[0061] Step S13, the summary data layer aggregates the basic data according to the current data processing task corresponding to the target business, obtains the structured data corresponding to the target business, and outputs it to the application data service layer;

[0062] Step S14: the application data service layer performs data analysis on the structured data according to the business configuration information corresponding to the current data processing task, and outputs corresponding data analysis results.

[0063] The present invention provides a solution based on OLAP architecture and distributed computing engine, and the beneficial effects include: 1. Data base and storage optimization: Doris database is used as the data base, which has the characteristics of column storage and distributed architecture, and can effectively improve query performance and reduce storage space requirements. Through the combination of OLTP and OLAP, Doris can adapt to the dual needs of online business services and offline data analysis, and provide stable and efficient data support for capital analysis. 2. Automated data integration and processing: The ETLCloud platform realizes the automatic extraction, conversion and storage of data, provides intuitive interfaces and components, simplifies the data processing process, and reduces the learning cost. The platform supports the access of multiple data sources, including files, databases and APIs, and ensures the efficiency and flexibility of data processing through process design and task scheduling. 3. Real-time data access and consistency: Flink CDC technology is used to realize the capture of real-time data of business systems, ensure the consistency and real-time of data, and provide the latest and accurate data source for capital analysis. 4. Data stratification and quality control: Implement data stratification management strategy, establish a strict quality control system, ensure the accuracy and integrity of data, effectively prevent data pollution and leakage, and enhance the credibility of data assets. 5. Permission management and role definition: Clarify user roles and permission allocation, such as data reading, writing and analysis permissions, to ensure data security and compliance, while improving the flexibility and efficiency of data use.

[0064] Below, each step of the data analysis method in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0065] In step S11, according to the preconfigured data extraction strategy of the target business, the original data corresponding to the target business is extracted from the target data source, and the original data is loaded into the operational data storage layer of the data warehouse and stored.

[0066] In this example implementation, the scheme is explained by taking the data analysis task of the Internet financial industry as an example. Among them, the above-mentioned target business can be a financial data analysis business. The above-mentioned original data can be real-time data or non-real-time data corresponding to the financial business. Of course, in other exemplary embodiments of the present disclosure, the above-mentioned target business can also be an online commodity sales business, and the corresponding original data can be commodity sales-related data; or, the target business can also be one or more government management businesses corresponding to the government management system, and the corresponding original data can be the corresponding government data. The present disclosure does not specifically limit the business type of the target business.

[0067] Exemplarily, extracting the original data corresponding to the target business from the target data source according to the data extraction strategy preconfigured for the target business includes:

[0068] Calling the ETL Cloud tool to obtain the non-real-time data from the target data source according to the data extraction strategy; and

[0069] The Flink CDC tool is called to extract the real-time data according to the data extraction strategy; the target data source includes business data.

[0070] Exemplary, reference Figure 2 As shown in the figure, in the data extraction and output stage, you can use the ETL Cloud data integration tool and the Flink CDC data integration framework to customize data extraction rules and implement real-time or scheduled extraction of multi-source heterogeneous data. You can configure the data extraction strategy according to the data analysis requirements of the target business. The strategy can include parameters such as data source, data extraction method, and data analysis target. You can use the pre-configured strategy template to build the current data extraction strategy.

[0071] In this example implementation, for non-real-time data, the data extraction strategy corresponding to the non-real-time data includes customized: data source directory, data conversion configuration information, and data loading configuration information.

[0072] The method further comprises:

[0073] Step S21, in response to the application creation operation inputted in the graphical user interface, creating an ELT process corresponding to the target service;

[0074] Step S22, configuring a data source directory according to the input data source type and data source configuration information, so as to extract data according to the data source directory;

[0075] Step S23, configuring at least one data processing node according to the data requirements corresponding to the target business, so as to determine the data conversion configuration information according to the data processing node; wherein the data processing node includes: at least one of data cleaning, data conversion, data filtering, formatting, and field configuration;

[0076] Step S24, determining the data loading configuration information according to the output component information and the data output path information.

[0077] Specifically, non-real-time data usually refers to data that is collected, processed, and analyzed after a period of time after the data is generated. This type of data comes from a wide range of sources, including historical records, offline data, archived information in traditional databases, and log data from service systems. Due to its diverse sources and large scale, non-real-time data presents complex and diversified characteristics. Therefore, efficient and systematic data processing processes are crucial to discovering the value of data. This solution uses the ETL Cloud data integration platform to achieve visual process control and task scheduling configuration for non-real-time data to optimize data processing efficiency and quality.

[0078] For example, in the graphical user interface of the ETL Cloud data integration platform, users can create new applications for specific ETL requirements of different scenarios and systems to facilitate differentiated management. Each application corresponds to a specific business line or system requirement, which helps to systematically manage and improve the maintainability and scalability of the application. This structured approach not only facilitates monitoring and tracking of ETL processes, but also effectively reduces the complexity of cross-system collaboration.

[0079] Exemplarily, users can use the ETL Cloud data integration platform to configure data extraction strategies for non-real-time data in advance. Specifically, in order to adapt to various subsequent data processing scenarios, you can first select and configure the database type and the corresponding configuration information to complete the configuration of the data source. The purpose of this step is to build a centralized and easy-to-manage data source directory to ensure the smoothness and efficiency of the ETL (data extraction, transformation, loading) process. Through unified data source management, resources can be easily called and managed according to needs to optimize the overall efficiency and flexibility of data processing. For example, the data source configuration information may include data source type (such as short connection or connection pool), data source classification, configuration instructions, data source ID, user ID, database driver and other parameters.

[0080] For example, you can select library table input, file input or API interface as the entry point for data extraction based on the data source type. Among them, library table input is generally used to extract data from the database; file input is suitable for reading data from local or remote file systems; API interface input is used to obtain data from external systems or services. The configuration information of each component must be filled in in detail, including library table configuration information, file path, API interface configuration and other key information to ensure the accuracy and effectiveness of data extraction.

[0081] In the data conversion phase, you can select and configure the data nodes to be processed according to business needs. This process allows data to be cleaned, converted, filtered, and other operations, such as formatting dates, creating new fields, removing duplicate data, etc. Through carefully designed conversion rules, the data will be adjusted to a format suitable for subsequent processing, improving the data's usability and analytical value.

[0082] In addition, the data loading phase involves selecting and configuring output components to output the processed data to a target location. This can be another database, data warehouse, data lake, or other storage system. The corresponding configuration information can include the target system name, data table or file path, etc., to ensure that the data can be loaded to the expected location accurately.

[0083] By integrating the above data extraction, transformation, and loading operations, a complete ETL process is constructed. Each step should be closely connected to form a seamless data flow path to ensure the continuity and efficiency of data processing.

[0084] In this exemplary embodiment, the non-real-time data processing process may include: 1) Data extraction: Using the data source connection capability of the ETL Cloud tool, non-real-time data can be extracted from various sources, including: historical records, data in offline databases, and service log information, and the required data can be obtained quickly and accurately. 2) Data conversion and cleaning: Based on the personalized data conversion rules pre-defined by the user, the extracted data is cleaned, formatted, merged, and other operations are performed to ensure data quality. The ETL Cloud tool supports a variety of data conversion components, such as date conversion, text processing, data format standardization, etc., to meet the diverse needs of data processing. 3) Data loading: The converted and cleaned data is loaded into the target data storage system, such as a data warehouse, data lake, etc., through the integration capabilities of the ETL Cloud tool, to prepare for subsequent analysis and application. 4) Task scheduling and automation: The ETL Cloud tool provides flexible task scheduling functions, which can set scheduled tasks according to business needs to realize the automation process of data processing and ensure the continuity and timeliness of data processing.

[0085] In this exemplary embodiment, the method further includes:

[0086] Step S31, opening a Binlog log file in a source database and adding corresponding configuration information, so as to configure the Binlog log file as a data source, and extracting the original data in real time from the data source;

[0087] Step S32: establishing, by using the Flink CDC tool, a first mapping table corresponding to the target data table of the database in the data warehouse and a second mapping table corresponding to the source database; and establishing a mapping relationship between the first mapping table and the second mapping table;

[0088] Step S33, performing data cleaning and / or conversion processing on the extracted raw data based on the data conversion configuration information of the Flink CDC tool;

[0089] Step S34: performing data synchronization processing on the raw data that has been cleaned and / or converted based on the mapping relationship.

[0090] For example, real-time data refers to data that is accessed and processed almost immediately after it is generated or collected. It is different from batch data in traditional data processing. It may be stored, transmitted, cleaned, converted, and other steps after the data is generated before it can be used for analysis and processing. The processing of real-time data requires the system to respond quickly to the generation of data and provide timely insights, decisions, and support. Real-time data can be processed using a streaming processing framework, which can perform complex data processing logic such as filtering, aggregation, and conversion. Due to Doris data's friendly support for Flink tools, this solution uses the Flink CDC framework to process real-time data, and synchronizes business change data to the Doris database in real time through Binlog.

[0091] For example, you can use the Flink CDC tool to build a Streaming ELT job from MySQL to the Doris database, which includes the functions of whole database synchronization, table structure change synchronization, and sub-database and sub-table synchronization. The real-time data processing flow may include: 1) Enable Binlog in the source database and add the configuration in MySQL: log_bin = mysql_bin binlog-format = Rowserver-id = 1. 2) Use Flink CDC to create a mapping table for the Doris target table in Flink and a mapping table for the source database table. 3) Use Flink's general conversion or custom functions to clean and convert data. 4) Submit the insertion task of the mapping table for data synchronization.

[0092] In this exemplary embodiment, reference Figure 3As shown, the architecture of the data warehouse 20 may include: an operational data storage layer (Operational Data Store, ODS) 201, a detail data layer (Data Warehouse Detail, DWD) 202, a summary data layer (Data Warehouse Summary, DWS) 203, an application data service layer (Application Data Service, ADS) 204, and a dimensional data mart layer (Dimensional Data Mart, DIM) 205.

[0093] In this exemplary embodiment, the method further includes:

[0094] Using Hive data warehouse tool to build the operational data storage layer based on Hadoop distributed structure;

[0095] The detailed data layer, summary data layer, and dimension data mart layer are constructed based on the Doris database.

[0096] In some exemplary embodiments, the method further includes: operating the data storage layer to perform preliminary data processing on the loaded original data, and outputting the preliminary processed original data to the detailed data layer; wherein the preliminary data processing includes: at least one of data cleaning, format conversion, and redundant information cleaning.

[0097] For example, the ODS layer uses Hive, which is based on the Hadoop distributed system and can store and process PB-level data and provide external structured SQL queries. The operational data storage layer is mainly used to store the original data of the capital system. This layer usually contains a large amount of original data, such as transaction data, account opening data, etc. The key goal of the operational data layer is to accurately capture all data generated during the operation of the system, and the structure is consistent with the source data. The naming rule is owd_database type_data name_incremental identifier, for example: odw_mysql_order_time.

[0098] For example, the ODS layer can convert the original data, such as performing preliminary cleaning and format conversion on the data at the ODS layer, removing redundant information and inconsistent data, and ensuring the consistency and accuracy of the data.

[0099] For example, for the table design of the ODS layer, A. create tables according to different data sources and data formats, such as data in Json and tsv formats. B. do not need to manually define the parsing syntax, retain the field name of the source data, and use the syntax of the parsing mapping of the Hive 3.x version, for example:

[0100] CREATE TABLE ods_text_order_time(account string,money double,timedate)

[0101] ROW FORMAT SERDE`org.apache.hadoop.hive.sarde2.JsonSerDe`

[0102] STORED AS TEXTFILE;

[0103] In step S12, the operational data storage layer outputs the original data to the detailed data layer; the detailed data layer performs data preprocessing on the original data to obtain basic data, and outputs the basic data to the summary data layer.

[0104] In this exemplary embodiment, the DWD layer data can be aggregated and processed according to business areas and specific scenarios to form a detailed business view. This process makes the data have higher analytical value and provides a solid foundation for subsequent decision support.

[0105] For example, the database of the DWD layer can be built based on Doris. The detailed data layer of the data warehouse performs data cleaning, verification and preliminary processing based on the ODS layer, and generates more detailed business indicators and data models according to the field. This is an important working layer in the data warehouse, which is mainly responsible for converting the original data into the data structure and dimensions required by the analysis system. It is necessary to process the data of the ODS layer with null values, verify finiteness, standardize data formats, unify data standards, transcode data, and clean business rules. The naming rule is dwd_field_data name, for example: dwd_fund_aoocunt.

[0106] In this exemplary embodiment, the detailed data layer performs data preprocessing on the original data to obtain basic data, including: constructing a corresponding data table model based on the target field corresponding to the data analysis object of the target business and the data type corresponding to the target field; and preprocessing the original data based on the data table model to obtain the basic data.

[0107] For example, a data table model is constructed with account information as the theme. As shown in Table 1, the corresponding target fields may include keywords such as account ID, account name, account type, and account balance.

[0108] Table 1

[0109]

[0110] The corresponding table creation statements may include:

[0111]

[0112]

[0113] For example, a data table model is established with account opening information as the theme. As shown in Table 2, the corresponding data types may include: currency code, account opening date, account opening bank, account opening address, account status and other information.

[0114] Table 2

[0115]

[0116] The corresponding table creation statements may include:

[0117]

[0118]

[0119] For example, when constructing a data table model based on transaction flow information, referring to Table 3, the corresponding target fields may include: associated account ID, transaction date, delivery time, creation time, transaction source and other keywords.

[0120] Table 3

[0121]

[0122]

[0123] Correspondingly, the table creation statement can include:

[0124]

[0125] In step S13, the summary data layer aggregates the basic data according to the current data processing task corresponding to the target business, obtains the structured data corresponding to the target business, and outputs it to the application data service layer;

[0126] In this exemplary embodiment, the summary data layer can aggregate and summarize the data in the DWD layer by field according to business needs to form a wide table, thereby improving data analysis performance. The wide table of this layer is generated by aggregation and grouping operations of fact tables and dimension tables. It is necessary to roll up the dimensions, summarize indicators, and merge indicators for the data in the DWD layer. The naming rule is dws_field_data name, for example: dws_fund_day.

[0127] For example, the materialized view of the Doris database can be used to implement the aggregation and analysis functions of the DWS layer. The materialized view of Doris is a data storage structure that pre-calculates and stores a static copy of the view results. This design pattern can help improve data query performance, especially when the same data set needs to be accessed frequently, because accessing the materialized view is faster than dynamically recalculating the view.

[0128] For example, to analyze the geographical distribution of account transactions, when building a materialized view, the corresponding statements may include:

[0129]

[0130] In addition, the SQL used includes: SELECT transaction_location, COUNT(*)num FROM dwd_fund_transactions GROUP BY transaction_location;, which automatically calls the materialized view

[0131] In step S14, the application data service layer performs data analysis on the structured data according to the business configuration information corresponding to the current data processing task, and outputs corresponding data analysis results.

[0132] In this exemplary embodiment, the DIM layer can be constructed using the Doris database. The application data service layer can be used to store system or program related data, such as configuration information, log data, interactive business data; interface design and security management are required, and the naming rule is ads_data name, for example: ads_config.

[0133] For example, a data table model may be established for the area information (dim_area_info), as shown in Table 4, and the corresponding fields may include: area code, area name, area level, and the like.

[0134] Table 4

[0135]

[0136]

[0137] The corresponding table creation statements may include:

[0138]

[0139] For example, a data table model may be established for currency information (dim_currency_info), as shown in Table 5, and corresponding fields may include: currency code, currency name, currency symbol, level, etc.

[0140] Table 5

[0141]

[0142] Correspondingly, the table creation statement can include:

[0143]

[0144]

[0145] For example, a data table model can be established for institutional information, as shown in Table 6. The corresponding asset management can include keywords such as institution code, name, and type.

[0146] Table 6

[0147]

[0148] Correspondingly, the table creation statement can include:

[0149]

[0150]

[0151] For example, a corresponding data table model may be established based on bank information, as shown in Table 7, where the corresponding fields may include keywords such as bank code, bank name, address, and telephone number.

[0152] Table 7

[0153]

[0154] Correspondingly, the table creation statement can include:

[0155]

[0156] In this exemplary embodiment, the method further includes: constructing a dimensional data mart layer based on the data detail layer, and storing public data corresponding to the target business in the dimensional data mart layer; wherein the public data can be reused in the application data service layer.

[0157] Exemplary, reference Figure 3As shown in the figure, the dimensional data mart layer is built on the basis of the DWD layer and is used to store dimensional data. It can be reused in data processing and analysis processes to improve efficiency and improve business scenarios. The naming rule is dim_dimension name, for example: ads_bank_info. The public data generated in the ETL process, such as basic entities and metadata, is processed by the ETLCloud platform and output to the DIM (dimensional data mart) layer. As a public data resource library, the DIM layer provides a unified and reusable data model for the entire system.

[0158] In this exemplary embodiment, the method further includes:

[0159] Step S41, receiving a data query request;

[0160] Step S42, converting the data query request into a structured query statement and performing decomposition processing to obtain multiple sub-query statements;

[0161] Step S43: distribute the multiple sub-query statements to the distributed storage nodes corresponding to the data warehouse, so that each of the distributed storage nodes processes the sub-query statements.

[0162] Exemplarily, for the data query request for the target business currently received from the user, the data query request can be decomposed into multiple sub-query requests according to the keyword field corresponding to the data query request and the index, and the corresponding sub-query statement is distributed to the corresponding storage node at the ODS layer. By dispersing and storing data in different database instances, the uniform distribution of data is achieved, and the single-point pressure is reduced, thereby improving the system processing capacity and response speed. By reasonably setting the database index, the index is optimized, and then the query performance is optimized, the data reading time is reduced, and the query efficiency is improved. By deploying a distributed database system, the data query request is distributed to multiple nodes. For example, the data query request is decomposed into multiple sub-requests according to the index, which not only improves the concurrent processing capability of the database, but also enhances the robustness and reliability of the system, and realizes flexible horizontal expansion.

[0163] For example, the ODS layer data is configured not to be open to the outside world. The ODS layer data flow is input to the DWD layer, the DWD layer data flow is now input to the DWS layer, and the DWS layer data flow is to the ADS layer, which can effectively prevent data pollution and ensure data security. In addition, when managing permissions, permissions can be configured in ETLCloud, Flink, Doris, and report generation tools. Permission settings can also be reviewed and updated regularly to ensure security and compliance.

[0164] The data analysis method provided in the embodiment of the present disclosure can be executed on the server side, or by the user-side intelligent terminal device and the server side in collaboration. It can be applied to data analysis tasks in various industries, can formulate a complete ETL process for capital data based on a visual interface, provide non-real-time massive data import, and ensure the synchronous update of real-time data. The system designed by this solution can achieve the following technical effects in the field of capital data processing and analysis:

[0165] Efficient data processing and analysis: By combining ETL Cloud and Flink CDC tools, the system can achieve real-time extraction, rapid conversion, efficient loading of non-real-time data and real-number data, as well as efficient storage based on Doris database, ensuring high efficiency of data processing and real-time data analysis. Especially for the processing of large amounts of financial data, this solution significantly improves the data processing speed through optimized data processing processes, greatly shortening the cycle from data collection to analysis.

[0166] Automated data management: The system can automatically perform tasks such as data extraction, conversion, and loading, reducing the pressure of manual operations and improving the automation level of data processing. At the same time, through the visual interface of the ETL Cloud tool, users can easily design and manage data processing processes, improving the flexibility and maintainability of data processing.

[0167] Reliable data quality: By introducing quality control mechanisms in all aspects of data processing, including data extraction, conversion, loading and other steps, the system can effectively avoid data pollution and errors, ensuring the quality of data input into the Doris database. This not only improves the accuracy of data analysis, but also reduces the risk of problems in subsequent analysis.

[0168] Flexible data access and analysis: By building materialized views in the Doris database, the system provides fast data access for different business scenarios. The construction of materialized views speeds up data query, allowing data analysts to obtain required data more quickly and conveniently, supporting ad hoc analysis and real-time decision-making.

[0169] Enhanced data security and compliance: Strict data security and permission management mechanisms are taken into consideration in the system design. By defining user roles and permissions, data access control is achieved, ensuring data security and complying with the data protection and compliance requirements of the financial industry.

[0170] Scalability and extensibility: With the high scalability of the distributed computing framework Flink and the Doris database, the system can flexibly expand as the business scale expands, support future data growth needs, and reduce system maintenance costs.

[0171] The system designed by this solution has broad application prospects in the field of financial data processing and analysis. The following are specific potential application scenarios and advantages:

[0172] Real-time fund flow monitoring: Flink CDC (Change Data Capture) technology can be used to monitor and analyze fund flows in real time, helping financial institutions to promptly identify potential risk points, such as abnormal fund flows and money laundering activities, and enhance risk warning capabilities.

[0173] Personalized financial product recommendations: Through automated data extraction, conversion, and loading processes, the system can quickly integrate multi-dimensional data such as users’ transaction history, risk preferences, market performance, etc., to provide users with personalized investment advice or financial product recommendations, thereby improving customer satisfaction and product conversion rates.

[0174] Risk assessment and credit rating: The system can efficiently process and analyze capital flow data, and combine historical credit records and other relevant data to conduct real-time or regular risk assessment and credit rating for customers, assisting financial institutions in implementing more accurate risk control strategies.

[0175] Market dynamics analysis: By building a materialized view based on the Doris database, the system can provide market analysts with fast and intuitive market dynamics analysis tools, including trading volume trends, hot investment areas analysis, etc., to help financial institutions develop more informed investment strategies.

[0176] Compliance and audit support: Automated data processing processes ensure data consistency and integrity, helping financial institutions better meet regulatory requirements, simplify the process of generating compliance reports, reduce manual errors, and improve audit efficiency.

[0177] Data-driven business decisions: Through the integrated analysis of real-time and historical data, the system can support data-driven business decisions, such as optimizing capital allocation, adjusting product strategies, predicting market trends, etc., thereby improving overall business operation efficiency and competitiveness.

[0178] Cross-departmental collaboration and sharing: For large financial groups, the system can serve as a data sharing platform to promote the flow of information between different departments, such as data exchange between risk control departments and marketing departments, to achieve more comprehensive business support.

[0179] Insight into potential business opportunities: Through in-depth data analysis, the system can discover potential business opportunities in fund management, such as identifying high-growth markets, under-explored customer groups, optimizing cash flow strategies, etc., providing strong data support for business development.

[0180] In short, by integrating advanced data processing technologies, the solution builds an efficient, automated and scalable system that can provide comprehensive support in the field of fund data processing and analysis, helping financial institutions improve business efficiency, enhance decision-making capabilities, optimize risk management, and gain an advantage in competition. With the growing demand for data-driven decision-making in the financial industry, the system has extremely high application value and market prospects.

[0181] This method optimizes the data base and storage: Doris database is used as the data base, which has the characteristics of column storage and distributed architecture, and can effectively improve query performance and reduce storage space requirements. Through the combination of OLTP and OLAP, Doris can adapt to the dual needs of online business services and offline data analysis, and provide stable and efficient data support for capital analysis. The ETLCloud platform realizes automatic data extraction, conversion and storage, provides intuitive interfaces and components, simplifies the data processing process, and reduces learning costs. The platform supports the access of multiple data sources, including files, databases and APIs, and ensures the efficiency and flexibility of data processing through process design and task scheduling. Real-time data access and consistency: Flink CDC technology is used to capture real-time data of the business system, ensure data consistency and real-time, and provide the latest and accurate data source for capital analysis. Implement data hierarchical management strategies, establish a strict quality control system, ensure data accuracy and integrity, effectively prevent data pollution and leakage, and enhance the credibility of data assets. Clarify user roles and permission allocation, such as data reading, writing and analysis permissions, to ensure data security and compliance, while improving the flexibility and efficiency of data use.

[0182] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0183] For further reference, Figure 4 As shown, in the embodiment of this example, a data analysis system 40 is also provided, including:

[0184] The data extraction module 401 is used to extract the original data corresponding to the target business from the target data source according to the data extraction strategy preconfigured for the target business, and load the original data into the operational data storage layer of the data warehouse for storage;

[0185] The data warehouse 402 includes: an operation data storage layer, a detailed data layer, a summary data layer, and a data service layer, wherein the operation data storage layer outputs the original data to the detailed data layer; the detailed data layer performs data preprocessing on the original data to obtain basic data, and outputs the basic data to the summary data layer; the summary data layer aggregates the basic data according to the current data processing task corresponding to the target business, obtains the structured data corresponding to the target business, and outputs it to the application data service layer; the application data service layer performs data analysis on the structured data according to the business configuration information corresponding to the current data processing task, and outputs the corresponding data analysis results

[0186] In some exemplary embodiments, the data warehouse further includes: a dimensional data mart layer, wherein the dimensional data mart layer is constructed based on the data detail layer, and the public data corresponding to the target business is stored in the dimensional data mart layer; wherein the public data can be reused in the application data service layer.

[0187] Exemplarily, the operational data storage layer is mainly used to store raw data, such as raw data of the capital system. For example, raw data can be transaction data, account opening data, etc. The key target data of the operational data storage layer is accumulated, accurately capturing all data generated during the operation of the system, and the structure is consistent with the source data. The naming rule is owd_database type_data name_incremental identifier, for example: odw_mysql_order_time.

[0188] The detailed data layer cleans, verifies and preliminarily processes the data based on the ODS layer data, and generates more detailed business indicators and data models according to the domain. The detailed data layer is an important working layer in the data warehouse. It is mainly responsible for converting the original data into the data structure and dimension required by the analysis system. It is necessary to handle the null value of the ODS layer data, verify the finiteness, standardize the data format, unify the data standard, transcode the data, and clean the business rules. The naming rule is dwd_domain_data name, for example: dwd_fund_aoocunt. Among them, the detailed data layer is built based on the Doris database, and the materialized view of the Doris database is used to implement the aggregation and analysis functions of the DWS layer. The materialized view of Doris is a data storage structure that can pre-calculate and store a static copy of the view results. This design pattern can help improve data query performance, especially when the same data set needs to be accessed frequently, because accessing the materialized view is faster than dynamically recalculating the view.

[0189] The summary data layer aggregates the data in the DWD layer by field through aggregation and aggregation according to business needs to form a wide table, thereby improving data analysis performance. The wide table of this layer is generated by aggregation and grouping operations of fact tables and dimension tables. The data of the DWD layer needs to be rolled up in dimensions, indicators are aggregated, and indicators are merged. The naming rule is dws_field_data name, for example: dws_fund_day. Among them, the summary data layer is built based on the Doris database architecture.

[0190] The application data layer is used to store system or program related data, such as configuration information, log data, interactive business data; interface design and security management are required, and the naming rule is ads_data name, for example: ads_config. The application data layer uses Hive, based on the Hadoop distributed system, which can store and process PB-level data and provide external structured SQL queries.

[0191] The dimension data mart layer is built on the basis of the DWD layer and is used to store dimensional data. It can be reused in the data processing and analysis process to improve efficiency and improve business scenarios. The naming rule is dim_dimension name, for example: ads_bank_info. Among them, the dimension data mart layer is built with the Doris data architecture. The data of the dimension data mart layer is relatively large and is stored in Doris partitions and buckets. There are two main advantages: First, the Doris database architecture is column storage, which can compress the occupied space. Second, it is convenient for data processing and dimension aggregation analysis of the data in the data warehouse layer DW. Among them, the DW layer is mainly the DWD layer and the DWS layer.

[0192] Figure 5 A schematic diagram of an electronic device suitable for implementing an embodiment of the present invention is shown.

[0193] It should be noted that Figure 5 The electronic device 1000 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0194] like Figure 5As shown, the electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, ROM 1002 and RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0195] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0196] In particular, according to an embodiment of the present invention, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0197] Specifically, the electronic device may be a smart mobile electronic device such as a mobile phone, a tablet computer or a laptop computer, or may be a smart electronic device such as a desktop computer, or a smart wearable device.

[0198] It should be noted that the storage medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any storage medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0199] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0200] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0201] It should be noted that, as another aspect, the present application also provides a storage medium, which can be included in an electronic device; or it can exist independently without being installed in the electronic device. The above storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement the following Figure 1 The individual steps of the method are shown.

[0202] In one embodiment, the present application provides a computer program product, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0203] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0204] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0205] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data analysis method, characterized in that: The method comprises: According to the data extraction strategy preconfigured for the target business, the original data corresponding to the target business is extracted from the target data source, and the original data is loaded into the operational data storage layer of the data warehouse and stored; The operation data storage layer outputs the raw data to the detailed data layer; the detailed data layer performs data preprocessing on the raw data to obtain basic data, and outputs the basic data to the summary data layer; The summary data layer aggregates the basic data according to the current data processing task corresponding to the target business, obtains the structured data corresponding to the target business, and outputs it to the application data service layer; The application data service layer performs data analysis on the structured data according to the business configuration information corresponding to the current data processing task, and outputs corresponding data analysis results.

2. The method according to claim 1, characterized in that The method further comprises: A dimensional data mart layer is constructed based on the data detail layer, and public data corresponding to the target business is stored in the dimensional data mart layer; wherein the public data can be reused in the application data service layer.

3. The method according to claim 1, characterized in that The method further comprises: Using Hive data warehouse tool to build the operational data storage layer based on Hadoop distributed structure; The detailed data layer, summary data layer, and dimension data mart layer are constructed based on the Doris database.

4. The method according to claim 1, characterized in that: The original data includes non-real-time data and real-time data corresponding to the target business; The extracting the original data corresponding to the target business from the target data source according to the data extraction strategy preconfigured for the target business includes: Calling the ETL Cloud tool to obtain the non-real-time data from the target data source according to the data extraction strategy; and The Flink CDC tool is called to extract the real-time data according to the data extraction strategy; the target data source includes business data.

5. The method according to claim 4, characterized in that The data extraction strategy corresponding to the non-real-time data includes customized: data source directory, data conversion configuration information, data loading configuration information; The method further comprises: In response to an application creation operation inputted in the graphical user interface, an ELT process corresponding to the target service is created; Configure a data source directory according to the input data source type and data source configuration information, so as to extract data according to the data source directory; At least one data processing node is configured according to the data requirements corresponding to the target business, so as to determine the data conversion configuration information according to the data processing node; wherein the data processing node includes: at least one of data cleaning, data conversion, data filtering, formatting, and field configuration; The data loading configuration information is determined according to the output component information and the data output path information.

6. The method according to claim 4, characterized in that The method further comprises: Open the Binlog log file in the source database and add corresponding configuration information to configure the Binlog log file as a data source, and extract the original data in real time from the data source; Using the Flink CDC tool, establish a first mapping table corresponding to the target data table of the database in the data warehouse and a second mapping table corresponding to the source database; and establish a mapping relationship between the first mapping table and the second mapping table; Performing data cleaning and / or conversion processing on the extracted raw data based on the data conversion configuration information of the Flink CDC tool; The original data processed by data cleaning and / or conversion is synchronously processed based on the mapping relationship.

7. The method according to claim 1, characterized in that The method further comprises: Receive data query requests; Convert the data query request into a structured query statement and decompose it to obtain multiple sub-query statements; The multiple sub-query statements are distributed to the distributed storage nodes corresponding to the data warehouse, so that each of the distributed storage nodes processes the sub-query statements.

8. The method according to claim 1, characterized in that The detailed data layer performs data preprocessing on the original data to obtain basic data, including: Constructing a corresponding data table model according to the target field corresponding to the data analysis object of the target business and the data type corresponding to the target field; The original data is preprocessed based on the data table model to obtain the basic data.

9. A data analysis system, characterized in that: include: A data extraction module is used to extract the original data corresponding to the target business from the target data source according to the data extraction strategy preconfigured for the target business, and load the original data into the operational data storage layer of the data warehouse for storage; A data warehouse, used for the operation data storage layer to output the raw data to the detail data layer; The detailed data layer performs data preprocessing on the original data to obtain basic data, and outputs the basic data to the summary data layer; The summary data layer aggregates the basic data according to the current data processing task corresponding to the target business, obtains the structured data corresponding to the target business, and outputs it to the application data service layer; The application data service layer performs data analysis on the structured data according to the business configuration information corresponding to the current data processing task, and outputs corresponding data analysis results.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data analysis method according to any one of claims 1 to 8 is implemented.

Images