Business data processing method, device and data center

Abstract

The embodiments of the present application provide a business data processing method, device and data middle platform, which can be used in the field of big data technology. The method includes: obtaining target data processing rules and at least one type of target business field corresponding to the target data retrieval request for the distributed system; extracting the business data corresponding to each of the target business fields from the distributed system; performing data processing on the business data corresponding to each of the target business fields based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request. The present application can effectively improve the comprehensiveness and reliability of querying or retrieving data in a distributed database, and can improve the efficiency and automation of the data retrieval service provided by the distributed database.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, in particular to the field of big data technology, and specifically to a business data processing method, device and data middle platform. Background Art

[0002] With the rapid development of science and technology, more and more companies are beginning to adopt distributed systems such as Hadoop clusters to digitally manage business data. Taking commercial banks as an example, since commercial banks involve a large amount of business data in multiple business processes, distributed systems can be used to manage these business data. When it is necessary to query or retrieve certain business data, the data can be obtained by accessing the distributed system.

[0003] However, in the business data processing process of existing distributed systems, it is usually limited to a single business process, ignoring the related data between multiple businesses, and therefore lacks a deep understanding of the data. In the big data system of commercial banks, the increasingly diversified business types have caused the duplication of systems and applications, and there is a waste of data, computing and human resources. Barriers between business systems also lead to the emergence of data islands, making it difficult to plan internal and external data globally. For example, if a user wants to extract customer asset data, he can only manually analyze the deposit data and liability data involved in advance, and then retrieve the corresponding data from the data storage locations corresponding to the deposit data and liability data. This method obviously has problems such as low efficiency, poor reliability and lack of comprehensiveness in data query or retrieval. Summary of the invention

[0004] In response to the problems in the prior art, the present application provides a business data processing method, device and data middle platform, which can effectively improve the comprehensiveness and reliability of querying or retrieving data in a distributed database, and can improve the efficiency and automation of data retrieval services provided by the distributed database.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides a business data processing method, comprising:

[0007] Obtaining a target data processing rule and at least one type of target business domain corresponding to a target data retrieval request of a distributed system;

[0008] Extracting business data corresponding to each of the target business areas from the distributed system;

[0009] Based on the target data processing rules, data processing is performed on the business data corresponding to each of the target business areas to obtain the visualized target data corresponding to the target data retrieval request.

[0010] Furthermore, the step of obtaining target data processing rules and at least one type of target business domain corresponding to the target data retrieval request of the distributed system includes:

[0011] Based on the business object application, the target data retrieval request for the distributed system sent by the business front end is received;

[0012] Obtaining a target data processing rule corresponding to the target data retrieval request;

[0013] And, it is determined whether the content of the target data retrieval request corresponds to a preset target indicator. If so, at least one target business field corresponding to the target indicator is obtained from a unified indicator library preset in the distributed system.

[0014] Furthermore, extracting the business data corresponding to each of the target business areas from the distributed system includes:

[0015] It is searched from each business domain subject table of the data warehouse preset in the distributed system whether the business data corresponding to each of the target business domains are included; if so, the business data corresponding to each of the target business domains are extracted from the data warehouse.

[0016] Furthermore, it also includes:

[0017] If the data warehouse does not contain the business data corresponding to each of the target business fields, extracting the business data corresponding to each of the target business fields from the data lake preset in the distributed system;

[0018] Among them, each of the business domain subject tables in the data warehouse is pre-processed based on the business source data in the data lake.

[0019] Furthermore, it also includes:

[0020] The business data corresponding to each business system is stored in a data lake preset in the distributed system;

[0021] Process the business data in the data lake to obtain subject tables for various business fields and a unified indicator library;

[0022] Each of the business domain subject tables is stored in a preset data warehouse in the distributed system.

[0023] Furthermore, the business domain subject table is used to store the corresponding relationship between the aggregate primary key corresponding to each business data and each attribute information used as the business domain, wherein the aggregate primary key is information used to uniquely identify the corresponding business data;

[0024] The unified indicator library is used to store a one-to-one or one-to-many correspondence between each indicator and each business field.

[0025] In a second aspect, the present application provides a business data processing device, including:

[0026] A request reading module, used to obtain target data processing rules and at least one type of target business domain corresponding to a target data retrieval request for a distributed system;

[0027] A data extraction module, used to extract business data corresponding to each of the target business areas from the distributed system;

[0028] The data processing module is used to process the business data corresponding to each of the target business areas based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request.

[0029] In a third aspect, the present application provides a data middle platform, including: a source data loading layer, a data indicator processing layer, a business object application layer and a value realization service layer connected in sequence;

[0030] The business object application layer includes various types of business object applications, and the business object application layer is used to implement the business data processing method. The business object application layer extracts the business data corresponding to each of the target business fields from the data indicator processing layer or the source data loading layer;

[0031] The source data loading layer includes: a data lake, which is used to load and store the business data corresponding to each business system;

[0032] The data indicator processing layer includes: a data warehouse and a unified indicator library; the data warehouse is used to store the subject tables of each business field obtained by processing the business data in the data lake; the one-to-one or one-to-many correspondence between each indicator in the unified indicator library and each business field;

[0033] The value realization service layer includes: various types of business front desks, and each business front desk corresponds to each of the business object applications one by one.

[0034] In a fourth aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the business data processing method when executing the program.

[0035] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the business data processing method when executed by a processor.

[0036] It can be seen from the above technical scheme that the present application provides a business data processing method, device and data middle platform, the method comprising: obtaining target data processing rules and at least one type of target business field corresponding to a target data retrieval request for a distributed system; extracting business data corresponding to each of the target business fields from the distributed system; performing data processing on the business data corresponding to each of the target business fields based on the target data processing rules to obtain visualized target data corresponding to the target data retrieval request. By obtaining the target business field corresponding to the target data retrieval request for a distributed database, and extracting at least one type of pre-stored business data corresponding to the target business field from the distributed database, the barriers between business systems can be broken based on the target business field and the business field subject table for pre-stored business data. It avoids the formation of data islands and can make full use of the accumulated large amount of data assets to effectively improve the comprehensiveness and reliability of data retrieval in distributed databases; by obtaining the target data processing rules corresponding to the target data retrieval request for the distributed database, and performing data processing on the business data corresponding to the target business field based on the target data processing rules, so as to obtain the visualized target data corresponding to the target data retrieval request, so as to obtain the visualized target data corresponding to the target data retrieval request, the efficiency and reliability of the data retrieval service can be effectively improved, and the corresponding target data can be displayed quickly, accurately and highly automatically for the target data retrieval requests put forward by different types of applications corresponding to different types of business front desks, thereby effectively improving the efficiency and accuracy of data retrieval and effectively improving the user experience of business front desk operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is a structural diagram of the data middle platform in the embodiment of the present application.

[0039] Figure 2 It is a first flow chart of the business data processing method in the embodiment of the present application.

[0040] Figure 3 It is a flowchart of step 100 in the business data processing method in an embodiment of the present application.

[0041] Figure 4 It is a first flow chart of step 200 in the business data processing method in the embodiment of the present application.

[0042] Figure 5 It is a second flow chart of step 200 in the business data processing method in the embodiment of the present application.

[0043] Figure 6 It is a second flow chart of the business data processing method in an embodiment of the present application.

[0044] Figure 7 It is a structural diagram of the business data processing device in an embodiment of the present application.

[0045] Figure 8 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] It should be noted that the business data processing method, device and data middle platform disclosed in the present application can be used in the field of big data technology, and can also be used in any field except the field of big data technology. The application field of the business data processing method, device and data middle platform disclosed in the present application is not limited.

[0048] Taking commercial banks as an example, traditional digital banking construction is often limited to a single business process, ignoring the related data of multiple businesses and lacking a deep understanding of data. In the big data system of commercial banks, the increasingly diversified business types have led to the duplication of systems and applications, and there is a waste of data, computing and human resources. Barriers between business systems also lead to the emergence of data islands, making it difficult to plan internal and external data globally. At the same time, the definitions of indicators with the same name vary in different business areas, which can easily lead to data chimneys. The above situation has caused the low efficiency of data use in commercial banks, the large amount of accumulated data assets have not been fully utilized, and the barriers between business segments have not been broken. Therefore, it is urgent to deeply explore the connections between data, realize greater value realization of data, and empower business scenarios.

[0049] Based on the above content, one embodiment of the present application considers building a data middle platform under the commercial banking system, so as to realize the all-round utilization of banking business data, data connectivity in all business fields, and sharing of unified data standard services, meet the data application needs of various business fields, quickly support business scenario innovation, and realize data value realization.

[0050] The data middle platform mentioned in one or more embodiments of the present application is to build a standard data asset system by precipitating common data between businesses, formulating unified data standards and quality systems, and realizing the connectivity of data in various business fields, providing convenient and fast data support for front-end businesses, and meeting the data needs of enterprise businesses. The data middle platform is also based on the enterprise's unique business model and organizational structure, supported by tangible products and implementation methodologies, to build a mechanism that continuously turns data into assets and serves the business.

[0051] In response to the problems of low efficiency, poor reliability and incompleteness in data query or retrieval in existing business data processing methods, the embodiments of the present application respectively provide a business data processing method, a business data processing device, a data middle platform, and a computer-readable storage medium for an electronic device. By obtaining the target business field corresponding to the target data retrieval request for a distributed database, and extracting at least one type of pre-stored business data corresponding to the target business field from the distributed database, it is possible to break down the barriers between business systems based on the target business field and the business field subject table for pre-stored business data, avoid the formation of data islands, and make full use of the large amount of accumulated data assets, effectively improving the comprehensiveness of retrieval of data in distributed databases. It improves the surface quality and reliability of data retrieval services by obtaining the target data processing rules corresponding to the target data retrieval request for the distributed database, and performing data processing on the business data corresponding to the target business field based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request. It can effectively improve the efficiency and reliability of data retrieval services, and can quickly, accurately and highly automatically display the corresponding target data for target data retrieval requests made by different types of applications corresponding to different types of business front desks, thereby effectively improving the efficiency and accuracy of data retrieval and effectively improving the user experience of business front desk operators.

[0052] In one or more embodiments of the present application, the distributed system may refer to a big data service cloud or an HBase cluster, etc., wherein the HBase cluster (Hadoop Database) refers to a high-reliability, high-performance, column-oriented, scalable distributed storage system, consisting of a master node Master and a slave node Region Server, wherein the Master can also be specifically written as HMaster, and the Region Server can also be written as HRegionServer or RegionServer, etc.

[0053] Based on the above content, the present application also provides a data center for implementing the business data processing method provided in one or more embodiments of the present application, see Figure 1The data middle platform includes a sequentially connected source data loading layer, a data indicator processing layer, a business object application layer and a value realization service layer; the business object application layer includes various types of business object applications (such as personal applications, risk applications, management applications and corporate applications, etc.), and the business object application layer can be a specific implementation of the business data processing device mentioned in one or more embodiments of the present application, used to implement the business data processing method mentioned in one or more embodiments of the present application, and the business object application layer extracts each of the target business fields (A business system, B business system) from the data indicator processing layer or the source data loading layer from the data source layer. The data loading layer includes: a data lake, which is used to load and store the business data corresponding to each business system; the data indicator processing layer includes: a data warehouse and a unified indicator library; the data warehouse is used to store the subject tables of each business field obtained by processing the business data in the data lake; the one-to-one or one-to-many correspondence between each indicator of the unified indicator library and each business field; the value realization service layer includes: various types of business front desks, and each business front desk corresponds to each of the business object applications. It can be understood that the business data processing device can be a server that is communicatively connected to other servers and / or client devices.

[0054] It is understandable that the client device may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0055] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0056] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0057] The details are described in detail through the following embodiments and application examples.

[0058] In order to solve the problems of low efficiency, poor reliability and incompleteness in data query or retrieval in the existing business data processing methods, the present application provides an embodiment of a business data processing method, see Figure 2 The business data processing method executed by the business data processing device specifically includes the following contents:

[0059] Step 100: Obtain target data processing rules and at least one type of target business domain corresponding to a target data retrieval request for a distributed system.

[0060] In step 100, the specific method of obtaining the target data processing rule corresponding to the target data retrieval request for the distributed system can be to directly read the target data processing rule specified by the target data retrieval request from the request content corresponding to the target data retrieval request, or to identify the processing keyword from the request content corresponding to the target data retrieval request based on a preset processing keyword identification method, and select the target data processing rule corresponding to the keyword from the correspondence between the pre-stored processing keyword and the data processing rule.

[0061] In step 100, the method for obtaining at least one type of target business domain corresponding to the target data retrieval request for the distributed system may be to obtain the business domain keyword from the request content corresponding to the target data retrieval request based on a preset business domain keyword identification method, and select the target business domain corresponding to the business domain keyword from the correspondence between the pre-stored business domain keyword and the business domain. Alternatively, the indicator keyword may be obtained from the request content corresponding to the target data retrieval request based on a preset indicator keyword identification method, and at least one target business domain corresponding to the indicator keyword may be searched in a pre-stored unified indicator library.

[0062] For example, if the request content corresponding to the target data retrieval request is a request to obtain the customer's monthly average assets, then the indicator keyword (i.e., the target indicator) is obtained as "assets" from the monthly average assets, and the two target business areas "deposits" and "liabilities" corresponding to the indicator keyword are searched in the pre-stored unified indicator library, and the processing keyword "monthly average" is obtained from the request content, then the target data processing rule corresponding to "monthly average" is selected from the correspondence between the pre-stored processing keywords and the data processing rules as the monthly average solution algorithm.

[0063] Step 200: extracting business data corresponding to each of the target business areas from the distributed system.

[0064] Step 300: Based on the target data processing rules, data processing is performed on the business data corresponding to each of the target business areas to obtain the visualized target data corresponding to the target data retrieval request.

[0065] From the above description, it can be seen that the business data processing method provided in the embodiment of the present application, by obtaining the target business field corresponding to the target data retrieval request for the distributed database, and extracting at least one type of pre-stored business data corresponding to the target business field from the distributed database, can break the barriers between business systems based on the target business field and the business field subject table that pre-stores the business data, avoid the formation of data islands, and make full use of the accumulated large amount of data assets to effectively improve the comprehensiveness and reliability of the data retrieved from the distributed database; by obtaining the target data processing rules corresponding to the target data retrieval request for the distributed database, and performing data processing on the business data corresponding to the target business field based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request, to obtain the visualized target data corresponding to the target data retrieval request, the efficiency and reliability of the data retrieval service can be effectively improved, and the corresponding target data can be displayed quickly, accurately and highly automatically for the target data retrieval requests proposed by different types of applications corresponding to different types of business front desks, thereby effectively improving the efficiency and accuracy of data retrieval, and effectively improving the user experience of business front desk operators.

[0066] In order to improve the comprehensiveness and reliability of the business data corresponding to the target business field, an embodiment of the business data processing method provided in this application is provided. Figure 3 , step 100 in the business data processing method specifically includes the following contents:

[0067] Step 110: Receive a target data retrieval request for a distributed system sent by a business frontend based on a business object application.

[0068] Step 120: Obtain the target data processing rule corresponding to the target data retrieval request.

[0069] Step 130: Determine whether the content of the target data retrieval request corresponds to a preset target indicator. If yes, obtain at least one target business field corresponding to the target indicator from a unified indicator library preset in the distributed system.

[0070] Specifically, the business object applications include various service applications, such as customer management systems, employee performance systems, risk identification systems, etc. According to the needs, data is first obtained from the data warehouse and indicator processing applications for further analysis and processing; if the indicator processing layer does not have the required data, data can be directly obtained from the data lake for processing. Visual analysis of data assets can be achieved, empowering enterprise data operations.

[0071] For example, in a customer portrait application, the basic information of the corresponding customer can be obtained from the aggregate table of the data warehouse. The rest of the data can be extracted and loaded in the data lake and data warehouse according to the functions you want to achieve. For example, if the indicator defines the customer's monthly average assets as assets = deposits + liabilities, then according to this principle, data can be extracted from the tables in the data lake and data warehouse that record deposit and liability information, and then logical processing can be performed based on the "monthly average" required to obtain the target data.

[0072] From the above description, it can be seen that the business data processing method provided in the embodiment of the present application can effectively improve the comprehensiveness and reliability of the business data corresponding to the target business field by obtaining at least one of the target business fields corresponding to the target indicator from the unified indicator library preset in the distributed system. Based on the indicators, it is possible to obtain the correlation between different types of business data, thereby further breaking down the barriers between different business systems.

[0073] In order to improve the convenience and efficiency of obtaining business data, an embodiment of the business data processing method provided in this application is shown in FIG. Figure 4 , step 200 in the business data processing method specifically includes the following contents:

[0074] Step 210: Search each business domain subject table of the data warehouse preset in the distributed system to see whether it contains business data corresponding to each of the target business domains. If so, execute step 220.

[0075] Step 220: extracting business data corresponding to each of the target business areas from the data warehouse.

[0076] From the above description, it can be seen that the business data processing method provided in the embodiment of the present application can effectively improve the convenience and efficiency of obtaining business data by obtaining the business data corresponding to each target business field corresponding to the target indicator from the business field subject table in the data warehouse, thereby further improving the efficiency of business data retrieval.

[0077] In order to improve the reliability, comprehensiveness and wide applicability of business data extraction, an embodiment of the business data processing method provided in this application is as follows: Figure 5 , step 200 in the business data processing method further specifically includes the following contents:

[0078] If the business data corresponding to each of the target business fields is not found in each of the business field subject tables, step 230 is executed: if the data warehouse does not contain the business data corresponding to each of the target business fields, the business data corresponding to each of the target business fields is extracted from the data lake preset in the distributed system; wherein each of the business field subject tables in the data warehouse is pre-processed based on the business source data in the data lake.

[0079] From the above description, it can be seen that the business data processing method provided in the embodiment of the present application can effectively improve the reliability, comprehensiveness and wide applicability of business data extraction by extracting the business data corresponding to each of the target business fields in the data lake preset in the distributed system when the business data corresponding to each of the target business fields is not included in the data warehouse, thereby providing a double guarantee for obtaining the visual target data corresponding to the target data retrieval request.

[0080] In order to improve the convenience and efficiency of subsequent retrieval of business data, an embodiment of the business data processing method provided in this application is provided. Figure 6 The business data processing method may further include the following contents before step 100:

[0081] Step 010: Store the business data corresponding to each business system in a data lake preset in the distributed system.

[0082] Step 020: Process the business data in the data lake to obtain subject tables for each business field and a unified indicator library.

[0083] Step 030: Store each of the business domain subject tables in a preset data warehouse in the distributed system.

[0084] From the above description, it can be seen that the business data processing method provided in the embodiment of the present application, by storing the business data corresponding to each business system in the data lake preset in the distributed system; processing the business data in the data lake to obtain the subject table of each business field and a unified indicator library, can effectively improve the convenience and efficiency of subsequent retrieval of business data, can realize the acquisition of the correlation between different types of business data, and thus can further break down the barriers between different business systems.

[0085] In order to provide an effective and reliable data basis for the process of extracting business data corresponding to each of the target business fields from the distributed system, in one embodiment of the business data processing method provided in the present application, the business field subject table is used to store the correspondence between the aggregate primary key corresponding to each of the business data and each attribute information used as the business field, wherein the aggregate primary key is information used to uniquely identify the corresponding business data; the unified indicator library is used to store a one-to-one or one-to-many correspondence between each indicator and each business field.

[0086] From the above description, it can be seen that the business data processing method provided in the embodiment of the present application, by clarifying the relationship between the business fields and the aggregate primary keys and indicators, can provide an effective and reliable data basis for the process of extracting the business data corresponding to each of the target business fields from the distributed system, thereby further improving the convenience and efficiency of subsequent retrieval of business data.

[0087] From the software level, in order to solve the problems of low efficiency, poor reliability and incompleteness of data query or retrieval in the existing business data processing methods, the present application provides an embodiment of a business data processing device for executing all or part of the content of the business data processing method, see Figure 7 , the business data processing device specifically includes the following contents:

[0088] The request reading module 10 is used to obtain target data processing rules and at least one type of target business domain corresponding to the target data retrieval request for the distributed system.

[0089] In the request reading module 10, the specific method of obtaining the target data processing rule corresponding to the target data retrieval request for the distributed system can be to directly read the target data processing rule specified by the target data retrieval request from the request content corresponding to the target data retrieval request, or to identify the processing keyword from the request content corresponding to the target data retrieval request based on a preset processing keyword identification method, and select the target data processing rule corresponding to the keyword from the correspondence between the pre-stored processing keyword and the data processing rule.

[0090] In the request reading module 10, the method of obtaining at least one type of target business domain corresponding to the target data retrieval request for the distributed system can be to obtain the business domain keyword from the request content corresponding to the target data retrieval request based on a preset business domain keyword identification method, and select the target business domain corresponding to the business domain keyword from the correspondence between the pre-stored business domain keyword and the business domain. Alternatively, the indicator keyword can be obtained from the request content corresponding to the target data retrieval request based on a preset indicator keyword identification method, and at least one target business domain corresponding to the indicator keyword can be searched in a pre-stored unified indicator library.

[0091] For example, if the request content corresponding to the target data retrieval request is a request to obtain the customer's monthly average assets, then the indicator keyword (i.e., the target indicator) is obtained as "assets" from the monthly average assets, and the two target business areas "deposits" and "liabilities" corresponding to the indicator keyword are searched in the pre-stored unified indicator library, and the processing keyword "monthly average" is obtained from the request content, then the target data processing rule corresponding to "monthly average" is selected from the correspondence between the pre-stored processing keywords and the data processing rules as the monthly average solution algorithm.

[0092] The data extraction module 20 is used to extract the business data corresponding to each of the target business areas from the distributed system.

[0093] The data processing module 30 is used to process the business data corresponding to each of the target business areas based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request.

[0094] The embodiments of the business data processing device provided in the present application can be specifically used to execute the processing flow of the embodiments of the business data processing method in the above embodiments. Its functions will not be described in detail here, and reference can be made to the detailed description of the above method embodiments.

[0095] From the above description, it can be seen that the business data processing device provided in the embodiment of the present application, by obtaining the target business field corresponding to the target data retrieval request for the distributed database, and extracting at least one type of pre-stored business data corresponding to the target business field from the distributed database, can break the barriers between business systems based on the target business field and the business field subject table that pre-stores the business data, avoid the formation of data islands, and make full use of the accumulated large amount of data assets to effectively improve the comprehensiveness and reliability of the data retrieved from the distributed database; by obtaining the target data processing rules corresponding to the target data retrieval request for the distributed database, and performing data processing on the business data corresponding to the target business field based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request, to obtain the visualized target data corresponding to the target data retrieval request, the efficiency and reliability of the data retrieval service can be effectively improved, and the corresponding target data can be displayed quickly, accurately and highly automatically for the target data retrieval requests proposed by different types of applications corresponding to different types of business front desks, thereby effectively improving the efficiency and accuracy of data retrieval, and effectively improving the user experience of business front desk operators.

[0096] Based on the above-mentioned business data processing device, the embodiment of the present application also provides a data middle platform, including a source data loading layer, a data indicator processing layer, a business object application layer and a value realization service layer connected in sequence; wherein the business object application layer can be the above-mentioned business data processing device, see Figure 1 The data center specifically includes the following contents:

[0097] 1. Paste source data loading layer:

[0098] As the entrance to the data center, the data lake application loads the source files of all front-end business systems and converts them into business tables in the distributed database to realize the access, conversion, and storage of business system data.

[0099] 2. Data indicator processing layer: Data warehouse applications and data indicator processing applications use the business tables in the data lake as source tables, classify and process the source data according to the bank's various business objects and uniformly formulated indicator standards, and precipitate business field subject tables and unified data indicators.

[0100] 3. Business object application layer: various service applications, such as customer management systems, employee performance systems, risk identification systems, etc., take data from data warehouses and indicator processing applications for further analysis and processing according to demand; if the indicator processing layer does not have the required data, it can directly take data from the data lake for processing. It can realize visual analysis of data assets and empower enterprise data-based operations.

[0101] 4. Value realization service layer: Various front-end business systems call the data of the business object application layer to perform business activities such as bank product recommendations, customer risk rating, and employee performance distribution. The business data accumulated in the process is fed back to the data center to form a complete data closed loop.

[0102] From the above description, it can be seen that the data middle platform provided in the embodiment of the present application, by obtaining the target business field corresponding to the target data retrieval request for the distributed database, and extracting at least one type of pre-stored business data corresponding to the target business field from the distributed database, can break down the barriers between business systems based on the target business field and the business field subject table that pre-stores business data, avoid the formation of data islands, and make full use of the accumulated large amount of data assets to effectively improve the comprehensiveness and reliability of data retrieval from the distributed database; by obtaining the target data processing rules corresponding to the target data retrieval request for the distributed database, and performing data processing on the business data corresponding to the target business field based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request, to obtain the visualized target data corresponding to the target data retrieval request, the efficiency and reliability of the data retrieval service can be effectively improved, and the corresponding target data can be displayed quickly, accurately and highly automatically for the target data retrieval requests made by different types of applications corresponding to different types of business front desks, thereby effectively improving the efficiency and accuracy of data retrieval, and effectively improving the user experience of business front desk operators.

[0103] In order to further illustrate this solution, this application also provides a specific application example of a data middle platform construction method under a commercial banking system, involving the field of enterprise-level data middle platform, so as to realize the all-round utilization of banking business data, data connectivity in all business fields, and unified data standard service sharing, meet the data application needs of various business fields, quickly support business scenario innovation, and realize data value realization.

[0104] The application example of this application divides the data middle-end architecture of a commercial bank into a source data loading layer, a data indicator processing layer, a business object application layer, and a value realization service layer according to the flow order of data processing and analysis.

[0105] The data flow of the entire middle platform is realized through the job scheduling tools of various applications (such as data warehouse technology ETL (Extract-Transform-Load), etc.), which mainly includes three elements: business entity table database, job scheduling server (such as ETL server, etc.), and job knowledge base.

[0106] 1. Paste source data loading layer:

[0107] As the entrance to the data center, the data lake application loads the source files of all front-end business systems and converts them into business tables in the distributed database to realize the access, conversion, and storage of business system data.

[0108] Build plan:

[0109] (1) Overall process: File loading is achieved through job-based ETL, a Hadoop cluster is built to store business tables, a scheduling server is built to deploy ETL-related scripts, and an ORACLE knowledge base is built to store ETL rules and logic;

[0110] The knowledge base configures the job trigger logic to call up all jobs. Each job corresponds to a file loading task. The corresponding loading algorithm is configured according to business needs, and the ETL process is performed on the job ETL server. The final business table is stored in the database corresponding to the Hadoop cluster.

[0111] (2) Table creation principle: Create a data lake table (dcm.tablexxx) corresponding to each source table (tablexxx). In principle, the field types in the data lake table correspond to the field types in the source system table.

[0112] (3) Loading logic:

[0113] Algorithm A: Append the latest data to the stock table;

[0114] F1 algorithm: all data are deleted and all data are added;

[0115] F2 algorithm: UPDATE (data update) + INSERT (data insertion);

[0116] F3 algorithm: standard historical zipper algorithm.

[0117] 2. Data indicator processing layer: Data warehouse applications and data indicator processing applications use the business tables in the data lake as source tables, classify and process the source data according to the bank's various business objects and uniformly formulated indicator standards, and precipitate business field subject tables and unified data indicators.

[0118] Data warehouse loading process: The underlying technology, ETL, knowledge base and data lake construction logic are the same.

[0119] Data warehouse processing principles:

[0120] (1) Extract data from the data lake without adding new derived fields, only changing the arrangement of the fields;

[0121] (2) It is composed of aggregated primary key information (usually customer number) + attribute information (describing the characteristics of the object);

[0122] For example: the customer domain subject table structure is shown in Table 1.

[0123] Table 1

[0124] Customer Number name Document type …… Contact Details

[0125] In the above Table 1, the customer number is the aggregate primary key, and the name, certificate type and contact information are all attribute information.

[0126] Data indicator processing application: record the definition and processing methods of various indicator data, such as:

[0127] Customer assets = deposits + liabilities

[0128] 3. Business object application layer: various service applications, such as customer management systems, employee performance systems, risk identification systems, etc., take data from data warehouses and indicator processing applications for further analysis and processing according to demand; if the indicator processing layer does not have the required data, it can directly take data from the data lake for processing. It can realize visual analysis of data assets and empower enterprise data-based operations.

[0129] For example, in a customer portrait application, the basic information of the corresponding customer can be obtained from the aggregate table of the data warehouse. The rest of the data can be extracted and loaded in the data lake and data warehouse according to the functions you want to achieve. For example, the customer's monthly average assets, the indicator 2 defines assets = deposits + liabilities. According to this principle, data is extracted from the tables that record deposit and liability information in the data lake and data warehouse, and then logically processed according to the "monthly average" you need to obtain the target data.

[0130] 4. Value realization service layer: Various front-end business systems call the data of the business object application layer to perform business activities such as bank product recommendations, customer risk rating, and employee performance distribution. The business data accumulated in the process is fed back to the data center to form a complete data closed loop.

[0131] For example, the SMS push system at the front desk calls on the monthly average assets of customers in the three customer portraits, and recommends private banking services via SMS to customers whose monthly average assets are greater than a certain value.

[0132] From the above description, it can be seen that the data middle platform construction method under the commercial banking system provided by the application example of this application realizes the construction of a data asset system under the commercial banking system with customers as the main body and business value as the guide, forming a universal shared, flexible and fast data middle platform. It connects the data of various business fields and precipitates common data assets, making data easier to understand and more flexible to use, improving the circulation speed of data in the asset system, and forming a complete closed loop of data storage, processing, precipitation and application.

[0133] From the hardware level, in order to solve the problems of low efficiency, poor reliability and incompleteness of data query or retrieval in the existing business data processing methods, the present application provides an embodiment of an electronic device for implementing all or part of the content of the business data processing method, and the electronic device specifically includes the following content:

[0134] Figure 8 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 8 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Figure 8 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0135] In one embodiment, the business data processing function may be integrated into a central processing unit. The central processing unit may be configured to perform the following control:

[0136] Step 100: Obtain target data processing rules and at least one type of target business domain corresponding to a target data retrieval request for a distributed system.

[0137] In step 100, the specific method of obtaining the target data processing rule corresponding to the target data retrieval request for the distributed system can be to directly read the target data processing rule specified by the target data retrieval request from the request content corresponding to the target data retrieval request, or to identify the processing keyword from the request content corresponding to the target data retrieval request based on a preset processing keyword identification method, and select the target data processing rule corresponding to the keyword from the correspondence between the pre-stored processing keyword and the data processing rule.

[0138] In step 100, the method for obtaining at least one type of target business domain corresponding to the target data retrieval request for the distributed system may be to obtain the business domain keyword from the request content corresponding to the target data retrieval request based on a preset business domain keyword identification method, and select the target business domain corresponding to the business domain keyword from the correspondence between the pre-stored business domain keyword and the business domain. Alternatively, the indicator keyword may be obtained from the request content corresponding to the target data retrieval request based on a preset indicator keyword identification method, and at least one target business domain corresponding to the indicator keyword may be searched in a pre-stored unified indicator library.

[0139] For example, if the request content corresponding to the target data retrieval request is a request to obtain the customer's monthly average assets, then the indicator keyword (i.e., the target indicator) is obtained as "assets" from the monthly average assets, and the two target business areas "deposits" and "liabilities" corresponding to the indicator keyword are searched in the pre-stored unified indicator library, and the processing keyword "monthly average" is obtained from the request content, then the target data processing rule corresponding to "monthly average" is selected from the correspondence between the pre-stored processing keywords and the data processing rules as the monthly average solution algorithm.

[0140] Step 200: extracting business data corresponding to each of the target business areas from the distributed system.

[0141] Step 300: Based on the target data processing rules, data processing is performed on the business data corresponding to each of the target business areas to obtain the visualized target data corresponding to the target data retrieval request.

[0142] From the above description, it can be seen that the electronic device provided by the embodiment of the present application, by obtaining the target business field corresponding to the target data retrieval request for the distributed database, and extracting at least one type of pre-stored business data corresponding to the target business field from the distributed database, can break the barriers between business systems based on the target business field and the business field subject table that pre-stores the business data, avoid the formation of data islands, and make full use of the accumulated large amount of data assets to effectively improve the comprehensiveness and reliability of the data retrieved from the distributed database; by obtaining the target data processing rules corresponding to the target data retrieval request for the distributed database, and performing data processing on the business data corresponding to the target business field based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request, to obtain the visualized target data corresponding to the target data retrieval request, the efficiency and reliability of the data retrieval service can be effectively improved, and the corresponding target data can be displayed quickly, accurately and highly automatically for the target data retrieval requests proposed by different types of applications corresponding to different types of business front desks, thereby effectively improving the efficiency and accuracy of data retrieval, and effectively improving the user experience of business front desk operators.

[0143] In another embodiment, the business data processing device may be configured separately from the central processor 9100. For example, the business data processing device may be configured as a chip connected to the central processor 9100, and the business data processing function is implemented under the control of the central processor.

[0144] like Figure 8As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 8 In addition, the electronic device 9600 may also include Figure 8 For components not shown, reference may be made to the prior art.

[0145] like Figure 8 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0146] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0147] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0148] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0149] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0150] The communication module 9110 is a transmitter / receiver 9110 that sends and receives signals via an antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0151] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0152] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the business data processing method in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps of the business data processing method in the above embodiments are implemented by the execution subject being a server or a client. For example, when the processor executes the computer program, the following steps are implemented:

[0153] Step 100: Obtain target data processing rules and at least one type of target business domain corresponding to a target data retrieval request for a distributed system.

[0154] In step 100, the specific method of obtaining the target data processing rule corresponding to the target data retrieval request for the distributed system can be to directly read the target data processing rule specified by the target data retrieval request from the request content corresponding to the target data retrieval request, or to identify the processing keyword from the request content corresponding to the target data retrieval request based on a preset processing keyword identification method, and select the target data processing rule corresponding to the keyword from the correspondence between the pre-stored processing keyword and the data processing rule.

[0155] In step 100, the method for obtaining at least one type of target business domain corresponding to the target data retrieval request for the distributed system may be to obtain the business domain keyword from the request content corresponding to the target data retrieval request based on a preset business domain keyword identification method, and select the target business domain corresponding to the business domain keyword from the correspondence between the pre-stored business domain keyword and the business domain. Alternatively, the indicator keyword may be obtained from the request content corresponding to the target data retrieval request based on a preset indicator keyword identification method, and at least one target business domain corresponding to the indicator keyword may be searched in a pre-stored unified indicator library.

[0156] For example, if the request content corresponding to the target data retrieval request is a request to obtain the customer's monthly average assets, then the indicator keyword (i.e., the target indicator) is obtained as "assets" from the monthly average assets, and the two target business areas "deposits" and "liabilities" corresponding to the indicator keyword are searched in the pre-stored unified indicator library, and the processing keyword "monthly average" is obtained from the request content, then the target data processing rule corresponding to "monthly average" is selected from the correspondence between the pre-stored processing keywords and the data processing rules as the monthly average solution algorithm.

[0157] Step 200: extracting business data corresponding to each of the target business areas from the distributed system.

[0158] Step 300: Based on the target data processing rules, data processing is performed on the business data corresponding to each of the target business areas to obtain the visualized target data corresponding to the target data retrieval request.

[0159] As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present application can break down the barriers between business systems based on the target business field and the business field subject table that pre-stores business data by obtaining the target business field corresponding to the target data retrieval request for the distributed database, and extract at least one type of pre-stored business data corresponding to the target business field from the distributed database, thereby avoiding the formation of data islands, and making full use of the accumulated large amount of data assets to effectively improve the comprehensiveness and reliability of data retrieval from the distributed database; by obtaining the target data processing rules corresponding to the target data retrieval request for the distributed database, and performing data processing on the business data corresponding to the target business field based on the target data processing rules to obtain the visualized target data corresponding to the target data retrieval request, the efficiency and reliability of the data retrieval service can be effectively improved, and the corresponding target data can be displayed quickly, accurately and highly automatically for the target data retrieval requests proposed by different types of applications corresponding to different types of business front desks, thereby effectively improving the efficiency and accuracy of data retrieval, and effectively improving the user experience of business front desk operators.

[0160] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (devices), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A business data processing method, characterized in that: include: Store the business data corresponding to each business system into the data lake preset in the distributed system; Process the business data in the data lake to obtain subject tables for various business fields and a unified indicator library; Each of the business domain subject tables is stored in a preset data warehouse in the distributed system; the business domain subject table is used to store the corresponding relationship between the aggregate primary key corresponding to each business data and each attribute information used as a business domain, wherein the aggregate primary key is information used to uniquely identify the corresponding business data; the unified indicator library is used to store a one-to-one or one-to-many corresponding relationship between each indicator and each business domain; Obtaining a target data processing rule and at least one type of target business domain corresponding to a target data retrieval request of a distributed system; Extracting business data corresponding to each of the target business areas from the distributed system; Based on the target data processing rules, data processing is performed on the business data corresponding to each of the target business fields to obtain the visualized target data corresponding to the target data retrieval request; The obtaining of the target data processing rules corresponding to the target data retrieval request for the distributed system includes: The target data processing rule specified by the target data retrieval request is directly read from the request content corresponding to the target data retrieval request, or the processing keyword is identified from the request content corresponding to the target data retrieval request based on a preset processing keyword identification method, and the target data processing rule corresponding to the keyword is selected from the correspondence between the pre-stored processing keyword and the data processing rule.

2. The business data processing method according to claim 1, characterized in that: The obtaining of target data processing rules and at least one type of target business domain corresponding to the target data retrieval request for the distributed system includes: Based on the business object application, the target data retrieval request for the distributed system sent by the business front end is received; Obtaining a target data processing rule corresponding to the target data retrieval request; And, it is determined whether the content of the target data retrieval request corresponds to a preset target indicator. If so, at least one target business field corresponding to the target indicator is obtained from a unified indicator library preset in the distributed system.

3. The business data processing method according to claim 1, characterized in that: The extracting the business data corresponding to each of the target business areas from the distributed system includes: It is searched from each business domain subject table of the data warehouse preset in the distributed system whether the business data corresponding to each of the target business domains are included; if so, the business data corresponding to each of the target business domains are extracted from the data warehouse.

4. The business data processing method according to claim 3, characterized in that: Also includes: If the data warehouse does not contain the business data corresponding to each of the target business fields, extracting the business data corresponding to each of the target business fields from the data lake preset in the distributed system; Among them, each of the business domain subject tables in the data warehouse is pre-processed based on the business source data in the data lake.

5. A business data processing device, characterized in that: include: A request reading module, used to obtain target data processing rules and at least one type of target business domain corresponding to a target data retrieval request for a distributed system; A data extraction module, used to extract business data corresponding to each of the target business areas from the distributed system; A data processing module, used for processing the business data corresponding to each of the target business fields based on the target data processing rules, so as to obtain the visualized target data corresponding to the target data retrieval request; The business data processing device is also specifically used for: The business data corresponding to each business system is stored in a data lake preset in the distributed system; Process the business data in the data lake to obtain subject tables for various business fields and a unified indicator library; Each of the business domain subject tables is stored in a preset data warehouse in the distributed system; the business domain subject table is used to store the corresponding relationship between the aggregate primary key corresponding to each business data and each attribute information used as a business domain, wherein the aggregate primary key is information used to uniquely identify the corresponding business data; the unified indicator library is used to store a one-to-one or one-to-many corresponding relationship between each indicator and each business domain; The request reading module is specifically used for: The target data processing rule specified by the target data retrieval request is directly read from the request content corresponding to the target data retrieval request, or the processing keyword is identified from the request content corresponding to the target data retrieval request based on a preset processing keyword identification method, and the target data processing rule corresponding to the keyword is selected from the correspondence between the pre-stored processing keyword and the data processing rule.

6. A data middle platform, characterized in that: include: The source data loading layer, data indicator processing layer, business object application layer and value realization service layer are connected in sequence; The business object application layer includes various types of business object applications, and the business object application layer is used to implement the business data processing method according to any one of claims 1 to 4, and the business object application layer extracts the business data corresponding to each of the target business fields from the data indicator processing layer or the source data loading layer; The source data loading layer includes: a data lake, which is used to load and store the business data corresponding to each business system; The data indicator processing layer includes: a data warehouse and a unified indicator library; the data warehouse is used to store the subject tables of each business field obtained by processing the business data in the data lake; the one-to-one or one-to-many correspondence between each indicator in the unified indicator library and each business field; The value realization service layer includes: various types of business front desks, and each business front desk corresponds to each of the business object applications one by one.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the business data processing method according to any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the business data processing method described in any one of claims 1 to 4 is implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the business data processing method according to any one of claims 1 to 4 is implemented.

Images