Multi-source heterogeneous data processing method, device, system and equipment
Through data weaving technology, the table structure information and association relationships of different databases are obtained and correlated, and the number-patch application tasks are constructed and executed, which solves the problem of splitting financial institutions when dealing with multi-source heterogeneous data, and realizes effective data management and rapid value mining.
Patent Information
- Application Number
- CN202411770851.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-03-28
AI Technical Summary
When banks and other financial institutions are processed by massive and diversified data, they are restricted by heterogeneous data, and the data is fragmented, sharing and interaction are difficult, and they are unable to effectively manage and use multi-source heterogeneous data, and the value of mining data is limited.
Data weaving technology is adopted to obtain table structure information and association relationships of different databases, and to construct target number application tasks based on user combination operations to perform tasks to process heterogeneous data.
It realizes flexible processing of multi-source heterogeneous data in different bank databases, effectively manages and uses data, stimulates data user consumption efficiency, and accelerates the mining of data value.
Smart Images
Figure CN119719184A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, device, system and equipment for processing multi-source heterogeneous data. Background Art
[0002] In recent years, with the rapid development of social economy, the amount of data that major banks, as the centers of financial services, need to process has increased exponentially.
[0003] At present, in order to fully store various types of massive and diversified data, banks and other financial institutions usually adopt a "divide and conquer" strategy, that is, subdividing the storage method, data landing text, relational database and non-relational database, etc. However, due to the limitations of heterogeneous data, data use is relatively fragmented, sharing and interaction are becoming increasingly difficult, and it is impossible to effectively manage and use multi-source heterogeneous data and mine data value. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to provide a multi-source heterogeneous data processing method, device, system and equipment, which can realize flexible processing of multi-source data across databases, systems and clusters in banks and other financial institutions based on data weaving technology, so as to effectively manage and use multi-source heterogeneous data, stimulate data user consumption efficiency, and accelerate data value mining.
[0005] In a first aspect, an embodiment of the present application provides a multi-source heterogeneous data processing method, including:
[0006] Obtain the table structure information of different data tables in different databases of the bank;
[0007] According to the table structure information of different data tables in different databases of the bank, obtaining the association relationship between the data tables;
[0008] By means of the association relationship between the data tables, different data tables are associated to obtain associated data tables; and based on the data weaving technology, in response to the user's combined operation on the associated data tables and table structure information, a target data weaving application task is constructed;
[0009] The target data coding application task is executed to obtain processing results for heterogeneous data in different databases of the bank.
[0010] Optionally, acquiring the association relationship between the data tables according to the table structure information of the different data tables in the different databases of the bank includes:
[0011] Using data sampling method, obtaining table field data of each data table from table structure information of different data tables in different databases of the bank;
[0012] The similarity and frequency of the table field data of each data table are calculated, and the association relationship between the data tables is determined according to the calculation results.
[0013] Optionally, the user's combined operations on the associated data table and table structure information include field selection, screening conditions, grouping and sorting, and defining table information.
[0014] Optionally, executing the target data coding application task to obtain processing results for heterogeneous data in different databases of the bank includes:
[0015] Parsing the target digital coding application task to obtain task details; translating the task details into machine language;
[0016] By running the machine language, the target computer application task is executed, and the result after the task is executed is obtained and saved.
[0017] In a second aspect, an embodiment of the present application further provides a multi-source heterogeneous data processing device, including:
[0018] A first acquisition unit is used to acquire table structure information of different data tables in different databases of the bank;
[0019] A second acquisition unit, configured to acquire the association relationship between the data tables according to the table structure information of the different data tables in the different databases of the bank;
[0020] An associating unit, used to associate different data tables through the association relationship between the data tables to obtain associated data tables; and based on the data weaving technology, in response to the user's combined operation on the associated data tables and table structure information, construct a target data weaving application task;
[0021] The execution unit is used to execute the target data coding application task and obtain the processing results for the heterogeneous data in different databases of the bank.
[0022] Optionally, the second acquiring unit includes:
[0023] An acquisition subunit, used to acquire table field data of each data table from table structure information of different data tables in different databases of the bank by using a data sampling method;
[0024] The calculation subunit is used to calculate the similarity and frequency of the table field data of each data table, and determine the association relationship between each table according to the calculation result.
[0025] Optionally, the user's combined operations on the associated data table and table structure information include field selection, screening conditions, grouping and sorting, and defining table information.
[0026] Optionally, the execution unit includes:
[0027] The parsing subunit is used to parse the target digital coding application task to obtain task details; and translate the task details into machine language;
[0028] The execution subunit is used to execute the target computer application task by running the machine language, obtain the result after the task is executed and save it.
[0029] In a third aspect, the embodiment of the present application further provides a multi-source heterogeneous data processing system, the system comprising: a metadata management module, a data relationship management module, a data compilation application module and a task scheduling module;
[0030] The metadata management module is used to obtain different table structure information from different databases of the bank;
[0031] The data relationship management module is used to obtain the association relationship between the tables according to the table structure information of different tables in different databases of the bank;
[0032] The data weaving application module is used to associate different data tables through the association relationship between the data tables to obtain the associated data tables; and based on the data weaving technology, in response to the user's combined operation on the associated data tables and table structure information, construct a target data weaving application task;
[0033] The task scheduling module is used to execute the target data coding application task and obtain the processing results for the heterogeneous data in different databases of the bank.
[0034] The embodiment of the present application also provides a multi-source heterogeneous data processing device, including: a processor, a memory, and a system bus;
[0035] The processor and the memory are connected via the system bus;
[0036] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the multi-source heterogeneous data processing method.
[0037] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned multi-source heterogeneous data processing method.
[0038] The embodiment of the present application provides a multi-source heterogeneous data processing method, device, system and equipment, which first obtains the table structure information of different data tables in different databases of the bank, then obtains the association relationship between each data table according to the table structure information of different data tables in different databases of the bank, and then associates different data tables through the association relationship between each data table to obtain the associated data table; and based on the data weaving technology, in response to the user's combined operation on the associated data table and the table structure information, constructs a target data weaving application task, and then executes the target data weaving application task to obtain the processing result of the heterogeneous data in different databases of the bank. In this way, the multi-source heterogeneous data in different databases of the bank can be effectively managed and used, the consumption efficiency of data users can be stimulated, and the mining of data value can be accelerated. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0040] Figure 1 A corresponding relationship diagram between a data weaving structure and a multi-source heterogeneous data processing system provided in an embodiment of the present application;
[0041] Figure 2 A flowchart of a multi-source heterogeneous data processing method provided in an embodiment of the present application;
[0042] Figure 3 A schematic diagram of the relationship between similarity and text viscosity provided in an embodiment of the present application;
[0043] Figure 4 An example diagram of the data node sql provided in the embodiment of the present application;
[0044] Figure 5 A schematic diagram of the overall processing process of a multi-source heterogeneous data processing system based on data weaving technology provided in an embodiment of the present application;
[0045] Figure 6 A schematic diagram of the composition of a multi-source heterogeneous data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] With the advent of the big data era, in the financial field, the amount of data that banks and other financial institutions need to process has increased exponentially. At present, in order to fully store various types of massive and diverse data, banks and other financial institutions usually adopt a "divide and conquer" strategy, that is, segment storage methods, data landing text, relational databases and non-relational databases, etc. However, due to the limitations of heterogeneous data, data use is relatively fragmented, sharing and interaction are becoming increasingly difficult, and it is impossible to effectively manage and use multi-source heterogeneous data and mine data value.
[0047] Data weaving, as an innovative technology, has appeared many times in trend reports in the field of data and analytical technology. In particular, in 2022, it ranked first among the 12 strategic technology trends, which can effectively manage and use heterogeneous data, stimulate data consumption efficiency, and accelerate data value mining.
[0048] Therefore, the applicant thought that based on data weaving technology, a distributed computing framework could be selected to explore efficient and flexible processing of heterogeneous data across databases, systems, and clusters, build a multi-source heterogeneous data processing system, and divide it into modules such as metadata management, data relationship management, data weaving application, and task scheduling. Figure 1 As shown, it supports query processing of CSV structured text, structured data, and unstructured heterogeneous data joint tables, solving the pain points and difficulties of data fragmentation.
[0049] In practical applications, the embodiment of the present application provides a multi-source heterogeneous data processing method, which can be executed by a pre-built multi-source heterogeneous data processing system or other computer equipment. Specifically, firstly, the table structure information of different data tables in different databases of the bank is obtained, and then, according to the table structure information of different data tables in different databases of the bank, the association relationship between each data table is obtained. Then, through the association relationship between each data table, different data tables are associated to obtain the associated data table; and based on the data weaving technology, in response to the user's combined operation on the associated data table and the table structure information, a target data weaving application task is constructed, and then the target data weaving application task can be executed to obtain the processing results for the heterogeneous data in different databases of the bank. In this way, the multi-source heterogeneous data in different databases of the bank can be effectively managed and used, the consumption efficiency of data users can be stimulated, and the mining of data value can be accelerated.
[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0051] In order to make the description of the following embodiments clear and concise, a brief introduction to the relevant concepts or technologies is first given:
[0052] Data weaving: A decentralized, distributed architecture concept whose value lies in dynamically improving data usage through built-in analytical technology, while adding automation capabilities to overall data management to reduce data management workload.
[0053] Multi-source heterogeneous data: a collection of different databases, including relational databases and non-relational databases.
[0054] Batch pull: Pull data from the data source regularly. It is suitable for scenarios where data updates are infrequent or the real-time requirements for data are not high.
[0055] Customer basic information: record customer information, such as customer number, customer name, customer level, etc.
[0056] Transaction details: store transaction information, such as customer number, transaction date, transaction amount, etc.
[0057] Data file data volume (N): the number of data items contained in the data file.
[0058] Data volume (n): The number of data items stored after screening and cleaning. N is not greater than N.
[0059] Target (data) table: the corresponding table to which the data file is to be imported, including relational and non-relational database tables (oralce, mysql, hbase, hive, etc.).
[0060] Target (data) table data volume (M): the number of data entries in the target table.
[0061] Derivative frequency: The frequency of data import, which generally includes daily, monthly and yearly. Daily means importing once a day, monthly means importing once a month, and so on. Yearly means importing once a year.
[0062] Derivative method: the method of importing data, generally including incremental and full methods. Full means that when importing a data file, the data in the target (data) table is cleared first, and then imported, M = n. Incremental means directly importing the data file into the target table (part of the data already exists in the target (data) table), M = M + n.
[0063] First embodiment
[0064] See also Figure 2 , is a flow chart of a multi-source heterogeneous data processing method provided in this embodiment, the method comprising the following steps:
[0065] S201: Acquire table structure information of different data tables in different databases of the bank.
[0066] In this embodiment, in order to improve the processing effect of multi-source heterogeneous data in financial structures such as banks, the metadata is first collected in batch pull mode, that is, the table structure information of different data tables in different databases of the bank is obtained to perform the subsequent step S202. The advantage of this method is that the collection configuration is controllable, easy to monitor and maintain. The following takes tdsql as an example to introduce the collection process. The collection of metadata is divided into table structure metadata collection and field metadata collection, which can specifically include the following 4 steps:
[0067] 1) Use batch reading of table structure information of different data tables in different system databases of the bank, including but not limited to table name, table comment, table fields, etc., and store them in newMap objects while generating table information data files with today's date.
[0068] 2) Read the table information data file of #{current date - 1}, store the data in the oldMap object, and delete the table information data file of #{current date - 4} (to reduce storage space and prevent re-running).
[0069] 3) Calculate the symmetric difference between newMap and oldMap, and record the result in the change information table.
[0070] 4) Update the metadata information table according to the change information table.
[0071] After obtaining the symmetric difference between newMap and oldMap in step 3, the result can be stored in resMap, and the oldMap data can be traversed and compared with newMap. The comparison result has three states:
[0072] State 1: Delete (del), oldMap exists in the table, but newMap does not exist.
[0073] State 2: equal, both oldMap and newMap exist and are equal.
[0074] State 3: Modification (modefy), both oldMap and newMap exist, but are not equal.
[0075] For states 1 and 3, the table and its state are recorded in resMap. For states 2 and 3, the oldMap and newMap table information are deleted. For state 1, the oldMap table information is deleted.
[0076] After completing the traversal of oldMap, if the number of newMap data is greater than 0, all newMap data will be stored in resMap and marked as added (add). Different resMap states correspond to different metadata information table operations.
[0077] S202: Acquire the association relationship between the data tables according to the table structure information of different data tables in different databases of the bank.
[0078] In this embodiment, after obtaining the table structure information of different data tables in different databases of the bank through step S201, further, using the data sampling method, the table field data of each data table is obtained from the table structure information of different data tables in different databases of the bank. Then, the similarity and frequency of the table field data of each data table are calculated, and the association relationship between each table is determined according to the calculation result to perform the subsequent step S203.
[0079] Specifically, when obtaining the association between various data tables, such as the customer number in table a is equal to the customer number in table b, the association relationship is used to assist modeling and apply joint table processing. This application describes this method in detail through a customer basic information and transaction details table, wherein the customer basic information comes from a column-based database management system, and the transaction details are stored in a column-based database.
[0080] The extraction of association relationships mainly includes the following two steps:
[0081] (1) Data sampling: extract and record table field data.
[0082] Before comparing the similarity of fields, it is necessary to obtain the table field data. In order to make the similarity comparison results accurate and efficient, this application adopts data sampling to extract the first 1000 and last 1000 values of the field and splice them together as sampling values. In order to avoid multiple sampling and reduce the comparison efficiency, this application saves the sampled data in the data sampling table and updates it every three months, reducing the sampling frequency while ensuring timely data update.
[0083] (2) Relationship calculation: calculate similarity and frequency to determine data relationships.
[0084] After data sampling is completed, relationship calculation can be performed. Field relationships are mainly divided into equality, sub-equality and weak equality. Equality means that the field storage values are the same. When building, it can be directly expressed as table a. field 1 = table b. field 2, corresponding to the "=" in the SQL statement; similarly, sub-equality and weak equality correspond to "like", and weak equality means that it is not recommended unless necessary. It is recommended to use equality and sub-equality first. The calculation of association relationships mainly includes similarity calculation and frequency calculation.
[0085] Step 1: Similarity calculation
[0086] When comparing the similarity of two fields, the similarity is determined based on the number of co-occurrences of the field values. The number of co-occurrences refers to the number of times two field values appear equal. Determine the number of intersections of the field sampling data, and then divide it by the number of smaller elements in the sampling data to get the similarity. If the similarity is greater than the set upper threshold and the field is a number or character, set the relationship to equal. If it is between the lower limit and the upper limit, set it to sub-equal.
[0087] The above comparison is more friendly to numeric and character fields, and the similarity result of text fields is lower. Therefore, if the similarity is less than the set threshold, the similarity function is used to compare the text and obtain the text viscosity. The relationship between data similarity and text viscosity is as follows: Figure 3 If the text viscosity is greater than the upper limit of the text threshold, it is set to secondary equality. If it is between the upper limit and the lower limit, it is set to weak equality. If it is less than the lower limit, no relationship is recorded.
[0088] Step 2: Calculate the quantity threshold
[0089] When calculating similarity, if the amount of data is small and the number of co-occurrences is close to the amount of data, a higher similarity can be obtained. Extreme case: If the amount of field data is small (such as 1), only 1 occurrence of equal data is required for the similarity to reach 100%. To prevent this, this application sets a quantity threshold to exclude "strong associations" with small quantities. If the number of smaller fields is less than the threshold, the similarity is downgraded, equality is downgraded to secondary equality, secondary equality is weaker equality, and so on.
[0090] S203: through the association relationship between the various data tables, different data tables are associated to obtain associated data tables; and based on the data weaving technology, in response to the user's combined operation on the associated data tables and table structure information, a target data weaving application task is constructed.
[0091] In this embodiment, after determining the association relationship between each data table through step S202, different data tables can be further associated through the association relationship between each data table to obtain associated data tables; and based on data weaving technology, in response to the user's combined operation on the associated data table and table structure information, a target data weaving application task is constructed to execute the subsequent step S204.
[0092] The user's combined operations on the associated data table and table structure information may include, but are not limited to, field selection, screening conditions, grouping and sorting, and definition of table information.
[0093] Specifically, the target data weaving application task is constructed based on data weaving technology, that is, the data weaving laboratory, which shields the underlying data differences, ignores the "surge" of underlying data interactions, creates a "calm" data usage mode, provides users with convenient label modeling tools, digs out risk "reefs", further reveals hidden risks, and improves the quality and efficiency of risk disclosure. It is divided into the following steps:
[0094] Step 1: Add a new model. Enter the model name and model purpose in the pop-up page, and then click Confirm to enter the model adding page. The left side of the model adding page can be a multi-source data area and a tool area, and the right side can be an experimental area. In this way, the table structure information obtained through step S201 can be displayed to the user through this page, so that the user can select the corresponding table for operation, such as combination operation, dictionary translation, etc. From the user's perspective, the multi-source data area on the left side of the model adding page only contains two types of data, the original table and the model table. Among them, the original table comes from different relational and non-relational data sources. The user does not care about or need to understand the source of the table, that is, it provides non-differentiated data. The user can select any table for combination operation, dictionary translation, etc. The tool area includes operations such as combination, dictionary translation and filtering. The dictionary translation supports custom field dictionaries, and the filtering is based on the field to filter data to achieve the purpose of narrowing the data range. Combination means table connection, which will be described in detail later. The user can drag the required table from the multi-source data area to the data editing experimental area, and then use the data editing tool to operate the table.
[0095] Step 2: Click the operation tool button in the test area to perform dictionary translation, filtering and other operations. The combined operation is described in detail here. The combined operation supports users to create models independently using a 4-step mode, including field selection, filtering conditions, grouping and sorting, and defining table information.
[0096] <1> Field Selection
[0097] Select the fields to be displayed. You can rename the fields or use functions to combine them to get new fields. The display fields can be moved in order. In the field selection page, for multiple tables selected for combination, the associations of these tables can be obtained according to the relationship management. And select the fields to be retained after combination. In addition, you can perform function operations on some fields of these tables, such as sum, average, maximum value, etc.
[0098] <2> Conditional specification
[0099] It is divided into filtering conditions and default conditions. The default conditions are automatically filled by the system according to the table relationship, including table associations and associated fields. The default conditions provide the business with table relationship information and provide auxiliary information for the filtering conditions. In addition, users can choose whether to use this condition. The filtering conditions are customized by the user, and the operation is similar to selecting fields. The condition specification page displays the tables obtained according to the relationship management and determines the association relationship of the tables.
[0100] <3> Grouping and Sorting
[0101] If a function is used in the field selection stage, the non-function field is automatically filled. In the grouping options, users can only adjust the grouping order and cannot delete it. In addition, for function fields, users can add them to the grouping. Sorting supports ascending and descending sorting of any field. If you have previously selected a function in the " <1> If you have performed functions on the fields in the Field Selection step, you need to group them on the Grouping and Sorting page. Otherwise, you do not need to group them. In addition, users can sort the fields.
[0102] <4> Fill in the table information
[0103] Fill in the table information in Chinese, English and table description to complete the last step of table association. Click Save to proceed to the next step of label modeling. In the table information filling page, = can be used to determine the table name, table description and other information after multiple tables are combined, and the selected table, field, combination conditions, etc. are displayed below. Display table information and table logic.
[0104] Step 3: Check the result of the number coding
[0105] After completing the model editing, you can submit the task to the background for execution on the data editing modeling page. After the operation is completed, view the task table and result table in the data editing queue. The process table is saved for 3 months, and the result table is retained permanently. However, it is limited by the user's own space. If the space is full, you need to clean up the data before executing the task. The task table is as follows, and the data results can be downloaded. In this way, the task list page can provide services for users to view the tasks they submitted and the status of the tasks, as well as view the details of the task result table.
[0106] S204: Execute the target data coding application task to obtain the processing results of the heterogeneous data in different databases of the bank.
[0107] In this embodiment, after constructing the target digital coding application task through step S203, the target digital coding application task can be further parsed to obtain the task details; and the task details are translated into machine language, and then the target digital coding application task is executed by running the machine language to obtain the result after the task is executed and save it as the processing result of the heterogeneous data in different databases of the bank.
[0108] Among them, this application does not limit the content of task details, which can be understood as SQL described in various languages. For example, filter the data with an amount greater than 1000 in table a and the data with a payment mark of receipt in table b, and associate table a and table b through xxx.
[0109] Specifically, the implementation process of this step S204 may include the following three steps: model construction, task execution and result preservation. Among them, the model construction step is to convert the model defined on the front-end page into an executable calculation graph to facilitate subsequent optimization and execution; the task execution step is to submit the defined task to the distributed computing framework for step-by-step execution; the result preservation step is to permanently preserve the final result of the model (supporting user-initiated deletion), and the process table is stored for 3 months.
[0110] Step 1: Model building
[0111] The data model is a tree composed of multiple related nodes. Each node is a set of SQL, divided into data source objects, operation objects and result objects, such as Figure 4 As shown in the figure, the objects are arranged and the nodes are encapsulated to finally form a complete data weaving computational graph.
[0112] like Figure 4 As shown, data source object: create a data source adapter, extract data from heterogeneous data sources such as relational databases and non-relational databases, construct objects, shield users from differences in underlying data storage structures, and provide a non-differentiated, unified data access view.
[0113] Operation object: Build an operation optimizer to transform or combine one or more data source objects or operation objects, including filtering, grouping statistics, association, set operations, etc. Operation objects can be used to define a complete data analysis process.
[0114] Result object: records and stores the intermediate and final results of the model for users to view.
[0115] Step 2: Task execution
[0116] To execute tasks using a distributed computing framework, first register each node in the model building, and select different API registrations for different data source nodes. The distributed computing framework receives the registration information, serializes and verifies the registry and functions, and distributes and loads them to the computing nodes when necessary to execute subsequent data processing and computing logic. This application uses a stream-batch integrated approach to execute tasks and calls the execution method of the distributed computing framework.
[0117] Step 3: Save the results
[0118] When the node is successfully executed, the execution results are stored in the column-based database management system, and the data volume is recorded and the personal space usage is updated. Subsequently, the user can clean up the space on his own. In addition, the process table is cleaned up regularly in batches to avoid data accumulation.
[0119] In this way, by executing the above steps S201-S204, based on the concept of data weaving, a multi-source data and multi-source heterogeneous data wide database joint table query processing method is constructed, such as flexible query processing of relational database and column storage database joint table, so as to realize cross-database, cross-system, and cross-cluster multi-source data flexible processing. This application is not only an exploration of the concept of data weaving architecture, but also an active attempt and "step" to break through the "data island" and break down data barriers.
[0120] In summary, the present embodiment provides a multi-source heterogeneous data processing method, which first obtains the table structure information of different data tables in different databases of the bank, then obtains the association relationship between each data table according to the table structure information of different data tables in different databases of the bank, and then associates different data tables through the association relationship between each data table to obtain the associated data table; and based on the data weaving technology, in response to the user's combined operation on the associated data table and the table structure information, constructs a target data weaving application task, and then executes the target data weaving application task to obtain the processing result of the heterogeneous data in different databases of the bank. In this way, the multi-source heterogeneous data in different databases of the bank can be effectively managed and used, the consumption efficiency of data users can be stimulated, and the mining of data value can be accelerated.
[0121] Second embodiment
[0122] This embodiment will introduce a multi-source heterogeneous data processing system. For related content, please refer to the above method embodiment. The system includes: a metadata management module, a data relationship management module, a data coding application module and a task scheduling module.
[0123] Among them, the metadata management module is used to obtain different table structure information in different databases of the bank; the data relationship management module is used to obtain the association relationship between each table according to the table structure information of different tables in different databases of the bank; the data weaving application module is used to associate different data tables through the association relationship between each data table to obtain the associated data table; and based on the data weaving technology, in response to the user's combined operation on the associated data table and table structure information, construct the target data weaving application task; the task scheduling module is used to execute the target data weaving application task to obtain the processing results for the heterogeneous data in different databases of the bank.
[0124] It can be seen that this embodiment builds a distributed computing framework based on data weaving technology, and innovatively proposes a multi-source heterogeneous data processing method, which supports cross-database, cross-cluster, and cross-system joint table query analysis and processing of multiple types of heterogeneous data sources, such as tdsql and column storage database joint table query. The system includes four modules: metadata management, data relationship management, data weaving application, and data weaving task scheduling. Figure 5As shown, it shows the overall processing process of the multi-source heterogeneous data processing system based on the data weaving technology provided by this embodiment.
[0125] Among them, the metadata management module automatically identifies and records data from various data sources, providing basic information for data use.
[0126] Data relationship management module: Use automation technology to build data connection relationships, form data relationship maps, and intuitively display data association relationships.
[0127] Task scheduling module: schedules tasks submitted by digital applications based on task submission time, priority, allocated resources and other information, and returns the results to the user.
[0128] Data coding application module: calls the relationship module, provides table data association relationships, supports flexible selection of tables and fields for multi-source data joint query, and realizes data analysis and mining functions.
[0129] In summary, the multi-source heterogeneous data processing system provided by this embodiment first obtains the table structure information of different data tables in different databases of the bank through the metadata management module, then uses the data relationship management module to obtain the association relationship between each data table according to the table structure information of different data tables in different databases of the bank, and then uses the data weaving application module to associate different data tables through the association relationship between each data table to obtain the associated data table; and based on the data weaving technology, in response to the user's combined operation on the associated data table and the table structure information, constructs a target data weaving application task, and then can execute the target data weaving application task through the task scheduling module to obtain the processing results of the heterogeneous data in different databases of the bank. In this way, the multi-source heterogeneous data in different databases of the bank can be effectively managed and used, the consumption efficiency of data users can be stimulated, and the mining of data value can be accelerated.
[0130] Third embodiment
[0131] This embodiment will introduce a multi-source heterogeneous data processing device. For related content, please refer to the above method embodiment.
[0132] See also Figure 6 , is a schematic diagram of the composition of a multi-source heterogeneous data processing device provided in this embodiment, the device comprising:
[0133] The first acquisition unit 601 is used to acquire the table structure information of different data tables in different databases of the bank;
[0134] The second acquisition unit 602 is used to acquire the association relationship between the data tables according to the table structure information of different data tables in different databases of the bank;
[0135] The associating unit 603 is used to associate different data tables through the association relationship between the data tables to obtain associated data tables; and based on the data weaving technology, in response to the user's combined operation on the associated data tables and table structure information, construct a target data weaving application task;
[0136] The execution unit 604 is used to execute the target data coding application task to obtain the processing results for the heterogeneous data in different databases of the bank.
[0137] In an implementation of this embodiment, the second obtaining unit 602 includes:
[0138] An acquisition subunit, used to acquire table field data of each data table from table structure information of different data tables in different databases of the bank by using a data sampling method;
[0139] The calculation subunit is used to calculate the similarity and frequency of the table field data of each data table, and determine the association relationship between each table according to the calculation result.
[0140] In an implementation of this embodiment, the user's combined operations on the associated data table and table structure information include field selection, screening conditions, grouping and sorting, and defining table information.
[0141] In an implementation of this embodiment, the execution unit 604 includes:
[0142] The parsing subunit is used to parse the target digital coding application task to obtain task details; and translate the task details into machine language;
[0143] The execution subunit is used to execute the target computer application task by running the machine language, obtain the result after the task is executed and save it.
[0144] In summary, the multi-source heterogeneous data processing device provided in this embodiment first obtains the table structure information of different data tables in different databases of the bank, then obtains the association relationship between each data table according to the table structure information of different data tables in different databases of the bank, and then associates different data tables through the association relationship between each data table to obtain the associated data table; and based on the data weaving technology, in response to the user's combined operation on the associated data table and the table structure information, constructs a target data weaving application task, and then executes the target data weaving application task to obtain the processing result of the heterogeneous data in different databases of the bank. In this way, the multi-source heterogeneous data in different databases of the bank can be effectively managed and used, the consumption efficiency of data users can be stimulated, and the mining of data value can be accelerated.
[0145] Furthermore, an embodiment of the present application also provides a multi-source heterogeneous data processing device, including: a processor, a memory, and a system bus;
[0146] The processor and the memory are connected via the system bus;
[0147] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned multi-source heterogeneous data processing method.
[0148] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any implementation method of the above-mentioned multi-source heterogeneous data processing method.
[0149] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0150] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0151] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0152] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for processing multi-source heterogeneous data, characterized in that: include: Obtain the table structure information of different data tables in different databases of the bank; According to the table structure information of different data tables in different databases of the bank, obtaining the association relationship between the data tables; By means of the association relationship between the data tables, different data tables are associated to obtain associated data tables; and based on the data weaving technology, in response to the user's combined operation on the associated data tables and table structure information, a target data weaving application task is constructed; The target data coding application task is executed to obtain processing results for heterogeneous data in different databases of the bank.
2. The method according to claim 1, characterized in that The obtaining the association relationship between the data tables according to the table structure information of the different data tables in the different databases of the bank includes: Using data sampling method, obtaining table field data of each data table from table structure information of different data tables in different databases of the bank; The similarity and frequency of the table field data of each data table are calculated, and the association relationship between the data tables is determined according to the calculation results.
3. The method according to claim 1, characterized in that The user's combined operations on the associated data table and table structure information include field selection, screening conditions, grouping and sorting, and defining table information.
4. The method according to claim 1, characterized in that The executing of the target data coding application task to obtain the processing results of heterogeneous data in different databases of the bank includes: Parsing the target digital coding application task to obtain task details; translating the task details into machine language; By running the machine language, the target computer application task is executed, and the result after the task is executed is obtained and saved.
5. A multi-source heterogeneous data processing device, characterized in that: include: A first acquisition unit is used to acquire table structure information of different data tables in different databases of the bank; A second acquisition unit, configured to acquire the association relationship between the data tables according to the table structure information of the different data tables in the different databases of the bank; An associating unit, used to associate different data tables through the association relationship between the data tables to obtain associated data tables; and based on the data weaving technology, in response to the user's combined operation on the associated data tables and table structure information, construct a target data weaving application task; The execution unit is used to execute the target data coding application task and obtain the processing results for the heterogeneous data in different databases of the bank.
6. The device according to claim 5, characterized in that The second acquiring unit includes: An acquisition subunit, used to acquire table field data of each data table from table structure information of different data tables in different databases of the bank by using a data sampling method; The calculation subunit is used to calculate the similarity and frequency of the table field data of each data table, and determine the association relationship between each table according to the calculation result.
7. The device according to claim 5, characterized in that The user's combined operations on the associated data table and table structure information include field selection, screening conditions, grouping and sorting, and defining table information.
8. A multi-source heterogeneous data processing system, characterized in that: The system includes: a metadata management module, a data relationship management module, a data editing application module and a task scheduling module; The metadata management module is used to obtain different table structure information from different databases of the bank; The data relationship management module is used to obtain the association relationship between the tables according to the table structure information of different tables in different databases of the bank; The data weaving application module is used to associate different data tables through the association relationship between the data tables to obtain the associated data tables; and based on the data weaving technology, in response to the user's combined operation on the associated data tables and table structure information, construct a target data weaving application task; The task scheduling module is used to execute the target data coding application task and obtain the processing results for the heterogeneous data in different databases of the bank.
9. A multi-source heterogeneous data processing device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 4.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 4.