Data processing method and device

Abstract

The embodiments of this specification provide a data processing method and device, wherein the data processing method includes: obtaining at least two business data tables, and constructing an initial knowledge graph based on the fields contained in the at least two business data tables, wherein the at least two business data tables respectively correspond to different business types of the target business, determining the association relationship between different fields in the at least two business data tables based on historical operation data related to the at least two business data tables, updating the initial knowledge graph based on the association relationship, generating a target knowledge graph, and constructing a business wide table of the target business based on the target knowledge graph.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a data processing method. Background Art

[0002] A data warehouse (DW or DWH) is a structured data environment. It provides data support for applications such as data analysis, reporting, and data mining. Data warehouse management is a core component of data warehouse operations and maintenance, typically encompassing data maintenance and warehouse construction evaluation. The primary mission of a data warehouse is to organize, summarize, and reorganize information and provide it to decision makers in a timely manner. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0004] According to a first aspect of an embodiment of this specification, there is provided a data processing method, including:

[0005] Obtain at least two business data tables, and construct an initial knowledge graph based on the fields contained in the at least two business data tables, wherein the at least two business data tables respectively correspond to different business types of the target business;

[0006] determining, based on historical operation data associated with the at least two business data tables, association relationships between different fields in the at least two business data tables;

[0007] Update the initial knowledge graph according to the association relationship to generate a target knowledge graph;

[0008] A business wide table of the target business is constructed based on the target knowledge graph.

[0009] Optionally, constructing an initial knowledge graph based on the fields contained in the at least two business data tables includes:

[0010] The table identifier of the target business data table is used as the first node, the field identifiers corresponding to different fields in the target business data table are used as the second node, and the inclusion relationship between the target business data table and the different fields is used as the edge between the first node and the second node to construct an initial sub-knowledge graph corresponding to the target business data table, wherein the target business data table is one of the at least two business data tables, and the initial sub-knowledge graphs corresponding to the at least two business data tables together constitute the initial knowledge graph.

[0011] Optionally, updating the initial knowledge graph according to the association relationship to generate a target knowledge graph includes:

[0012] When it is determined that there is an association relationship between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the association relationship, and an edge is constructed between the first node of the first business data table and the first node of the second business data table to update the initial knowledge graph and generate a target knowledge graph.

[0013] Optionally, determining the association relationship between different fields in the at least two business data tables based on historical operation data related to the at least two business data tables includes:

[0014] A first association relationship between the at least two business data tables and a second association relationship between different fields in the at least two business data tables are determined based on historical operation data related to the at least two business data tables.

[0015] Optionally, updating the initial knowledge graph according to the association relationship to generate a target knowledge graph includes:

[0016] When it is determined that a first association relationship exists between the first business data table and the second business data table, an edge is constructed between a first node of the first business data table and a first node of the second business data table in the initial knowledge graph based on the first association relationship;

[0017] When it is determined that a second association relationship exists between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the second association relationship to update the initial knowledge graph and generate a target knowledge graph.

[0018] Optionally, the business wide table includes a data warehouse table;

[0019] Accordingly, the step of constructing the business wide table of the target business based on the target knowledge graph includes:

[0020] Use the field identifiers corresponding to each node in the target knowledge graph as fields to construct the initial data warehouse table;

[0021] Adjust the field positions in the initial data warehouse table according to the association relationship between the nodes in the target knowledge graph to generate an intermediate data warehouse table;

[0022] The business data in the at least two business data tables are added to the data units of the corresponding fields in the intermediate data warehouse table to generate a target data warehouse table.

[0023] Optionally, constructing a business wide table of the target business based on the target knowledge graph includes:

[0024] An enhanced entity relationship graph is constructed based on the target knowledge graph, and a business wide table of the target business is constructed according to the association relationship between different entities in the enhanced entity relationship graph.

[0025] Optionally, constructing an enhanced entity relationship graph based on the target knowledge graph includes:

[0026] Determine a first field and a second field in a correlation relationship between the at least two business data tables;

[0027] Deduplication processing is performed on the business data contained in the first field and the second field;

[0028] Determine the amount of business data contained in the first field and the second field according to the deduplication processing result, and divide the at least two business data tables into a master table and a slave table according to the data amount;

[0029] Determine the association relationship between the master table and the slave table according to the association relationship between the nodes in the target knowledge graph;

[0030] Based on the association relationship between the master table and the slave table, an enhanced entity relationship diagram is constructed.

[0031] Optionally, dividing the at least two business data tables into a master table and a slave table according to the data volume includes:

[0032] The business data table to which the target field containing the data amount greater than the preset data amount threshold in the first field and the second field belongs is divided into a master table, and the business data tables other than the master table in the at least two business data tables are divided into slave tables;

[0033] Accordingly, determining the association relationship between the master table and the slave table according to the association relationship between the nodes in the target knowledge graph includes:

[0034] According to the association relationship between each node in the target knowledge graph, the association relationship between the main table and each slave table is determined.

[0035] Optionally, adding the business data in the at least two business data tables to data units of corresponding fields in the intermediate data warehouse table to generate a target data warehouse table includes:

[0036] Determining a mapping relationship between a target business data table in the at least two business data tables and each field in the intermediate data warehouse table, wherein the target business data table is one of the at least two business data tables;

[0037] Based on the table structure of the intermediate data warehouse table, the business data in the target business data table is added to the data unit of the corresponding field in the intermediate data warehouse table according to the mapping relationship to generate the target data warehouse table.

[0038] Optionally, after constructing the target data warehouse table based on the target knowledge graph, the method further includes:

[0039] Receive a data query instruction, wherein the data query instruction carries a field identifier of a target field to be queried and a table identifier of a target business data table to which the field identifier belongs;

[0040] Using the table identifier as index information, and performing data index processing according to the mapping relationship between the table identifier and the target field in the business wide table;

[0041] Output the index results as data query results.

[0042] Optionally, after constructing the business wide table of the target business based on the target knowledge graph, the method further includes:

[0043] When it is detected that there is incremental data in the at least two business data tables, the business wide table is updated based on the incremental data.

[0044] Optionally, the data processing method further includes:

[0045] The field information contained in the at least two business data tables is input into a text processing model to perform similarity calculation, and the association relationship between different fields in the at least two business data tables is determined based on the similarity calculation result.

[0046] According to a second aspect of the embodiments of this specification, there is provided a data processing device, including:

[0047] an acquisition module configured to acquire at least two business data tables and construct an initial knowledge graph based on fields contained in the at least two business data tables, wherein the at least two business data tables respectively correspond to different business types of the target business;

[0048] a determining module configured to determine, based on historical operation data associated with the at least two business data tables, association relationships between different fields in the at least two business data tables;

[0049] A generation module is configured to update the initial knowledge graph according to the association relationship to generate a target knowledge graph;

[0050] A construction module is configured to construct a business wide table of the target business based on the target knowledge graph.

[0051] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0052] memory and processor;

[0053] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement any step of the data processing method.

[0054] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of any one of the data processing methods are implemented.

[0055] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0056] One embodiment of the present specification obtains at least two business data tables and constructs an initial knowledge graph based on the fields contained in the at least two business data tables, wherein the at least two business data tables respectively correspond to different business types of the target business, and determines the association relationship between different fields in the at least two business data tables based on historical operation data related to the at least two business data tables, updates the initial knowledge graph based on the association relationship, generates a target knowledge graph, and constructs a business wide table of the target business based on the target knowledge graph.

[0057] The embodiments of this specification generate a target knowledge graph by constructing an initial knowledge graph based on the fields in the business data table, and updating the association relationships between different nodes in the initial knowledge graph based on historical operation data, so that a business wide table can be constructed based on the target knowledge graph. To a certain extent, the construction of the business wide table can be automated or semi-automated, which is beneficial to reducing the cost required for the construction of the business wide table, improving the construction efficiency of the business wide table, and ensuring the accuracy of the construction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of a data processing method provided by one embodiment of this specification;

[0059] Figure 2ais a schematic diagram of the initial knowledge graph construction result provided by one embodiment of this specification;

[0060] Figure 2b is a schematic diagram of a target knowledge graph provided by an embodiment of this specification;

[0061] Figure 3 This is a flowchart of a data processing method provided by one embodiment of this specification;

[0062] Figure 4 This is a schematic diagram of the structure of a data processing device provided by one embodiment of this specification;

[0063] Figure 5 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0064] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0065] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0066] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0067] First, the terms involved in one or more embodiments of this specification are explained.

[0068] Metadata: Metadata describes data about data, descriptive information about data and information resources. Metadata can be described using attributes, such as the size of a data store, when it was collected, when it was updated, who maintains it, and other information.

[0069] Data assets: Data assets are data collection resources owned or controlled by individuals or enterprises, which can bring certain economic benefits.

[0070] Knowledge graph: Integrate relevant knowledge in vertical fields in the form of graph schema, describe domain knowledge through graph entities and relationships, and provide production value.

[0071] ER diagram: Entity-relationship diagram is a method that provides entities / attributes and relationships to describe the conceptual model of the real world.

[0072] Data warehouse wide table: A database table that groups together related indicators, dimensions, and attributes of the same business entity, used for data warehouse modeling and data mining of big data.

[0073] Data warehouse modeling: The way to construct tables in a data warehouse, specifically starting from the business process, selecting appropriate dimensions and integrating corresponding fact indicators.

[0074] In current business systems, online data is stored and used by OLTP data systems. In order to solve some data statistical reports and intelligent AI system learning, enterprises often build their own data warehouse systems based on OLAP systems. Due to the different data storage and usage methods of business systems, data in online system OLTP is often stored in a decentralized manner according to data paradigms and business domains, while data in the OLAP system of the data warehouse is often modeled by data in similar business domains for centralized storage and use. This involves the construction of data models in the data warehouse. Since the developers of online business systems and data warehouses belong to different teams and technical fields, the construction of data warehouses often relies on the domain knowledge of online business. This is a difficult problem in the data warehouse construction process. If there is a corresponding system that can help data warehouse developers automatically build business models, part of the basic work of data warehouse construction can be solved.

[0075] The data management system of the embodiment of this specification maintains the data and metadata data assets of the online system and data warehouse system in the enterprise. At the same time, it combines the business relationships and operational relationships of the data assets to construct a data asset knowledge graph, which stores the entity information of the enterprise's data assets and the association relationships between various business fields. Based on the data asset knowledge graph, the data warehouse table can be automatically constructed.

[0076] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0077] Figure 1 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0078] Step 102: Obtain at least two business data tables and construct an initial knowledge graph based on the fields contained in the at least two business data tables, wherein the at least two business data tables correspond to different business types of the target business.

[0079] In the embodiments of this specification, a data warehouse is a strategic collection of all types of data supporting a user's decision-making process. It is an environment that provides users with current and historical data for decision support, data that is difficult or impossible to obtain in traditional operational databases. Data warehouse technology is a general term for various technologies and modules designed to effectively integrate operational data into a unified environment to provide access to decision-making data. Its ultimate goal is to enable users to quickly and conveniently retrieve the information they need and provide decision support.

[0080] Traditional data warehouse business domain modeling is implemented manually by business domain experts. Business experts in a certain field of the data warehouse team study and summarize the business, design and develop models, and convert the data storage organization method of the online system into the data storage organization method in the data warehouse.

[0081] In actual OLTP systems, data storage is often distributed across various business systems due to business and performance requirements. For example, in the member domain, if four systems maintain member information, the corresponding member information is stored in four business data tables: ods_huiyuan_t1, ods_huiyuan_t2, ods_huiyuan_t3, and ods_huiyuan_t4. Each business data table stores information related to member users in different dimensions.

[0082] This data, scattered across four systems and four databases, represents the fragmented storage of member-domain data in OLTP. Each of these systems is used for business purposes. However, this fragmented data storage and usage is not user-friendly for data analysts and business operations personnel, making it difficult for them to analyze customer data. Therefore, data warehouse developers often use OLAP systems to remodel and develop the data. A common development approach involves integrating data from several business domains to construct a wide business table that contains most of the relevant data for that domain. For example, in the data warehouse, member-domain data would correspond to a wide table named ods_huiyuan_info. This table would have a unique user ID as the primary key, and all other fields would contain member-related information. Data warehouse business developers could then design a similar wide table model for the member-domain data and write the member-domain data from the business systems into this wide table model. Subsequently, data analysts and business personnel could use this wide table directly in the data warehouse for data analysis, eliminating the need to retrieve business data from the four systems in the business systems.

[0083] The data warehouse modeling process described above relies on data warehouse developers' understanding of the various subsystems within the online business system. Therefore, developers must be intimately familiar with the data models of the online business system to ensure the accuracy of the modeling results. This consumes significant manpower and time, especially when building data warehouse models for new business systems, which requires extensive business learning and model review.

[0084] The embodiments of this specification can use a data management system to manage the business data of each subsystem, and can use this part of the business data to build a data asset knowledge graph, thereby automatically building a business wide table based on the data asset knowledge graph, and the business wide table can be a data warehouse table.

[0085] Specifically, since the purpose of the data warehouse is to enable users to query the required information faster and more conveniently, and to provide decision support, the target business of the embodiment of this specification, that is, the service business that the business platform can provide to users, includes but is not limited to member purchase business, financial business or wealth management business, etc. The business type is the business type that the user needs to query for decision-making information. In the case that the target business is the member purchase business, the business type includes but is not limited to the member registration business type and member consumption business type in the member purchase business, or the wealth management product consulting business type or wealth management product trading business type in the wealth management business.

[0086] Since the business data related to different business types are stored independently in the business data tables corresponding to each business type, building a business wide table means integrating the data in each business data table to generate a data warehouse wide table. Therefore, when building a data warehouse table, you need to first obtain at least two business data tables to build an initial knowledge graph based on the business data (physical metadata) contained in each business data table, and then update the initial knowledge graph to generate a target knowledge graph.

[0087] In practical applications, the at least two business data tables mentioned above may be stored in a data management system. Therefore, obtaining the at least two business data tables means obtaining the at least two business data tables stored in the data management system.

[0088] In a specific implementation, constructing an initial knowledge graph based on the fields contained in the at least two business data tables includes:

[0089] The table identifier of the target business data table is used as the first node, the field identifiers corresponding to different fields in the target business data table are used as the second node, and the inclusion relationship between the target business data table and the different fields is used as the edge between the first node and the second node to construct an initial sub-knowledge graph corresponding to the target business data table, wherein the target business data table is one of the at least two business data tables, and the initial sub-knowledge graphs corresponding to the at least two business data tables together constitute the initial knowledge graph.

[0090] Specifically, since each business data table is independent of each other, when constructing the initial knowledge graph based solely on the business data tables, it is impossible to obtain the association relationship between the two business data tables. Therefore, it is only possible to construct the initial sub-knowledge graph corresponding to each business data table based on the business data contained in each business data table. Then, the at least two initial knowledge sub-graphs generated constitute the initial knowledge graph. Specifically, the table identifier of the target business data table can be used as the first node, the field identifiers corresponding to the different fields in the target business data table can be used as the second node, and the inclusion relationship between the target business data table and the different fields can be used as the edge between the first node and the second node to construct the initial sub-knowledge graph corresponding to the target business data table.

[0091] The schematic diagram of the initial knowledge graph construction result provided in the embodiment of this specification is as follows Figure 2a As shown, Figure 2a The initial knowledge graph displayed in the figure contains four initial sub-knowledge graphs, where each initial sub-knowledge graph is constructed from a business data table. The node in the middle of the initial sub-knowledge graph is the table identifier, representing the business data table, and the other nodes are the field identifiers of the fields contained in the business data table.

[0092] Step 104 : determining association relationships between different fields in the at least two business data tables based on historical operation data related to the at least two business data tables.

[0093] Specifically, after constructing the initial knowledge graph, the association relationship between the nodes in the initial knowledge graph can be determined based on historical operation data to update the initial knowledge graph.

[0094] Historical operation data can be used to represent historical operational relationships between different fields in the same business data table or between different business data tables. This includes, but is not limited to, operations such as user modifications and updates on a field in a business data table, or associations between two business data tables. Therefore, the associations between different fields in each business data table can be determined based on the historical operation data associated with each business data table.

[0095] For example, after querying the first business data table, the user turns to query the second business data table, which indicates that the first business data table and the second business data table are associated, and the user is associated with the first business data table and the second business data table respectively; for another example, the first business data table stores a user identification field, and the user identification of user U1 is stored under this field; the second business data table stores a commodity field, and the commodity M1 is stored under this field; the historical operation data includes data that user U1 purchased commodity M1. In this case, it can be determined that the user identification field in the first business data table and the commodity field in the second business data table are associated.

[0096] Alternatively, the association relationship between different fields in at least two business data tables can be determined through the field information in at least two business data tables. Specifically, the field information contained in the at least two business data tables can be input into a text processing model to perform similarity calculation, and the association relationship between different fields in the at least two business data tables can be determined based on the similarity calculation results.

[0097] Specifically, after constructing the initial knowledge graph based on the fields contained in at least two business data tables, the field information in each business data table can also be obtained, and knowledge extraction of the field information can be performed through machine learning to determine the association relationship between different fields in each business data table based on the extraction results. Specifically, the field information can be input into the text processing model for semantic recognition, and the similarity of the field information in each business data table can be calculated based on the semantic recognition results. Then, based on the similarity calculation results, it is determined that there is an association relationship between two fields whose similarity is greater than a preset similarity threshold, so that the initial knowledge graph is updated according to the association relationship.

[0098] In actual applications, the above-mentioned historical operation data (operation metadata) can also be stored in the data management system; or, since the above-mentioned at least two business data tables can be stored in the data management system, the field information in the at least two business data tables can be obtained from the data management system to determine the association relationship between different fields in the at least two business data tables based on the similarity between the field information.

[0099] Step 106: Update the initial knowledge graph according to the association relationship to generate a target knowledge graph.

[0100] Specifically, since the initial knowledge graph is constructed and generated by the fields contained in each business data table, and each business data table is independent of each other, the initial knowledge graph constructed and generated only based on the business data tables can only cover limited information. Therefore, the embodiments of this specification can obtain historical operation data and / or business metadata related to each business type, so as to determine the association relationship between different fields in each business data table based on the historical operation data and / or business metadata, and update the initial knowledge graph based on the association relationship to obtain the target knowledge graph, wherein the historical operation data and / or business metadata related to each business type can also be stored in the data management system.

[0101] During specific implementation, the initial knowledge graph is updated according to the association relationship to generate a target knowledge graph. Specifically, when it is determined that the first field in the first business data table and the second field in the second business data table have an association relationship, an edge is constructed between the second node corresponding to the first field and the second node corresponding to the second field in the initial knowledge graph based on the association relationship, and an edge is constructed between the first node of the first business data table and the first node of the second business data table to update the initial knowledge graph and generate a target knowledge graph.

[0102] Specifically, since the schematic diagram of the initial knowledge graph construction result is as follows Figure 2a As shown, Figure 2a In the data structure, there is a connection relationship only between the node corresponding to the table identifier of the business data table and the node corresponding to the field identifier of the field contained in the table. Neither the node corresponding to the table identifier nor the node corresponding to the field identifier has a connection relationship with the node corresponding to the table identifier of other business data tables or the node corresponding to the field identifier of other business data tables.

[0103] Therefore, the embodiments of this specification need to construct a connection relationship between the nodes corresponding to the field identifiers of the fields contained in the two business data tables based on the association relationship between different fields in each business data table. Specifically, when it is determined that there is an association relationship between the first field in the first business data table and the second field in the second business data table, an edge is constructed between the second node corresponding to the first field (the node corresponding to the field identifier of the field) and the second node corresponding to the second field in the initial knowledge graph based on the association relationship, and an edge is constructed between the first node of the first business data table (the node corresponding to the table identifier of the business data table) and the first node of the second business data table. Alternatively, when it is determined that there is an association relationship between the first field and the second field in the same business data table, an edge is constructed between the second node of the first field and the second node of the second field in the business data table to update the initial knowledge graph based on the connection relationship to generate a target knowledge graph.

[0104] The schematic diagram of the target knowledge graph provided in the embodiment of this specification is as follows Figure 2b As shown, Figure 2a Compared with the initial knowledge graph in Figure 2b The target knowledge graph in the has been added with the connection relationship between the nodes corresponding to the field identifiers of the fields in the same business data table or different business data tables.

[0105] Continuing with the above example, if the user identifier U1 contained in the user field in the first business data table purchases the product M1 contained in the product field in the second business data table, an edge can be established between the node corresponding to the user identifier U1 and the node corresponding to the product M1; similarly, if it is determined that there is a blood relationship, computational relationship, etc. between any two nodes, an edge can be established between the two nodes.

[0106] The embodiments of this specification update the association relationships between different nodes in the initial knowledge graph based on historical operation data, which can achieve automatic or semi-automatic construction of the knowledge graph to a certain extent, which is conducive to improving the construction efficiency of the target knowledge graph and ensuring the accuracy of the construction results of the target knowledge graph.

[0107] Alternatively, determining the association relationship between different fields in the at least two business data tables based on historical operation data related to the at least two business data tables includes:

[0108] A first association relationship between the at least two business data tables and a second association relationship between different fields in the at least two business data tables are determined based on historical operation data related to the at least two business data tables.

[0109] Furthermore, the initial knowledge graph is updated according to the association relationship to generate a target knowledge graph, including:

[0110] When it is determined that a first association relationship exists between the first business data table and the second business data table, an edge is constructed between a first node of the first business data table and a first node of the second business data table in the initial knowledge graph based on the first association relationship;

[0111] When it is determined that a second association relationship exists between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the second association relationship to update the initial knowledge graph and generate a target knowledge graph.

[0112] Specifically, historical operation data can be used to characterize historical operational relationships between different fields within the same business data table or between different business data tables. This includes, but is not limited to, user operations such as modifying or updating a field in a business data table, or associations between two business data tables. Therefore, based on the historical operation data associated with each business data table, it is possible to determine associations between business data tables, as well as associations between different fields within the same business data table or between different business data tables.

[0113] For example, after querying the first business data table, the user turns to query the second business data table, which indicates that the first business data table and the second business data table are associated, and the user is associated with the first business data table and the second business data table respectively; for another example, the user performs an association operation on the first business data table and the second business data table at a certain historical time point, which indicates that the first business data table and the second business data table are associated; or, the historical operation data includes data that user U1 purchased product M1. In this case, it can be determined that the user identification field in the first business data table and the product field in the second business data table are associated.

[0114] Therefore, based on the historical operation data related to the at least two business data tables, the association relationship between the at least two business data tables and between different fields in the at least two business data tables can be determined to update the initial knowledge graph based on the association relationship.

[0115] Among them, the initial knowledge graph is updated. Specifically, when it is determined that there is an association relationship between the first business data table and the second business data table, an edge is constructed in the initial knowledge graph between the first node of the first business data table and the first node of the second business data table based on the association relationship. When it is determined that there is a second association relationship between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the association relationship, so as to update the initial knowledge graph and generate a target knowledge graph.

[0116] In practical applications, when constructing an edge between the first node of a first business data table and the first node of a second business data table, one can first determine the domain knowledge that may exist between the first and second business data tables based on the association relationship between the first and second business data tables and in conjunction with the target business. The domain knowledge between business data tables may differ for different target businesses. For example, if the target business is e-commerce, the business data tables may include membership tables, transaction tables, and logistics tables. If the association relationship between these three tables is determined based on historical operation data (user shopping data or user operation data related to the three tables), the domain knowledge between the three tables can be determined to be "transaction-related." If the target business is finance, the domain knowledge between the business data tables can be "loan-related."

[0117] After determining the domain knowledge between the first business data table and the second business data table, an edge can be constructed between the first node of the first business data table and the first node of the second business data table, and the domain knowledge can be added to the initial knowledge graph as attribute information of the edge to realize the update of the knowledge graph.

[0118] In addition, for the node corresponding to the field identifier in the initial knowledge graph, that is, the second node, if it is determined that there is an association relationship between the first field and the second field in the business data table, an edge between the second node corresponding to the first field and the second node corresponding to the second field can be constructed in the initial knowledge graph, and the association relationship can be added to the initial knowledge graph as attribute information of the edge to update the initial knowledge graph and generate the target knowledge graph.

[0119] The embodiments of this specification combine the knowledge graph and data asset management fields, breaking the traditional storage and organization methods of data assets, and storing the association relationship between the business data table in the data asset and the fields in the business data table in the initial knowledge graph. The business metadata and / or operation metadata in the data management system are used to summarize the domain knowledge of the target business and the relationship information between different fields in the business data table, and then the domain knowledge and relationship information are introduced into the initial knowledge graph to construct a complete data asset knowledge graph belonging to an individual or enterprise. It can realize the automation or semi-automatic construction of the knowledge graph to a certain extent, which is conducive to improving the construction efficiency of the target knowledge graph and ensuring the accuracy of the construction results of the target knowledge graph.

[0120] Step 108: construct a business wide table of the target business based on the target knowledge graph.

[0121] Specifically, after generating the target knowledge graph, a business wide table of the target business can be automatically constructed based on the target knowledge graph. The business wide table can be a data warehouse table.

[0122] In specific implementation, the business wide table includes a data warehouse table;

[0123] Accordingly, the step of constructing the business wide table of the target business based on the target knowledge graph includes:

[0124] Use the field identifiers corresponding to each node in the target knowledge graph as fields to construct the initial data warehouse table;

[0125] Adjust the field positions in the initial data warehouse table according to the association relationship between the nodes in the target knowledge graph to generate an intermediate data warehouse table;

[0126] The business data in the at least two business data tables are added to the data units of the corresponding fields in the intermediate data warehouse table to generate a target data warehouse table.

[0127] Specifically, since the data warehouse table is composed of different fields, when constructing the target data warehouse table, the field identifiers corresponding to each node in the target knowledge graph can be used as the fields of the initial data warehouse table, and added to the initial data warehouse table to construct the initial data warehouse table. Then, according to the association relationship between the nodes in the target knowledge graph, the front and back positions of the fields in the initial data warehouse table can be adjusted. For example, if field Z1 is located in the first column of the initial data warehouse table, and field Z2 is located in the tenth column of the initial data warehouse table, but according to the target knowledge graph, it can be determined that there is an association relationship between field Z1 and field Z2, then field Z2 can be adjusted from the tenth column to the second column, but the specific adjustment method can be determined according to actual needs and is not limited here.

[0128] After these adjustments, the intermediate target warehouse table is generated. You can then add the business data from the business data tables to the data cells under the corresponding fields in the intermediate data warehouse table to generate the target data warehouse table. For example, if the first column in the intermediate data warehouse table is the member ID, you can add the member ID contained in each business data table to the data cells contained in the first column.

[0129] Among them, the business data in the at least two business data tables are added to the data units of the corresponding fields in the intermediate data warehouse table to generate the target data warehouse table. Specifically, the mapping relationship between the target business data table in the at least two business data tables and the fields in the intermediate data warehouse table can be determined first, wherein the target business data table is one of the at least two business data tables. Based on the table structure of the intermediate data warehouse table, the business data in the target business data table is added to the data units of the corresponding fields in the intermediate data warehouse table according to the mapping relationship to generate the target data warehouse table.

[0130] Specifically, since different business data tables contain different fields, when constructing the intermediate knowledge graph, a mapping relationship between the business data table and the fields in the intermediate data warehouse table can be established based on the inclusion relationship between the business data table and the fields. This mapping relationship is used to represent the storage location of the business data in the business data table in the data warehouse table.

[0131] In addition, one of the table structures of the intermediate data warehouse table is shown in Table 1.

[0132] Table 1

[0133] Member ID KEY1 KEY2 …… KEYn …… 1 V1 V2 …… …… …… 2 …… …… …… Vn …… …… …… …… …… …… ……

[0134] Therefore, after generating the intermediate data warehouse table, the mapping relationship between the target business data table in at least two business data tables and the fields in the intermediate data warehouse table can be determined first, and based on the table structure of the intermediate data warehouse table, the business data in the target business data table is added to the data unit of the corresponding field in the intermediate data warehouse table according to the mapping relationship to generate the target data warehouse table.

[0135] The embodiments of this specification generate a target knowledge graph by constructing an initial knowledge graph based on the fields in the business data table, and updating the association relationships between different nodes in the initial knowledge graph based on historical operation data, so that a data warehouse table can be constructed based on the target knowledge graph, which can realize automation or semi-automation of data warehouse modeling to a certain extent, which is conducive to reducing the cost required for constructing data warehouse tables, improving the construction efficiency of data warehouse tables, and ensuring the accuracy of construction results.

[0136] Alternatively, constructing a business wide table of the target business based on the target knowledge graph includes:

[0137] An enhanced entity relationship graph is constructed based on the target knowledge graph, and a business wide table of the target business is constructed according to the association relationship between different entities in the enhanced entity relationship graph.

[0138] The step of constructing an enhanced entity relationship graph based on the target knowledge graph includes:

[0139] Determine a first field and a second field in a correlation relationship between the at least two business data tables;

[0140] Deduplication processing is performed on the business data contained in the first field and the second field;

[0141] Determine the amount of business data contained in the first field and the second field according to the deduplication processing result, and divide the at least two business data tables into a master table and a slave table according to the data amount;

[0142] Determine the association relationship between the master table and the slave table according to the association relationship between the nodes in the target knowledge graph;

[0143] Based on the association relationship between the master table and the slave table, an enhanced entity relationship diagram is constructed.

[0144] Furthermore, according to the data volume, the at least two business data tables are divided into a master table and a slave table, including:

[0145] The business data table to which the target field containing the data amount greater than the preset data amount threshold in the first field and the second field belongs is divided into a master table, and the business data tables other than the master table in the at least two business data tables are divided into slave tables;

[0146] Accordingly, determining the association relationship between the master table and the slave table according to the association relationship between the nodes in the target knowledge graph includes:

[0147] According to the association relationship between each node in the target knowledge graph, the association relationship between the main table and each slave table is determined.

[0148] Specifically, the enhanced relational entity diagram, namely the Enhanced-ER model.

[0149] In the target knowledge graph, due to the introduction of business metadata and operation metadata, there are more associations between fields. In this case, the embodiment of this specification can determine whether there is an association between two business data tables based on the association, and thus construct an enhanced entity relationship diagram based on the association between the business data tables.

[0150] Among them, in the specific enhanced entity relationship diagram construction process, the first field and the second field that have an association relationship in at least two business data tables can be determined, that is, it is determined that in at least two business data tables, the first field in the first business data table and the second field in the second business data table have an association relationship, and then the business data contained in the first field and the second field can be deduplicated, and the data volume of the business data contained in the first field and the second field can be determined based on the deduplication result, and the business data table where the field containing the larger amount of business data in the first field and the second field (the target field, the target field is the first field or the second field) is located is used as the main table, and the other business data tables are used as slave tables, and then according to the association relationship between the nodes in the target knowledge graph, it is determined whether there is an association relationship between the main table and each slave table, so as to construct an enhanced entity relationship diagram based on the association relationship. For example, if there is a connection relationship between a node in the main table B1 and a node in a slave table B2, it is determined that the main table B1 and the slave table B2 have an association relationship; if there is a connection relationship between a node in the slave table B2 and a node in the slave table B3, but there is no connection relationship between the node in the main table B1 and any node in the slave table B3, it can still be determined that the main table B1 and the slave table B3 have an association relationship.

[0151] By displaying the product visualization results of the enhanced ER diagram built based on the target knowledge graph, it can be seen from the corresponding display results that there is a relationship between the originally unrelated business data tables, and this relationship can be derived from the field join relationship in the data management system. As long as there is a similar learning and collection mechanism, the knowledge graph can construct the corresponding relationship.

[0152] Based on the above enhanced ER capabilities based on the data asset knowledge graph, the embodiment of this specification can reverse model the membership tables of the four subsystems with associated fields, thereby forming a business wide table with ods_huiyuan_t1 as the driving table (main table), the uid_1 field of the ods_huiyuan_t1 table is associated with the uid_2 field of ods_huiyuan_t2, the uid_3 field of ods_huiyuan_t3, and the uid_4 field of ods_huiyuan_t4, which is consistent with the wide table constructed by manual modeling of the previous data warehouse, thereby realizing the function of automatically constructing business wide tables based on the data asset knowledge graph.

[0153] In addition, after constructing the business wide table of the target business based on the target knowledge graph, it also includes:

[0154] Receive a data query instruction, wherein the data query instruction carries a field identifier of a target field to be queried and a table identifier of a target business data table to which the field identifier belongs;

[0155] Using the table identifier as index information, and performing data index processing according to the mapping relationship between the table identifier and the target field in the business wide table;

[0156] Output the index results as data query results.

[0157] Specifically, since the data warehouse is a strategic collection that provides all types of data support for the user's decision-making process, it enables users to query the required information faster and more conveniently and provide decision support for users. Therefore, after generating the business wide table, that is, the target data warehouse table, it can receive the user's data query instructions, and use the table identifier of the target business data table in the data query instruction as index information. According to the mapping relationship between the table identifier and each field in the target data warehouse table, the business data under the target field is indexed and the index results are output.

[0158] Still taking the target business as a transaction business as an example, if the data query instruction is to query the business data under the member ID field in the business data table B1, the table identifier "B1" is used as the index information, and data index processing is performed based on the mapping relationship between the B table identifier "B1" and the member ID field in the target data warehouse table, and the index results (1, 2) are output.

[0159] Providing data query services to users through the target data warehouse table is conducive to providing users with the convenience of multi-category data query.

[0160] In addition, after constructing a business wide table based on the target knowledge graph, when it is detected that there is incremental data in the at least two business data tables, the business wide table is updated based on the incremental data.

[0161] The embodiments of this specification utilize the association between the business domains of the data asset knowledge graph and metadata to successfully and automatically construct a business wide table under the business domain, unifying the business data scattered in several independent business systems to form the wide table model required in the data warehouse. This process does not require the intervention of data warehouse modeling developers or the learning of business domain knowledge, and can achieve automation or semi-automation of data warehouse modeling to a certain extent, which is conducive to reducing the cost required to build data warehouse tables, improving the construction efficiency of data warehouse tables, and ensuring the accuracy of construction results.

[0162] One embodiment of the present specification obtains at least two business data tables and constructs an initial knowledge graph based on the fields contained in the at least two business data tables, wherein the at least two business data tables respectively correspond to different business types of the target business, and determines the association relationship between different fields in the at least two business data tables based on historical operation data related to the at least two business data tables, updates the initial knowledge graph based on the association relationship, generates a target knowledge graph, and constructs a business wide table of the target business based on the target knowledge graph.

[0163] The embodiments of this specification generate a target knowledge graph by constructing an initial knowledge graph based on the fields in the business data table, and updating the association relationships between different nodes in the initial knowledge graph based on historical operation data, so that a business wide table can be constructed based on the target knowledge graph. To a certain extent, the construction of the business wide table can be automated or semi-automated, which is beneficial to reducing the cost required for the construction of the business wide table, improving the construction efficiency of the business wide table, and ensuring the accuracy of the construction results.

[0164] The following combined Figure 3 , taking the application of the data processing method provided in this specification in member shopping as an example, the data processing method is further explained. Figure 3 A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0165] Step 302: Obtain at least two member data tables stored in the data management system, wherein the at least two member data tables correspond to different business types.

[0166] Step 304: Build an initial sub-knowledge graph corresponding to the target member data table, using the table identifier of the target member data table as the first node, the field identifiers corresponding to the different fields in the target member data table as the second node, and the inclusion relationship between the target member data table and the different fields as the edge between the first node and the second node. The target business data table is one of the at least two business data tables.

[0167] Step 306: The initial sub-knowledge graphs corresponding to at least two member data tables are combined to form an initial knowledge graph.

[0168] Step 308: Determine the association relationship between different fields in the at least two member data tables based on the historical operation data related to the at least two member data tables stored in the data management system.

[0169] Step 310, when it is determined that there is an association relationship between the first field and the second field in at least two member data tables, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the association relationship to update the initial knowledge graph and generate a target knowledge graph.

[0170] Step 312: any one of the at least two member data tables whose data volume is greater than a preset data volume threshold is divided into a master table, and the other member data tables other than the master table are divided into slave tables.

[0171] Step 314: Determine the association relationship between the master table and each slave table based on the association relationship between each node in the target knowledge graph, and construct an enhanced entity relationship graph based on the association relationship between the master table and each slave table.

[0172] Step 316: construct a target data warehouse table based on the association relationships between different entities in the enhanced entity relationship diagram.

[0173] The embodiments of this specification generate a target knowledge graph by constructing an initial knowledge graph based on the fields in the member data table, and updating the association relationships between different nodes in the initial knowledge graph based on historical operation data, so that a data warehouse table can be constructed based on the target knowledge graph, which can realize automation or semi-automation of data warehouse modeling to a certain extent, which is conducive to reducing the cost required for constructing data warehouse tables, improving the construction efficiency of data warehouse tables, and ensuring the accuracy of construction results.

[0174] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of this specification. Figure 4 As shown, the device includes:

[0175] An acquisition module 402 is configured to acquire at least two business data tables and construct an initial knowledge graph based on fields contained in the at least two business data tables, wherein the at least two business data tables respectively correspond to different business types of the target business;

[0176] The determining module 404 is configured to determine the association relationship between different fields in the at least two business data tables based on the historical operation data related to the at least two business data tables;

[0177] A generation module 406 is configured to update the initial knowledge graph according to the association relationship to generate a target knowledge graph;

[0178] The construction module 408 is configured to construct a business-wide table of the target business based on the target knowledge graph.

[0179] Optionally, the acquisition module 402 is further configured to:

[0180] The table identifier of the target business data table is used as the first node, the field identifiers corresponding to different fields in the target business data table are used as the second node, and the inclusion relationship between the target business data table and the different fields is used as the edge between the first node and the second node to construct an initial sub-knowledge graph corresponding to the target business data table, wherein the target business data table is one of the at least two business data tables, and the initial sub-knowledge graphs corresponding to the at least two business data tables together constitute the initial knowledge graph.

[0181] Optionally, the generating module 406 is further configured to:

[0182] When it is determined that there is an association relationship between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the association relationship, and an edge is constructed between the first node of the first business data table and the first node of the second business data table to update the initial knowledge graph and generate a target knowledge graph.

[0183] Optionally, the determining module 404 is further configured to:

[0184] A first association relationship between the at least two business data tables and a second association relationship between different fields in the at least two business data tables are determined based on historical operation data related to the at least two business data tables.

[0185] Optionally, the generating module 406 is further configured to:

[0186] When it is determined that a first association relationship exists between the first business data table and the second business data table, an edge is constructed between a first node of the first business data table and a first node of the second business data table in the initial knowledge graph based on the first association relationship;

[0187] When it is determined that a second association relationship exists between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the second association relationship to update the initial knowledge graph and generate a target knowledge graph.

[0188] Optionally, the business wide table includes a data warehouse table;

[0189] Accordingly, the construction module 408 is further configured to:

[0190] Use the field identifiers corresponding to each node in the target knowledge graph as fields to construct the initial data warehouse table;

[0191] Adjust the field positions in the initial data warehouse table according to the association relationship between the nodes in the target knowledge graph to generate an intermediate data warehouse table;

[0192] The business data in the at least two business data tables are added to the data units of the corresponding fields in the intermediate data warehouse table to generate a target data warehouse table.

[0193] Optionally, the building module 408 is further configured to:

[0194] An enhanced entity relationship graph is constructed based on the target knowledge graph, and a business wide table of the target business is constructed according to the association relationship between different entities in the enhanced entity relationship graph.

[0195] Optionally, the building module 408 is further configured to:

[0196] Determine a first field and a second field in a correlation relationship between the at least two business data tables;

[0197] Deduplication processing is performed on the business data contained in the first field and the second field;

[0198] Determine the amount of business data contained in the first field and the second field according to the deduplication processing result, and divide the at least two business data tables into a master table and a slave table according to the data amount;

[0199] Determine the association relationship between the master table and the slave table according to the association relationship between the nodes in the target knowledge graph;

[0200] Based on the association relationship between the master table and the slave table, an enhanced entity relationship diagram is constructed.

[0201] Optionally, the building module 408 is further configured to:

[0202] The business data table to which the target field containing the data amount greater than the preset data amount threshold in the first field and the second field belongs is divided into a master table, and the business data tables other than the master table in the at least two business data tables are divided into slave tables;

[0203] Accordingly, determining the association relationship between the master table and the slave table according to the association relationship between the nodes in the target knowledge graph includes:

[0204] According to the association relationship between each node in the target knowledge graph, the association relationship between the main table and each slave table is determined.

[0205] Optionally, the building module 408 is further configured to:

[0206] Determining a mapping relationship between a target business data table in the at least two business data tables and each field in the intermediate data warehouse table, wherein the target business data table is one of the at least two business data tables;

[0207] Based on the table structure of the intermediate data warehouse table, the business data in the target business data table is added to the data unit of the corresponding field in the intermediate data warehouse table according to the mapping relationship to generate the target data warehouse table.

[0208] Optionally, the data processing device further includes a query module configured to:

[0209] Receive a data query instruction, wherein the data query instruction carries a field identifier of a target field to be queried and a table identifier of a target business data table to which the field identifier belongs;

[0210] Using the table identifier as index information, and performing data index processing according to the mapping relationship between the table identifier and the target field in the business wide table;

[0211] Output the index results as data query results.

[0212] Optionally, the data processing device further includes a processing module configured to:

[0213] When it is detected that there is incremental data in the at least two business data tables, the business wide table is updated based on the incremental data.

[0214] Optionally, the data processing device further includes an input module configured to:

[0215] The field information contained in the at least two business data tables is input into a text processing model to perform similarity calculation, and the association relationship between different fields in the at least two business data tables is determined based on the similarity calculation result.

[0216] The above is a schematic diagram of a data processing device according to this embodiment. It should be noted that the technical solution of the data processing device and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the above-mentioned data processing method.

[0217] Figure 5 The block diagram of a computing device 500 according to one embodiment of the present disclosure is shown. Components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0218] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)), whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0219] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 5 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0220] Computing device 500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 500 can also be a mobile or stationary server.

[0221] The processor 520 is configured to execute the following computer-executable instructions, which implement the steps of the above-mentioned data processing method when executed by the processor.

[0222] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data processing method are of the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the above-mentioned data processing method.

[0223] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0224] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned data processing method.

[0225] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0226] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above-mentioned data processing method are of the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the above-mentioned data processing method.

[0227] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0228] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0229] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0230] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0231] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Obtain at least two business data tables stored in a data management system, and construct an initial knowledge graph based on fields contained in the at least two business data tables according to domain knowledge and relationship information, wherein the at least two business data tables respectively correspond to different business types of the target business; determining, based on historical operation data related to the at least two business data tables stored in a data management system, association relationships between different fields in the at least two business data tables; Update the initial knowledge graph according to the association relationship to generate a target knowledge graph; Constructing a business wide table of the target business based on the target knowledge graph; The constructing of an initial knowledge graph based on the fields contained in the at least two business data tables includes: The table identifier of the target business data table is used as the first node, the field identifiers corresponding to different fields in the target business data table are used as the second node, and the inclusion relationship between the target business data table and the different fields is used as the edge between the first node and the second node to construct an initial sub-knowledge graph corresponding to the target business data table, wherein the target business data table is one of the at least two business data tables, and the initial sub-knowledge graphs corresponding to the at least two business data tables together constitute an initial knowledge graph; The determining, based on historical operation data related to the at least two business data tables and stored in the data management system, the association relationship between different fields in the at least two business data tables includes: determining, based on historical operation data associated with the at least two business data tables stored in a data management system, a first association relationship between the at least two business data tables and a second association relationship between different fields in the at least two business data tables; The updating of the initial knowledge graph according to the association relationship to generate a target knowledge graph includes: When it is determined that a first association relationship exists between the first business data table and the second business data table, an edge is constructed between a first node of the first business data table and a first node of the second business data table in the initial knowledge graph based on the first association relationship; When it is determined that a second association relationship exists between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the second association relationship to update the initial knowledge graph and generate a target knowledge graph.

2. The data processing method according to claim 1, wherein updating the initial knowledge graph according to the association relationship to generate a target knowledge graph comprises: When it is determined that there is an association relationship between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the association relationship, and an edge is constructed between the first node of the first business data table and the first node of the second business data table to update the initial knowledge graph and generate a target knowledge graph.

3. The data processing method according to any one of claims 1 to 2, wherein the business wide table comprises a data warehouse table; Accordingly, the step of constructing the business wide table of the target business based on the target knowledge graph includes: Use the field identifiers corresponding to each node in the target knowledge graph as fields to construct the initial data warehouse table; Adjust the position of the fields in the initial knowledge graph according to the association relationship between the nodes in the target knowledge graph to generate an intermediate data warehouse table; The business data in the at least two business data tables are added to the data units of the corresponding fields in the intermediate data warehouse table to generate a target data warehouse table.

4. The data processing method according to claim 1, wherein constructing a business wide table of the target business based on the target knowledge graph comprises: An enhanced entity relationship graph is constructed based on the target knowledge graph, and a business wide table of the target business is constructed according to the association relationship between different entities in the enhanced entity relationship graph.

5. The data processing method according to claim 4, wherein constructing an enhanced entity relationship graph based on the target knowledge graph comprises: Determine a first field and a second field in a correlation relationship between the at least two business data tables; Deduplication processing is performed on the business data contained in the first field and the second field; Determine the amount of business data contained in the first field and the second field according to the deduplication processing result, and divide the at least two business data tables into a master table and a slave table according to the data amount; Determine the association relationship between the master table and the slave table according to the association relationship between the nodes in the target knowledge graph; Based on the association relationship between the master table and the slave table, an enhanced entity relationship diagram is constructed.

6. The data processing method according to claim 5, wherein dividing the at least two business data tables into a master table and a slave table according to the data volume comprises: The business data table to which the target field containing the data amount greater than the preset data amount threshold in the first field and the second field belongs is divided into a master table, and the business data tables other than the master table in the at least two business data tables are divided into slave tables; Accordingly, determining the association relationship between the master table and the slave table according to the association relationship between the nodes in the target knowledge graph includes: According to the association relationship between each node in the target knowledge graph, the association relationship between the main table and each slave table is determined.

7. The data processing method according to claim 3, wherein adding the business data in the at least two business data tables to the data units of the corresponding fields in the intermediate data warehouse table to generate the target data warehouse table comprises: Determining a mapping relationship between a target business data table in the at least two business data tables and each field in the intermediate data warehouse table, wherein the target business data table is one of the at least two business data tables; Based on the table structure of the intermediate data warehouse table, the business data in the target business data table is added to the data unit of the corresponding field in the intermediate data warehouse table according to the mapping relationship to generate the target data warehouse table.

8. The data processing method according to claim 1, further comprising: The field information contained in the at least two business data tables is input into a text processing model to perform similarity calculation, and the association relationship between different fields in the at least two business data tables is determined based on the similarity calculation result.

9. A data processing device comprising: an acquisition module configured to acquire at least two business data tables stored in a data management system, and construct an initial knowledge graph based on the fields contained in the at least two business data tables according to domain knowledge and relationship information, wherein the at least two business data tables respectively correspond to different business types of the target business; a determining module configured to determine association relationships between different fields in the at least two business data tables based on historical operation data related to the at least two business data tables stored in the data management system; A generation module is configured to update the initial knowledge graph according to the association relationship to generate a target knowledge graph; A construction module is configured to construct a business wide table of the target business based on the target knowledge graph; The determining, based on historical operation data related to the at least two business data tables, the association relationship between different fields in the at least two business data tables includes: determining, based on historical operation data associated with the at least two business data tables, a first association relationship between the at least two business data tables and a second association relationship between different fields in the at least two business data tables; The updating of the initial knowledge graph according to the association relationship to generate a target knowledge graph includes: When it is determined that a first association relationship exists between the first business data table and the second business data table, an edge is constructed between a first node of the first business data table and a first node of the second business data table in the initial knowledge graph based on the first association relationship; When it is determined that a second association relationship exists between the first field in the first business data table and the second field in the second business data table, an edge is constructed in the initial knowledge graph between the second node corresponding to the first field and the second node corresponding to the second field based on the second association relationship to update the initial knowledge graph and generate a target knowledge graph.

10. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 8.

Images