Data relationship determination method, apparatus, device, and storage medium

Abstract

The application discloses a data relationship determination method, comprising the following steps: acquiring a plurality of first data tables in real time, and analyzing to-be-classified data in the plurality of first data tables to obtain a plurality of second data tables; the second data tables change according to changes of the first data tables; determining the weight of each second data table; determining a first network relationship graph formed by the plurality of second data tables according to the weight; determining a data screening model according to the weight and the first network relationship graph; establishing a third data table according to the data screening model, and associating the third data table with the plurality of second data tables to form a second network relationship graph, so as to realize the update of the data relationship. The application realizes the update of the data relationship, and realizes the determination and automatic update of the logical relationship between data resources.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and storage medium for determining data relationships. Background Technology

[0002] In existing technologies, the processing of big data relies on manual pre-understanding of the logical relationships between data tables and the data format of real-time data in the tables. When new data resources enter the system, the system cannot understand the logical relationships and business significance of the new data resources. At the same time, the method of screening data based on manually defined data models also has limitations and cannot realize the automatic iterative update of the model based on data resources. Summary of the Invention

[0003] Based on this, it is necessary to propose a method, apparatus, device, and storage medium for determining data relationships to address the above problems.

[0004] A method for determining data relationships, the method comprising:

[0005] Multiple first data tables are acquired in real time, and the data to be classified in the multiple first data tables is parsed to obtain multiple second data tables; the second data tables change according to the changes in the first data tables.

[0006] Determine the weight of each of the second data tables;

[0007] A first network relationship diagram composed of the plurality of second data tables is determined based on the weights;

[0008] The data screening model is determined based on the weights and the first network relationship diagram;

[0009] A third data table is established based on the data screening model, and the third data table is associated with the multiple second data tables to form a second network relationship diagram, so as to realize the updating of the data relationship.

[0010] In one embodiment, the plurality of first data tables are respectively data tables composed of data from relational databases, data tables composed of data from non-relational databases, data tables composed of data from streaming real-time databases, data tables composed of data from message databases, and data tables composed of data from data warehouses.

[0011] In one embodiment, the data relationship determination method further includes: obtaining the first data table from the relational database, non-relational database, streaming real-time database, message database, and data warehouse through a driver or third-party application package.

[0012] In one embodiment, parsing the data to be classified from multiple data tables to obtain multiple second data tables includes:

[0013] The data in the first data table is categorized according to value type to obtain the second data table.

[0014] In one embodiment, the value type includes: numeric, floating-point, date, time, character, and monetary types.

[0015] In one embodiment, determining the weight of each of the second data tables includes:

[0016] There are connection relationships between multiple second data tables. The number of times each second data table is linked to other second data tables is determined. Each time it is linked, the weight is increased by 1. If a linked second data table is also linked to other second data tables, the weight of the parent table is increased by 0.1. The parent table is the second data table that links to other second data tables.

[0017] In one embodiment, the second network diagram is a mesh structure diagram, and the plurality of third data tables are connected by connecting lines, the connecting lines being marked with the weights of the connected third data tables; the weights change according to the changes in the data in the first data table.

[0018] A data relationship determination device, comprising:

[0019] The parsing module is used to acquire multiple first data tables in real time and parse the data to be classified in the multiple data tables to obtain multiple second data tables.

[0020] A calculation module is used to determine the weight of each of the second data tables;

[0021] A determining module is used to determine a first network relationship diagram composed of the plurality of second data tables based on the weights;

[0022] The modeling module is used to determine the data screening model based on the weights and the network relationship diagram;

[0023] An update module is used to establish a third data table based on the data screening model, and associate the third data table with the multiple second data tables to form a second network relationship diagram, so as to update the data relationship.

[0024] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps:

[0025] Multiple first data tables are acquired in real time, and the data to be classified in the multiple data tables is parsed to obtain multiple second data tables; the second data tables change according to the changes in the first data tables.

[0026] Determine the weight of each of the second data tables;

[0027] A first network relationship diagram composed of the plurality of second data tables is determined based on the weights;

[0028] The data screening model is determined based on the weights and the network diagram.

[0029] A third data table is established based on the data screening model, and the third data table is associated with the multiple second data tables to form a second network relationship diagram, so as to realize the updating of the data relationship.

[0030] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0031] Multiple first data tables are acquired in real time, and the data to be classified in the multiple data tables is parsed to obtain multiple second data tables; the second data tables change according to the changes in the first data tables.

[0032] Determine the weight of each of the second data tables;

[0033] A first network relationship diagram composed of the plurality of second data tables is determined based on the weights;

[0034] The data screening model is determined based on the weights and the network diagram.

[0035] A third data table is established based on the data screening model, and the third data table is associated with the multiple second data tables to form a second network relationship diagram, so as to realize the updating of the data relationship.

[0036] This application parses the data to be classified in a first data table acquired in real time to obtain multiple second data tables; determines the weight of each second data table; and establishes a third data table based on the weight and the data screening model determined by the first network relationship diagram, and associates the third data table with the multiple second data tables to form a second network relationship diagram to update the data relationship; thus realizing the determination and automatic updating of the logical relationship between data resources. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] in:

[0039] Figure 1 This is a diagram illustrating the application environment of a data relationship determination method in one embodiment.

[0040] Figure 2 This is a flowchart of a data relationship determination method in one embodiment;

[0041] Figure 3 This is a structural block diagram of a data relationship determination device in one embodiment;

[0042] Figure 4 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Figure 1 This is a diagram illustrating the application environment of the data relationship determination method in one embodiment. (Refer to...) Figure 1 This data relationship determination method is applied to a data relationship determination system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. The terminal 110 provides a first data table, and the server 120 acquires multiple first data tables in real time and parses the data to be classified in the multiple first data tables to obtain multiple second data tables; the second data tables change according to the changes in the first data tables.

[0045] In existing technologies, the processing of big data relies on manual pre-understanding of the logical relationships between data tables and the data format of real-time data in the tables. When new data resources enter the system, the system cannot understand the logical relationships and business significance of the new data resources. At the same time, the method of screening data based on manually defined data models also has limitations and cannot realize the automatic iterative update of the model based on data resources.

[0046] like Figure 2 As shown, in one embodiment, this application provides a data relationship determination method, which is applied to a server. This embodiment illustrates the application to a server, and the data relationship determination method specifically includes the following steps:

[0047] S10: Acquire multiple first data tables in real time, and parse the data to be classified in the multiple first data tables to obtain multiple second data tables; the second data tables change according to the changes in the first data tables;

[0048] S20: Determine the weight of each of the second data tables;

[0049] S30: Determine the first network relationship diagram composed of the plurality of second data tables according to the weights;

[0050] S40: Determine the data screening model based on the weights and the first network relationship graph; the "automated construction process" mentioned here refers to the computer traversing all possible connection relationship calculation methods based on the "relationship graph" and "value type" described above. For example, based on the connection relationship between two table objects, perform logical calculations on numeric, floating-point, date, time, character, and amount types in the two table objects in various ways such as accumulation, averaging, and counting the number of records, and generate the corresponding data screening model script. The system creates a temporary table based on the generated script, and then uses the created temporary table as a data source to access the "relationship graph" described above. This process is repeated to form the continuous evolution and upgrading of the "relationship graph".

[0051] S50: Establish a third data table based on the data screening model, and associate the third data table with the multiple second data tables to form a second network relationship diagram, so as to update the data relationship.

[0052] This application parses the data to be classified in a first data table acquired in real time to obtain multiple second data tables; determines the weight of each second data table; and establishes a third data table based on the weight and the data screening model determined by the first network relationship diagram, and associates the third data table with the multiple second data tables to form a second network relationship diagram to update the data relationship; thus realizing the determination and automatic updating of the logical relationship between data resources.

[0053] In one embodiment, the plurality of first data tables are respectively data tables composed of data from relational databases, data tables composed of data from non-relational databases, data tables composed of data from streaming real-time databases, data tables composed of data from message databases, and data tables composed of data from data warehouses.

[0054] In one embodiment, the data relationship determination method further includes: obtaining the first data table from the relational database, non-relational database, streaming real-time database, message database, and data warehouse via a driver or third-party application package. The value types include: numeric, floating-point, date, time, character, and monetary types.

[0055] Specifically, this application requires targeted logical relationship analysis based on different types of data resources to understand the meaning of each field or key. Specifically, it derives the importance of the data by examining the value types of the data to be classified in the first data table and the logical relationships between them, and then analyzes this information. The "value types" this method focuses on mainly include: numeric, floating-point, date, time, character, and monetary data types—data types that change in the database. The method monitors the changes in these value types in real time according to the time sequence to discover the changing trends of the data to be classified (i.e., business data) over time. The "logical relationships" between the data to be classified mainly refer to the primary and foreign key relationships between tables, that is, the primary and foreign key relationships between multiple first data tables, used to describe the joining methods between the first data tables. For example, if a primary key or code field of a first data table is associated with several foreign key fields of other first data tables, it indicates that this first data table is of high importance and has many connections with other related first data tables, thus deriving the importance of this first data table in the entire big data system.

[0056] In one embodiment, parsing the data to be classified from multiple data tables to obtain multiple second data tables includes: classifying the data in the first data table according to the value type to obtain the second data tables.

[0057] In one embodiment, determining the weight of each of the second data tables includes:

[0058] There are connection relationships between multiple second data tables. The number of times each second data table is linked to other second data tables is determined. Each time it is linked, the weight is increased by 1. If a linked second data table is also linked to other second data tables, the weight of the parent table is increased by 0.1. The parent table is the second data table that links to other second data tables.

[0059] Specifically, the weight is mainly counted based on the number of other second data tables connected to each second data table. Each time it is connected, the weight is increased by 1. When the linked second data table is connected to other second data tables, the weight of the parent table is increased by 0.1, and so on, until all the connection relationships between the second data tables are traversed.

[0060] In one embodiment, the second network diagram is a mesh structure diagram, and the plurality of third data tables are connected by connecting lines, the connecting lines being marked with the weights of the connected third data tables; the weights change according to the changes in the data in the first data table.

[0061] Specifically, because the data in the first data table acquired in real time constantly changes according to business needs—such as different user permissions, different business scenarios, and different data resources for each unit—the second network relationship graph is updated in real time based on changes in these relationships. This second network relationship graph is a mesh-like structure that connects all the third data tables. Each connection line is marked with a corresponding weight, and the weights on the connections in the second network relationship graph are updated in real time according to changes in business requirements.

[0062] Step S40: Determine the data screening model based on the weights and the first network relationship diagram; specifically, based on the first network relationship diagram and value types mentioned above, the computer traverses all possible connection relationship calculation methods, such as: based on the connection relationship between two second data tables, performing logical calculations on numeric, floating-point, date, time, character, and amount types in the two second data tables in various ways, such as accumulation, averaging, and counting the number of records, and generating a corresponding data screening model script. The system creates a temporary table based on the generated script, and then uses the created temporary table as a data source to access the first network relationship diagram described above, and so on, to form a second network relationship diagram, so as to realize the updating and upgrading of the data relationship.

[0063] Based on the above embodiments, the first network diagram, second network diagram, first data table, second data table, third data table, data to be classified in the first data table, and data screening model obtained in this application can be called by external systems through specific program interfaces to achieve data sharing.

[0064] A data relationship determination device, such as Figure 3 As shown, it includes:

[0065] The parsing module 100 is used to acquire multiple first data tables in real time and parse the data to be classified in the multiple data tables to obtain multiple second data tables.

[0066] The calculation module 200 is used to determine the weight of each of the second data tables;

[0067] The determining module 300 is used to determine the first network relationship diagram composed of the plurality of second data tables according to the weights;

[0068] Modeling module 400 is used to determine a data screening model based on the weights and the network relationship diagram;

[0069] The update module 500 is used to establish a third data table according to the data screening model, and associate the third data table with the multiple second data tables to form a second network relationship diagram, so as to realize the update of the data relationship.

[0070] Figure 4 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 4 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program that, when executed by the processor, enables the processor to implement an age recognition method. The internal memory may also store a computer program that, when executed by the processor, enables the processor to implement the age recognition method. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0071] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps:

[0072] S10: Acquire multiple first data tables in real time, and parse the data to be classified in the multiple first data tables to obtain multiple second data tables; the second data tables change according to the changes in the first data tables;

[0073] S20: Determine the weight of each of the second data tables;

[0074] S30: Determine the first network relationship diagram composed of the plurality of second data tables according to the weights;

[0075] S40: Determine the data screening model based on the weights and the first network relationship diagram;

[0076] S50: Establish a third data table based on the data screening model, and associate the third data table with the multiple second data tables to form a second network relationship diagram, so as to update the data relationship.

[0077] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0078] S10: Acquire multiple first data tables in real time, and parse the data to be classified in the multiple first data tables to obtain multiple second data tables; the second data tables change according to the changes in the first data tables;

[0079] S20: Determine the weight of each of the second data tables;

[0080] S30: Determine the first network relationship diagram composed of the plurality of second data tables according to the weights;

[0081] S40: Determine the data screening model based on the weights and the first network relationship diagram;

[0082] S50: Establish a third data table based on the data screening model, and associate the third data table with the multiple second data tables to form a second network relationship diagram, so as to update the data relationship.

[0083] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for determining data relationships, characterized in that, The method includes: The system acquires multiple first data tables in real time and parses the data to be classified in the multiple first data tables to obtain multiple second data tables, including: classifying the data in the first data tables according to the value type to obtain the second data tables; the second data tables change according to the changes in the first data tables. Determining the weight of each second data table includes: there are connection relationships between multiple second data tables; determining the number of times each second data table is linked to other second data tables; each time it is linked, the weight is increased by 1; if a linked second data table is also linked to other second data tables, the weight of the parent table is increased by 0.1; the parent table is the second data table that links to other second data tables. A first network relationship diagram composed of the plurality of second data tables is determined based on the weights; Determining a data screening model based on the weights and the first network relationship graph includes: dynamically generating a data screening model by traversing all possible connection relationship calculation methods based on the weights and the connection relationships and value type characteristics between data tables in the first network relationship graph; Establishing a third data table based on the data screening model includes: generating an executable script based on the data screening model, creating a third data table based on the script, and associating the third data table with the plurality of second data tables to form a second network relationship graph to achieve the updating of the data relationship.

2. The data relationship determination method according to claim 1, characterized in that, The multiple first data tables are respectively data tables composed of data from relational databases, data tables composed of data from non-relational databases, data tables composed of data from streaming real-time databases, data tables composed of data from message databases, and data tables composed of data from data warehouses.

3. The data relationship determination method according to claim 2, characterized in that, Also includes: The first data table is obtained from the relational database, non-relational database, streaming real-time database, message database, and data warehouse through a driver or third-party application package.

4. The data relationship determination method according to claim 1, characterized in that, The value types include: numeric, floating-point, date, time, character, and amount.

5. The data relationship determination method according to claim 2, characterized in that, The second network diagram is a mesh structure diagram, in which the multiple third data tables are connected by connecting lines, and the weights of the connected third data tables are marked on the connecting lines; the weights change according to the changes in the data in the first data table.

6. A data relationship determination device, characterized in that, include: The parsing module is used to acquire multiple first data tables in real time and parse the data to be classified in the multiple first data tables to obtain multiple second data tables, including: classifying the data in the first data tables according to the value type to obtain the second data tables; the second data tables change according to the changes in the first data tables; The calculation module determines the weight of each second data table, including: there are connection relationships between multiple second data tables, determining the number of times each second data table is linked to other second data tables, and incrementing the weight by 1 for each link; if a linked second data table is also linked to other second data tables, the weight of the parent table is incremented by 0.1; the parent table is the second data table that links to other second data tables. A determining module is used to determine a first network relationship diagram composed of the plurality of second data tables based on the weights; The modeling module is used to determine a data screening model based on the weights and the network relationship graph, including: dynamically generating a data screening model by traversing all possible connection relationship calculation methods based on the weights and the connection relationships and value type characteristics between data tables in the first network relationship graph; An update module is used to establish a third data table based on the data screening model, including: generating an executable script based on the data screening model, creating a third data table based on the script; and associating the third data table with the plurality of second data tables to form a second network relationship diagram to achieve the updating of the data relationship.

7. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method as claimed in any one of claims 1 to 5.

Images