Data repair method and device, electronic equipment, medium and program product

Abstract

The present disclosure provides a knowledge graph-based data repair method and device, electronic equipment, medium and computer program product. The above method and device can be used in the field of artificial intelligence technology. The knowledge graph-based data repair method comprises: acquiring m storage process information in a t-second time period, each storage process information comprising the source and destination of data; extracting superior information, subordinate information and the parent-child relationship between the superior information and the subordinate information according to the m storage process information, wherein the superior information comprises a parent storage process-table-field, and the subordinate information comprises a child storage process-table-field; constructing a knowledge graph according to the superior information, the subordinate information and the parent-child relationship, wherein the nodes of the knowledge graph are constructed according to the superior information and the subordinate information, and the edges of the knowledge graph are constructed according to the parent-child relationship; determining a problem data chain in the knowledge graph according to the problem data in the t-second time period obtained in advance; and repairing the problem data chain.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a data repair method, apparatus, electronic device, medium, and computer program product based on knowledge graphs. Background Technology

[0002] Production data from financial market trading systems flows into Oracle databases, undergoes processing through one or more stored procedures, and is then applied to various business modules and downstream applications. Determining the processing logic of table fields often requires manual analysis and troubleshooting. Furthermore, when errors occur in production data, all related business modules and downstream applications of the financial market trading system will be affected, potentially triggering production problems. The causes of data errors are varied, including program logic errors, network interruptions, or server crashes. Summary of the Invention

[0003] In view of this, the present disclosure provides a knowledge graph-based data repair method, apparatus, electronic device, computer-readable storage medium, and computer program product that are highly intelligent, efficient, and accurate.

[0004] One aspect of this disclosure provides a knowledge graph-based data repair method, comprising: acquiring m stored procedure information within a time period of t seconds, each stored procedure information including the source and destination of data, wherein m is an integer greater than or equal to 1, and t is greater than or equal to 1; extracting superior information, subordinate information, and parent-child relationships between the superior and subordinate information based on the m stored procedure information, wherein the superior information includes a parent stored procedure-table-field, and the subordinate information includes a child stored procedure-table-field; constructing a knowledge graph based on the superior information, subordinate information, and parent-child relationships, wherein the nodes of the knowledge graph are constructed based on the superior information and subordinate information, and the edges of the knowledge graph are constructed based on the parent-child relationships; determining a problem data chain in the knowledge graph based on pre-obtained problem data within the time period of t seconds; and repairing the problem data chain.

[0005] According to the knowledge graph-based data repair method of this disclosure, by extracting superior information, subordinate information, and parent-child relationships from m stored procedure information within a time period of t seconds, a knowledge graph of the stored procedure information within the time period of t seconds can be constructed based on the superior information, subordinate information, and parent-child relationships. Therefore, based on the edge relationships of the knowledge graph, other data associated with the problematic data within the time period of t seconds can be found, thereby enabling the repair of the problematic data and related data. The data repair method of this disclosure has low reliance on manual labor, high degree of intelligence, and high efficiency and accuracy.

[0006] In some embodiments, extracting superior information, subordinate information, and parent-child relationships between the superior and subordinate information based on the m stored procedure information includes: extracting superior information, subordinate information, and parent-child relationships from each of the stored procedure information to obtain intermediate data results; and integrating the m intermediate data results according to integration rules to obtain final data results, wherein the final data results include superior information, subordinate information, and parent-child relationships between the superior and subordinate information from the m stored procedure information.

[0007] In some embodiments, the integration rule includes: merging the m intermediate data results, and in every two intermediate data results, when the table-field in the superior information of one intermediate data result is the same as the table-field in the subordinate information of the other intermediate data result, the subordinate information of the other intermediate data result is used as the superior information in the final data result, and the superior information of one of the intermediate data results is used as the subordinate information.

[0008] In some embodiments, determining the problem data chain in the knowledge graph based on the problem data within the pre-obtained t-second time period includes: matching nodes in the knowledge graph as problem nodes based on the problem data within the pre-obtained t-second time period, wherein the problem data includes a problem table and / or problem fields; and determining the problem data chain using a graph traversal algorithm based on the problem nodes.

[0009] In some embodiments, repairing the problematic data chain includes: identifying the upstream node of the problematic data in the problematic data chain as a node to be confirmed; identifying the problematic data and its downstream nodes in the problematic data chain as nodes to be repaired; and repairing the nodes to be repaired.

[0010] In some embodiments, repairing the node to be repaired includes: writing a custom script for the node to be repaired and / or performing task rescheduling on the node to be repaired.

[0011] Another aspect of this disclosure provides a knowledge graph-based data repair apparatus, comprising: an acquisition module, configured to acquire m stored procedure information within a time period of t seconds, each stored procedure information including the source and destination of data, wherein m is an integer greater than or equal to 1, and t is greater than or equal to 1; an extraction module, configured to extract superior information, subordinate information, and parent-child relationships between the superior and subordinate information based on the m stored procedure information, wherein the superior information includes a parent stored procedure-table-field, and the subordinate information includes a child stored procedure-table-field; a construction module, configured to construct a knowledge graph based on the superior information, subordinate information, and parent-child relationships, wherein the nodes of the knowledge graph are constructed based on the superior information and subordinate information, and the edges of the knowledge graph are constructed based on the parent-child relationships; a determination module, configured to determine a problem data chain in the knowledge graph based on pre-obtained problem data within the time period of t seconds; and a repair module, configured to repair the problem data chain.

[0012] Another aspect of this disclosure provides an electronic device including one or more processors and one or more memories, wherein the memories are used to store executable instructions that, when executed by the processor, implement the method described above.

[0013] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0014] Another aspect of this disclosure provides a computer program product including a computer program comprising computer executable instructions that, when executed, implement the method described above. Attached Figure Description

[0015] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 This illustration schematically shows an exemplary system architecture to which methods and apparatus can be applied according to embodiments of the present disclosure;

[0017] Figure 2 A flowchart illustrating a knowledge graph-based data repair method according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 3 This schematically illustrates a flowchart of extracting parent information, child information, and parent-child relationships between parent and child information based on m stored procedure information, according to an embodiment of the present disclosure.

[0019] Figure 4 A schematic diagram of a knowledge graph according to an embodiment of the present disclosure is shown.

[0020] Figure 5 This illustration schematically shows a flowchart of determining a question data chain in a knowledge graph based on obtained question data, according to an embodiment of the present disclosure;

[0021] Figure 6 A flowchart illustrating the repair of a problematic data link according to an embodiment of this disclosure is shown schematically;

[0022] Figure 7 This schematically illustrates a flowchart of repairing a node to be repaired according to an embodiment of the present disclosure;

[0023] Figure 8 A schematic diagram illustrating the structure of a knowledge graph-based data repair apparatus according to an embodiment of the present disclosure is shown.

[0024] Figure 9 A schematic diagram illustrating the structure of an extraction module according to an embodiment of the present disclosure is shown.

[0025] Figure 10 A schematic diagram illustrating the structure of the determining module according to an embodiment of the present disclosure is shown.

[0026] Figure 11 A schematic diagram illustrating the structure of a repair module according to an embodiment of the present disclosure is shown.

[0027] Figure 12 A schematic diagram illustrating the structure of a repair unit according to an embodiment of the present disclosure is shown.

[0028] Figure 13 A schematic diagram illustrating the structure of a knowledge graph-based data repair apparatus according to an embodiment of the present disclosure is shown.

[0029] Figure 14 This schematically illustrates a workflow diagram of a knowledge graph-based data repair apparatus according to an embodiment of the present disclosure;

[0030] Figure 15 A block diagram of an electronic device according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0031] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0032] In the technical solution disclosed herein, the acquisition, storage, and application of user personal information all comply with relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals. In the technical solution disclosed herein, the acquisition, collection, storage, use, processing, transmission, provision, disclosure, and application of data all comply with relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] When using expressions such as "at least one of A, B, or C," it should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, features defined with "first" or "second" may explicitly or implicitly include one or more of the stated features.

[0035] Production data inflow into financial market trading systems o Oracle databases are processed through one or more stored procedures before being applied to various business modules and downstream applications. Determining the processing logic of table fields often requires manual analysis and troubleshooting. Furthermore, when errors occur in production data, all related business modules and downstream applications within the financial market trading system will be affected, potentially triggering production problems. The causes of data errors are varied, including program logic errors, network interruptions, or server crashes.

[0036] Data errors typically affect certain fields of a table. Stored procedures that only update other fields need to be excluded. The impact of erroneous data needs to be analyzed at the stored procedure, table, and field levels. Current emergency measures for handling data errors involve manual investigation to locate the affected stored procedures and re-execute the corresponding tasks. While this approach can identify field processing logic and repair erroneous data, it relies heavily on staff familiarity with financial market stored procedures, potentially leading to lengthy analysis times and slow data recovery efficiency.

[0037] This disclosure provides a knowledge graph-based data repair method, apparatus, electronic device, computer-readable storage medium, and computer program product. The knowledge graph-based data repair method includes: acquiring m stored procedure information within a t-second time period, each stored procedure information including the data source and destination, where m is an integer greater than or equal to 1, and t is greater than or equal to 1; extracting superior information, subordinate information, and parent-child relationships between the superior and subordinate information based on the m stored procedure information, where superior information includes a parent stored procedure-table-field, and subordinate information includes a child stored procedure-table-field; constructing a knowledge graph based on the superior information, subordinate information, and parent-child relationships, where nodes of the knowledge graph are constructed based on superior and subordinate information, and edges of the knowledge graph are constructed based on parent-child relationships; determining a problem data chain in the knowledge graph based on pre-obtained problem data within the t-second time period; and repairing the problem data chain.

[0038] It should be noted that the knowledge graph-based data repair method, apparatus, electronic device, computer-readable storage medium, and computer program product disclosed herein can be used in the field of artificial intelligence technology, or in any field other than artificial intelligence technology, such as the financial field. The field of this disclosure is not limited here.

[0039] Figure 1 This illustration schematically depicts an exemplary system architecture 100 to which knowledge graph-based data repair methods, apparatuses, electronic devices, computer-readable storage media, and computer program products can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0040] like Figure 1As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0041] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0042] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0043] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0044] It should be noted that the knowledge graph-based data repair method provided in this disclosure can generally be executed by server 105. Correspondingly, the knowledge graph-based data repair device provided in this disclosure can generally be located in server 105. The knowledge graph-based data repair method provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the knowledge graph-based data repair device provided in this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0046] The following will be based on Figure 1 The described scene, through Figures 2-7The knowledge graph-based data repair method according to the embodiments of this disclosure will be described in detail.

[0047] Figure 2 A flowchart illustrating a knowledge graph-based data repair method according to an embodiment of the present disclosure is shown.

[0048] like Figure 2 As shown, the knowledge graph-based data repair method in this embodiment includes operations S210 to S250.

[0049] In operation S210, information on m stored procedures within a time period of t seconds is retrieved. Each stored procedure includes the source and destination of the data, where m is an integer greater than or equal to 1, and t is greater than or equal to 1. For example, stored procedure information for tables and fields within a time period of t seconds can be retrieved from a financial market business database. There can be m stored procedures, and each can include keywords such as INSERT INTO, UPDATE, MERGE, and / or FROM.

[0050] In operation S220, based on information from m stored procedures, extract the parent-child information, the child-child information, and the parent-child relationships between them. The parent-child information includes the parent stored procedure, table, and fields, and the child stored procedure, table, and fields. It can be understood that the parent and child stored procedures can be identified based on the keywords of each stored procedure's information.

[0051] For example, in stored procedure P1, the stored procedure information records INSERT INTO table B (field b) SELECT field a FROM table A, therefore we can determine that the parent level is P1 - table A - field a, and the child level is P1 - table B - field b; In stored procedure P1, the stored procedure information records UPDATE table B SET field b = (SELECT field a FROM table B AWHERE table B.field c = table A.field d), therefore we can determine that the parent level is P1 - table A - field a, and the child level is P1 - table B - field b; In stored procedure P1, the stored procedure information records:

[0052] MERGE INTO Table B

[0053] USING table A ON table B.field c = table A.field d

[0054] WHEN MATCHED THEN

[0055] UPDATE SET tableB.fieldb = tableA.fielda

[0056] WHEN NOT MATCHED THEN

[0057] INSERT(tableB.fieldb) VALUES(tableA.fielda),

[0058] Therefore, we can determine that the superior information is P1-table A-field a, and the subordinate information is P1-table B-field b.

[0059] In operation S230, a knowledge graph is constructed based on superior information, subordinate information, and parent-child relationships. The nodes of the knowledge graph are constructed based on superior and subordinate information, and the edges of the knowledge graph are constructed based on parent-child relationships.

[0060] In operation S240, the problem data chain is determined in the knowledge graph based on the problem data within the pre-obtained time period t seconds. The problem data can be a table or a field in the table. Problem nodes can be found in the knowledge graph through the problematic table and / or field. Then, the problem data chain is determined by finding the nodes and edges that have a direct or indirect relationship with the problem nodes.

[0061] The S250 was used to repair the problematic data link.

[0062] According to the knowledge graph-based data repair method of this disclosure, by extracting superior information, subordinate information, and parent-child relationships from m stored procedure information within a time period of t seconds, a knowledge graph of the stored procedure information within the time period of t seconds can be constructed based on the superior information, subordinate information, and parent-child relationships. Therefore, based on the edge relationships of the knowledge graph, other data associated with the problematic data within the time period of t seconds can be found, thereby enabling the repair of the problematic data and related data. The data repair method of this disclosure has low reliance on manual labor, high degree of intelligence, and high efficiency and accuracy.

[0063] Figure 3 The flowchart illustrates an embodiment of the present disclosure of extracting parent information, child information, and parent-child relationships between parent and child information based on m stored procedure information.

[0064] Operation S220 extracts parent information, child information, and parent-child relationships between parent and child information based on m stored procedure information, including operations S221 and S222.

[0065] In operation S221, extract the parent-child information, the superior-subordinate information, and the parent-child relationship from the information of each stored procedure to obtain intermediate data results.

[0066] In operation S222, m intermediate data results are integrated according to the integration rules to obtain the final data result. The final data result includes the parent information, the child information, and the parent-child relationship between the parent and child information in the m stored procedure information.

[0067] In some specific examples, the integration rule may include merging m intermediate data results, and in every two intermediate data results, when the table-field in the parent information of one intermediate data result is the same as the table-field in the child information of the other intermediate data result, the child information of the other intermediate data result is used as the parent information in the final data result, and the parent information of one of the intermediate data results is used as the child information.

[0068] The following explanation uses stored procedures P1 and P2 as examples. The intermediate data obtained by extracting the parent, child, and parent-child relationships from the stored procedure information of stored procedure P1 is shown in Table 1.

[0069] Table 1

[0070] Parent stored procedure - table - field Sub-stored procedure - table - field P1-T2-F2 P1-T1-F1 P1-T3-F3 P1-T2-F2 P1-T4-F5 P1-T1-F4

[0071] The intermediate data results obtained by extracting the parent, child, and parent-child relationships from the stored procedure information of stored procedure P2 are shown in Table 2.

[0072] Table 2

[0073] Parent stored procedure - table - field Sub-stored procedure - table - field P2-T5-F6 P2-T3-F3 P2-T6-F7 P2-T5-F6 P2-T8-F9 P2-T7-F8 P2-T9-F10 P2-T7-F8

[0074] The intermediate data results obtained from stored procedures P1 and P2 are integrated according to the integration rules, and the final data results are shown in Table 3.

[0075] Table 3

[0076] Parent stored procedure - table - field Sub-stored procedure - table - field P1-T2-F2 P1-T1-F1 P1-T3-F3 P1-T2-F2 P1-T4-F5 P1-T1-F4 P2-T5-F6 P2-T3-F3 P2-T6-F7 P2-T5-F6 P2-T8-F9 P2-T7-F8 P2-T9-F10 P2-T7-F8 P2-T3-F3 P1-T3-F3

[0077] In Tables 1, 2, and 3, P represents a stored procedure, T represents a table, and F represents a field. Operations S221 and S222 facilitate the extraction of parent-child information, child-child information, and parent-child relationships between parent and child information based on information from m stored procedures.

[0078] Based on the data in Table 3, by using the parent stored procedure-table-field and child stored procedure-table-field as nodes, and the parent-child relationship between the parent stored procedure-table-field and child stored procedure-table-field as edges, a structure can be constructed as follows: Figure 4 The knowledge graph shown.

[0079] Figure 5The flowchart illustrating the process of determining a problem data chain in a knowledge graph based on problem data within a pre-obtained time period of t seconds, according to an embodiment of the present disclosure, is shown in the illustration.

[0080] Operation S240 determines the problem data chain in the knowledge graph based on the problem data within a pre-obtained time period of t seconds, including operations S241 and S242.

[0081] In operation S241, nodes in the knowledge graph are matched as problem nodes based on the problem data within a pre-obtained time period of t seconds. The problem data includes a problem table and / or problem fields.

[0082] In operation S242, based on the problem nodes, a graph traversal algorithm is used to determine the problem data chain. Figure 4 Taking a knowledge graph as an example, assuming F3 is the problem data, since nodes P1-T3-F3 and P2-T3-F3 both include F3, the problem nodes are determined to be P1-T3-F3 and P2-T3-F3. Using a graph traversal algorithm, nodes P1-T3-F3 and P2-T3-F3 are found in the knowledge graph. The connected subgraph containing nodes P1-T3-F3 and P2-T3-F3 is determined as the problem data chain. Figure 4 In the example, the problem data chain is P2-T6-F7→P2-T5-F6→P2-T3-F3→P1-T3-F3→P1-T2-F2→P1-T1-F1. Operations S241 and S242 facilitate the determination of the problem data chain within the knowledge graph based on the obtained problem data.

[0083] Figure 6 A flowchart illustrating the repair of a problematic data link according to an embodiment of this disclosure is shown schematically.

[0084] Operation S250 repairs the problematic data chain, including operations S251 to S253.

[0085] In operation S251, the upstream node of the problematic data in the problematic data chain is identified as the node to be confirmed.

[0086] In operation S252, the problematic data and its downstream nodes in the problematic data chain are identified as nodes to be repaired.

[0087] In operation S253, the node to be repaired is repaired.

[0088] Taking the problematic data chain P2-T6-F7→P2-T5-F6→P2-T3-F3→P1-T3-F3→P1-T2-F2→P1-T1-F1 as an example, the upstream nodes P2-T6-F7 and P2-T5-F6 of the problematic node P2-T3-F3 are identified as nodes to be confirmed; the downstream nodes P1-T2-F2 and P1-T1-F1 of the problematic node P2-T3-F3 are identified as nodes to be repaired; and the problematic nodes P1-T3-F3 and P2-T3-F3 are identified as nodes to be repaired. Therefore, nodes P1-T2-F2, P1-T1-F1, P1-T3-F3, and P2-T3-F3 need to be repaired. For nodes to be confirmed, manual verification can be used to confirm whether they are problematic nodes, and then corresponding measures can be taken. By operating S251 to S253, it is easy to repair the problematic data chain.

[0089] Figure 7 The flowchart illustrating the repair process of the node to be repaired according to an embodiment of the present disclosure is shown schematically.

[0090] Operation S253 repairs the node to be repaired, including operation S2531.

[0091] In operation S2531, a custom script for the node to be repaired is written and / or the task re-tune is executed on the node to be repaired. Operation S2531 facilitates the repair of the node to be repaired.

[0092] Based on the aforementioned knowledge graph-based data repair method, this disclosure also provides a knowledge graph-based data repair device 10. The following will be combined with... Figures 8-12 A detailed description is provided for the knowledge graph-based data repair device 10.

[0093] Figure 8 A schematic block diagram of a knowledge graph-based data repair apparatus 10 according to an embodiment of the present disclosure is shown.

[0094] The knowledge graph-based data repair device 10 includes an acquisition module 1, an extraction module 2, a construction module 3, a determination module 4, and a repair module 5.

[0095] Get module 1, which is used to perform operation S210: get m stored procedure information within a time period of t seconds. Each stored procedure information includes the source and destination of the data, where m is an integer greater than or equal to 1 and t is greater than or equal to 1.

[0096] Extraction module 2 is used to perform operation S220: based on m stored procedure information, extract the parent information, the child information, and the parent-child relationship between the parent and child information, where the parent information includes the parent stored procedure-table-field, and the child information includes the child stored procedure-table-field.

[0097] Module 3 is used to perform operation S230: construct a knowledge graph based on superior information, subordinate information and parent-child relationship, wherein the nodes of the knowledge graph are constructed based on superior information and subordinate information, and the edges of the knowledge graph are constructed based on parent-child relationship.

[0098] Module 4 is used to perform operation S240: determine the problem data chain in the knowledge graph based on the problem data within a pre-obtained time period of t seconds.

[0099] Repair module 5 is used to perform operation S250: repair the problematic data chain.

[0100] Figure 9 A schematic block diagram of the extraction module 2 according to an embodiment of the present disclosure is shown. The extraction module 2 includes an extraction unit 21 and a first determining unit 22.

[0101] Extraction unit 21 is used to extract the superior information, subordinate information and parent-child relationship from each stored procedure information to obtain intermediate data results.

[0102] The first determining unit 22 is used to integrate m intermediate data results according to the integration rules to obtain the final data result. The final data result includes the superior information, subordinate information and the parent-child relationship between the superior information and subordinate information in the m stored procedure information.

[0103] Figure 10 A schematic block diagram of the determining module 4 according to an embodiment of the present disclosure is shown. The determining module 4 includes a second determining unit 41 and a third determining unit 42.

[0104] The second determining unit 41 is used to match nodes in the knowledge graph as problem nodes based on the problem data within a pre-obtained time period of t seconds. The problem data includes a problem table and / or problem fields.

[0105] The third determining unit 42 is used to determine the problem data chain based on the problem nodes using a graph traversal algorithm.

[0106] Figure 11 A schematic block diagram of the repair module 5 according to an embodiment of the present disclosure is shown. The repair module 5 includes a fourth determining unit 51, a fifth determining unit 52, and a repair unit 53.

[0107] The fourth determining unit 51 is used to determine the upstream node of the problem data in the problem data chain as the node to be confirmed.

[0108] The fifth determining unit 52 is used to determine the problematic data and its downstream nodes in the problematic data chain as nodes to be repaired.

[0109] Repair unit 53 is used to repair the node to be repaired.

[0110] Figure 12 A schematic block diagram of a repair unit 53 according to an embodiment of the present disclosure is shown. The repair unit 53 includes a repair element 531.

[0111] Repair element 531 is used to write custom scripts for the node to be repaired and / or to re-tune the tasks performed on the node to be repaired.

[0112] According to the knowledge graph-based data repair apparatus 10 of this disclosure, by extracting superior information, subordinate information, and parent-child relationships from m stored process information within a t-second time period, a knowledge graph of the stored process information within the t-second time period can be constructed based on the superior information, subordinate information, and parent-child relationships. Therefore, based on the edge relationships of the knowledge graph, other data associated with the problem data within the t-second time period can be found, thereby enabling the repair of the problem data and associated data. The data repair method of this disclosure has low reliance on manual labor, high intelligence, and high efficiency and accuracy.

[0113] Furthermore, according to embodiments of this disclosure, any and multiple modules among the acquisition module 1, extraction module 2, construction module 3, determination module 4, and repair module 5 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in one module.

[0114] According to embodiments of this disclosure, at least one of the acquisition module 1, extraction module 2, construction module 3, determination module 4, and repair module 5 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three methods of software, hardware, and firmware, or in a suitable combination of any of these.

[0115] Alternatively, at least one of the acquisition module 1, extraction module 2, construction module 3, determination module 4, and repair module 5 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0116] The following reference Figure 13 and Figure 14 This document provides a detailed description of a knowledge graph-based data repair apparatus according to embodiments of the present disclosure. It is important to understand that the following description is merely illustrative and not intended to limit the scope of the disclosure.

[0117] The knowledge graph-based data repair apparatus according to embodiments of this disclosure includes the following five devices, such as... Figure 13 As shown.

[0118] 1. Stored Procedure Scanning Device: Input the names of all financial market stored procedures, download the corresponding stored procedure files, scan the stored procedure information, and obtain the tables and fields involved in each stored procedure, denoted as Stored Procedure_Table_Field. Furthermore, based on keywords such as INSERT, SELECT, UPDATE, MERGE, and FROM, the parent-child relationships between fields can be determined for each stored procedure. For example, stored procedure P1 contains the following INSERT statement (SQL has been greatly simplified):

[0119] INSERT INTO T1(F1)

[0120] SELECT T2.F2

[0121] FROM T2

[0122] INNER JOIN T3

[0123] ON T2.F3=T3.F4

[0124] Then you can get that stored procedure_table_field P1_T1_F1 comes from stored procedure_table_field P1_T2_F2 (P represents stored procedure, T represents table, and F represents field).

[0125] 2. Metadata storage device: Store the stored procedure_table_field information and parent-child relationship information of each stored procedure obtained by the stored procedure scanning device in a relational database. Taking stored procedures P1 and P2 as examples, the data is shown in Table 4 and Table 5.

[0126] Table 4

[0127] Parent stored procedure - table - field Sub-stored procedure - table - field P1-T2-F2 P1-T1-F1 P1-T3-F3 P1-T2-F2 P1-T4-F5 P1-T1-F4

[0128] Table 5

[0129] Parent stored procedure - table - field Sub-stored procedure - table - field P2-T5-F6 P2-T3-F3 P2-T6-F7 P2-T5-F6 P2-T8-F9 P2-T7-F8 P2-T9-F10 P2-T7-F8

[0130] Stored procedure P1 uses the T3_F3 field, while stored procedure P2 updates the T3_F3 field, indicating a parent-child relationship between them. Matching the parent stored procedure_table_field and child stored procedure_table_field between two different stored procedures reveals the parent-child relationship across stored procedures. The combined data from stored procedures P1 and P2 is shown in Table 6.

[0131] Table 6

[0132] Parent stored procedure - table - field Sub-stored procedure - table - field P1-T2-F2 P1-T1-F1 P1-T3-F3 P1-T2-F2 P1-T4-F5 P1-T1-F4 P2-T5-F6 P2-T3-F3 P2-T6-F7 P2-T5-F6 P2-T8-F9 P2-T7-F8 P2-T9-F10 P2-T7-F8 P2-T3-F3 P1-T3-F3

[0133] 3. Graph Conversion Device: The graph is drawn using nodes and edges, with "stored procedure table fields" as nodes and parent-child node dependencies as edges.

[0134] The source trajectory of node P1_T1_F1 in Table 6 is P2_T6_F7→P2_T5_F6→P2_T3_F3→P1_T3_F3→P1_T2_F2→P1_T1_F1. Following this method, Table 6 can be converted into a graph.

[0135] 4. Graph Query Transpose: Input any table and field information, and the corresponding stored procedure_table_field node can be matched. By using graph traversal algorithms (such as depth-first traversal algorithm, breadth-first traversal algorithm, etc.) to find the subgraph, the upstream and downstream dependencies of the table and field can be determined, and the data flow graph can be obtained.

[0136] 5. Data Repair Device: Input any table and field information to invoke the graph query device, locate the stored procedure_table_field node, and display the upstream and downstream stored procedure_table_field nodes involved. Define the upstream node as the node to be confirmed, and define the current node and downstream nodes as nodes to be repaired. Each node has a data repair operation button with options including writing a custom script and re-executing the task. Additionally, there is a one-click repair button. Clicking this button displays all involved nodes, and each node can freely select a repair method.

[0137] The workflow diagram of the knowledge graph-based data repair device is as follows: Figure 14 As shown. The specific working steps of the knowledge graph-based data repair device are as follows.

[0138] 1. Use a stored procedure scanning tool to obtain information about the fields in the stored procedure table and their parent-child relationships for each stored procedure.

[0139] 2. Use metadata storage devices to store parent-child relationship information of stored procedure tables and fields within a single stored procedure and across stored procedures into a relational database.

[0140] 3. Using a graph transformation device, convert the stored procedures, tables, fields, and parent-child relationships stored in the relational database into a graph. Use the stored procedures, tables, and fields as nodes and the parent-child node dependencies as edges to draw the graph.

[0141] 4. For the requirement to query upstream and downstream information of table fields, the graph query device matches the corresponding stored procedure_table_field node, and uses a graph traversal algorithm to find the subgraph involved in the stored procedure_table_field node, thereby determining the upstream and downstream dependencies of the table and field and obtaining the data flow graph.

[0142] 5. For the need to repair erroneous data, based on the input error table fields, following step 4, the upstream and downstream dependencies of that field in the table can be located, resulting in a data flow diagram. Based on this diagram, upstream nodes are defined as nodes to be confirmed, and the current node and downstream nodes are defined as nodes to be repaired. Querying the corresponding fields in the diagram can determine the minimum scope of data repair. Then, depending on the specific situation, the custom script writing function in the data repair device can be used to directly update the corresponding table fields, or the execution task re-adjustment function of the data repair device can be used to re-adjust the corresponding stored procedure, thereby achieving the goal of repairing the erroneous data.

[0143] This disclosure proposes a device capable of determining the processing logic of financial market table fields and repairing erroneous data with minimal time and computational costs. This device can quickly identify table fields that need updating and tasks that need to be rerun to correct the impact of erroneous data. This disclosure employs graph computing technology, constructing a relationship graph of stored procedures, tables, and field dimensions using a graph-based depth-first traversal algorithm. Based on this relationship graph, the update positions of table fields can be quickly located, reducing reliance on skilled personnel, improving the speed of data error repair, and saving time and computational costs.

[0144] Figure 15 A block diagram schematically illustrates an electronic device suitable for implementing the above-described method according to an embodiment of the present disclosure.

[0145] like Figure 15 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0146] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0147] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0148] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0149] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0150] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods of the embodiments of this disclosure.

[0151] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0152] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0153] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0156] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0157] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data repair method based on knowledge graphs, characterized in that, include: Obtain information on m stored procedures within a time interval of t seconds. Each stored procedure information includes the source and destination of the data, where m is an integer greater than or equal to 1 and t is greater than or equal to 1. Based on the m stored procedure information, extract the superior information, subordinate information, and the parent-child relationship between the superior information and the subordinate information, wherein the superior information includes the parent stored procedure-table-field, and the subordinate information includes the child stored procedure-table-field; A knowledge graph is constructed based on the superior information, the subordinate information, and the parent-child relationship, wherein the nodes of the knowledge graph are constructed based on the superior information and the subordinate information, and the edges of the knowledge graph are constructed based on the parent-child relationship; Based on the pre-obtained question data within the t-second time period, a question data chain is determined in the knowledge graph; and Repair the problematic data chain.

2. The method according to claim 1, characterized in that, The step of extracting the parent information, the child information, and the parent-child relationship between the parent information and the child information based on the m stored procedure information includes: Extract the parent-child information, sibling information, and parent-child relationship from each stored procedure information to obtain intermediate data results; and The m intermediate data results are integrated according to the integration rules to obtain the final data result, wherein the final data result includes the superior information, the subordinate information, and the parent-child relationship between the superior information and the subordinate information in the m stored procedure information.

3. The method according to claim 2, characterized in that, The integration rules include: The m intermediate data results are merged, and in every two intermediate data results, when the table-field in the superior information of one intermediate data result is the same as the table-field in the subordinate information of the other intermediate data result, the superior information of the other intermediate data result is used as the superior information and the subordinate information of one of the intermediate data results is used as the subordinate information in the final data result.

4. The method according to claim 1, characterized in that, The step of determining the question data chain in the knowledge graph based on the question data within the pre-obtained t-second time period includes: Based on the pre-obtained question data within the t-second time period, nodes in the knowledge graph are matched as question nodes; the question data includes a question table and / or question fields; and Based on the problem nodes, a graph traversal algorithm is used to determine the problem data chain.

5. The method according to claim 1, characterized in that, The repair of the problematic data chain includes: The upstream node of the problem data in the problem data chain is identified as the node to be confirmed. The problematic data and its downstream nodes in the problematic data chain are identified as nodes to be repaired; and Repair the node to be repaired.

6. The method according to claim 5, characterized in that, The repair of the node to be repaired includes: Write a custom script for the node to be repaired and / or perform task rescheduling on the node to be repaired.

7. A data repair device based on knowledge graphs, characterized in that, include: The acquisition module is used to acquire m stored procedure information within a time period of t seconds. Each stored procedure information includes the source and destination of the data, where m is an integer greater than or equal to 1 and t is greater than or equal to 1. The extraction module is used to extract the superior information, subordinate information, and parent-child relationship between the superior information and the subordinate information based on the m stored procedure information. The superior information includes parent stored procedure-table-field, and the subordinate information includes child stored procedure-table-field. A construction module is configured to construct a knowledge graph based on the superior information, the subordinate information, and the parent-child relationship, wherein the nodes of the knowledge graph are constructed based on the superior information and the subordinate information, and the edges of the knowledge graph are constructed based on the parent-child relationship; The determination module is used to determine a question data chain in the knowledge graph based on the question data within the pre-obtained t-second time period; and A repair module is used to repair the problematic data chain.

8. An electronic device, characterized in that, include: One or more processors; One or more memories are provided for storing executable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions that, when executed by a processor, implement the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The method includes a computer program comprising one or more executable instructions that, when executed by a processor, implement the method according to any one of claims 1 to 6.

Images