Method, system, medium and device for assisting data extraction

Abstract

The present invention provides a method, system, medium and device for assisting data extraction, including: step S1: constructing a data extraction knowledge base; step S2: dynamically maintaining a system evolution knowledge base; step S3: receiving a data extraction requirement written in natural language; step S4: performing natural language analysis on the data extraction requirement and extracting key elements; step S5: matching the closest N historical extraction requirements according to the similarity; step S6: making corresponding changes to the data extraction code of the historical requirements to obtain N query scripts; step S7: comparing the system data changes at the historical time point and the current time point, updating the N codes accordingly, and performing a rationality check; step S8: executing the updated code to obtain the corresponding data; step S9: verifying the accuracy of the data obtained by the N extractions and updating the data extraction knowledge base. The present invention can achieve the technical problem of automatically converting business requirements from natural language into data extraction scripts.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system, medium and device for assisting data extraction. Background Art

[0002] Data extraction is a common operation performed by business personnel in the process of business management. This operation is characterized by high frequency and tight time. It is difficult for extraction personnel to have a global understanding of the data distribution in various systems within the enterprise, and the constant changes in the systems involved make it difficult to reuse the original extraction plans. At the same time, some extractions involve multiple systems and have complex query conditions, which are not conducive to extraction personnel obtaining data quickly.

[0003] As enterprises grow in size and their businesses and processes become more complex, it becomes increasingly difficult for business personnel to extract data from systems. The difficulties include:

[0004] 1. It is difficult for business personnel to master database table analysis and query technology;

[0005] 2. Business personnel do not fully understand the data storage location and the relationship between various systems;

[0006] 3. Given that the system will iterate with the continuous development of the business, the existing historical business data query solution is difficult to meet the latest needs.

[0007] The invention patent with publication number CN112507353A discloses a data extraction method and device, which includes: when receiving a data extraction scheme, performing a first analysis on the data extraction scheme; when the first analysis satisfies a first condition, using the data extraction scheme as a scheme to be extracted; in response to an extraction operation according to the data extraction scheme, performing a second analysis on the extraction operation; when the second analysis satisfies a second condition, performing data extraction; generating a target file according to the extracted data; and sending the target file to a user who has made a data extraction request. Summary of the invention

[0008] In view of the defects in the prior art, the present invention provides a method, system, medium and device for auxiliary data extraction.

[0009] According to a method, system, medium and device for assisting data extraction provided by the present invention, the scheme is as follows:

[0010] In a first aspect, a method for assisting data extraction is provided, the method comprising:

[0011] Step S1: construct a data extraction knowledge base, which records the logs of historical data queries;

[0012] Step S2: Dynamically maintain the system evolution knowledge base and record the change details when each system is changed;

[0013] Step S3: receiving business requirements written in natural language;

[0014] Step S4: Perform natural language analysis on business requirements to extract key elements of the requirements;

[0015] Step S5: In the data extraction knowledge base, based on the extracted key elements, the closest N historical extraction requirements are matched according to the cosine similarity;

[0016] Step S6: Based on the key elements, the differences between each historical extraction requirement and the current business requirement are compared, and the codes of the historical requirements are changed accordingly to obtain N query scripts;

[0017] Step S7: Based on the system evolution knowledge base, compare the system data changes at the historical time point with the current time point, identify the system, database, table, and field involved in the code obtained in step S6 and replace them accordingly based on the data dictionary, update N codes accordingly, and check the rationality of the codes based on the code grammar;

[0018] Step S8: Execute the updated codes respectively to obtain corresponding data;

[0019] Step S9: Verify the accuracy of the acquired data and update the data extraction knowledge base.

[0020] Preferably, the log of historical data query recorded by the data extraction knowledge base in step S1 includes: historical data extraction requirements, key elements of the requirements, data extraction scripts and the corresponding relationship between the extracted data.

[0021] Preferably, the system evolution knowledge base in step S2 contains data dictionaries, table structures, data range change records of various systems in the enterprise at historical time points, and associations between various systems.

[0022] Preferably, the key elements in step S4 include: business data information items, relevant filtering restriction condition information items, sorting condition information items, aggregation and summary condition information items, connection condition information items, and grouping condition information items.

[0023] Preferably, the step S9 of verifying the accuracy of the obtained data includes:

[0024] If one of the data is accurate, the requirements, key elements and data extraction script corresponding to the data are added to the data extraction knowledge base;

[0025] If there is no accurate data among the N extracted data, the code with the highest matching degree among the original N codes is selected and manually modified. After the modification, the relevant requirements, key elements and scripts are added to the data extraction knowledge base.

[0026] In a second aspect, a system for assisting data extraction is provided, the system comprising: a data extraction knowledge base construction module, a system evolution knowledge base construction module, a business demand analysis module, a historical demand automatic matching module, a query script analysis and automatic generation module, and a query data evaluation module;

[0027] Among them, the data extraction knowledge base module is constructed: the historical business data requirements are analyzed, and the index is established and relevant information is saved according to the relevant structure including business requirements, key elements of requirements, extraction scripts and extracted data;

[0028] System evolution knowledge base construction module: Based on the system changes involved in data extraction, the distribution of each business data in each system at any time point is recorded by time slice;

[0029] Business requirements analysis module: performs natural language analysis on business requirements, strips away the business background and extracts the reason content in the business requirements, locates the extraction target in the business requirements, and then extracts the key elements in the business requirements;

[0030] Historical demand automatic matching module: automatically matches the most similar N historical extraction demands in the data extraction knowledge base based on cosine similarity;

[0031] Query script parsing and automatic generation module: compares the key elements of historical extraction requirements with current business requirements, generates current business requirement scripts from historical extraction processing scripts, and updates scripts based on the system evolution knowledge base;

[0032] Query data evaluation module: includes business personnel verifying the extracted data and updating the data extraction knowledge base based on the verification results.

[0033] According to a third aspect, a computer-readable storage medium storing a computer program is provided, wherein the computer program implements the steps of the method when executed by a processor.

[0034] According to a fourth aspect, an auxiliary data extraction device is provided, wherein the auxiliary data extraction device comprises the auxiliary data extraction system, or the auxiliary data extraction device comprises the computer-readable storage medium storing the computer program.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention is based on the analysis of historical data extraction needs and performs natural language analysis on new extraction needs, thereby automatically generating corresponding query scripts and completing data extraction operations, thereby improving the automation level of querying data and achieving the technical problem of automatically converting business needs from natural language into data extraction scripts. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0040] The embodiment of the present invention provides a method for assisting data extraction, which uses natural language analysis, key element similarity matching and other means to realize the automation process of data extraction from text requirements to data acquisition, and reduce the amount of development code engineering. Figure 1 As shown, the relevant steps of this method are as follows:

[0041] Step S1: Construct a data extraction knowledge base. The data extraction knowledge base records the logs of historical data queries, mainly including the correspondence between historical data extraction requirements, key elements of requirements, data extraction scripts, and extracted data. Among them, constructing a data extraction knowledge base has two functions. One is to save historical data extraction information, so that new extraction requirements can be easily compared with historical data extraction. Second, after the new data extraction is completed, the data extraction is updated to the knowledge base to continuously enrich the knowledge base. It is conducive to the subsequent emergence of new data extraction requirements, namely business demand analysis.

[0042] Step S2: Dynamically maintain the system evolution knowledge base, record the change details when each system is changed, and the system evolution knowledge base contains the table structure of each system in the enterprise at a historical time point, the data range change record and the relationship between each system. Among them, the system evolution knowledge base records the change history of the system, table and data dictionary in the enterprise. Since the historical data extraction information saved in step S1 corresponds to the system and table at a historical time point, in the process of processing new business needs, in order to reuse the historical business needs that are closest to it, the corresponding historical business needs need to be changed accordingly according to the latest system and table structure.

[0043] Step S3: The extractor writes the business requirements using natural language.

[0044] Step S4: Perform natural language analysis on the business requirements and extract key elements in the requirements, mainly including extracted business data information items, related filtering restrictions, sorting conditions, aggregation conditions, join conditions, grouping conditions, etc. For example, for the extraction requirement "Extract the summary value of the income amount of account B from January to December 2019 for customers with ID number A.", perform word segmentation on the requirement expression, remove the function words in the expression, and obtain the keywords in the requirement, such as ID number A (restriction), account B (restriction), time range from January to December 2019 (restriction), income amount (extracted information item), and summary plan (summation and summary).

[0045] Step S5: In the data extraction knowledge base, based on the key elements of this extraction, the closest N historical extraction requirements are matched according to the cosine similarity. The cosine similarity calculation method of two data extraction requirements A and B based on the key elements is as follows: A requirement lists keywords A1, A2, .. An according to business data information items, related filtering restriction condition information items, sorting condition information items, aggregation and summary condition information items, connection condition information items, and grouping condition information items, with a word frequency of a1, a2, ..., an. B requirement lists keywords B1, B2, .. Bn according to business data information items, related filtering restriction condition information items, sorting condition information items, aggregation and summary condition information items, connection condition information items, and grouping condition information items, with a word frequency of b1, b2, ..., bn. Where Ai and Bi are the same information items, the similarity between the two requirements can be expressed as: CosAB = (a1*b1+a2*b2+...+an*bn) / (sqrt(a1^2+a2^2+...+an^2)*sqrt(b1^2+b2^2+...+bn^2))

[0046] Step S6: Based on the key factors, compare the differences between each historical extraction requirement and the current business requirement, and make corresponding changes to the code of the historical extraction requirement to obtain N query scripts.

[0047] Step S7: Based on the system evolution knowledge base, compare the system data changes at the historical time point and the current time point, identify the system, database, table, and field involved in the code obtained in step S6, and replace them accordingly based on the data dictionary (for example: at a historical time point, a certain data was stored in the C field of table B of system A. After the system is transformed, the data is now stored in the Z field of table Y of system X. Replace A, B, and C in the query code with X, Y, and Z accordingly). Update N codes accordingly, and check the rationality of the codes based on the code grammar, mainly to check whether the libraries, tables, and fields involved in the codes are in the latest data dictionary and whether the data can be obtained.

[0048] Step S8: Execute the updated codes respectively to obtain corresponding data.

[0049] Step S9: The business personnel verifies the accuracy of the obtained data and updates the data extraction knowledge base:

[0050] If one of the data is accurate, the requirements, key elements and data extraction script corresponding to the data are added to the data extraction knowledge base;

[0051] If there is no accurate data among the N extracted data, the code with the highest matching degree among the original N codes is selected and manually modified. After the modification, the relevant requirements, key elements and scripts are added to the data extraction knowledge base.

[0052] The present invention also provides a system for assisting data extraction, comprising the following modules: a data extraction knowledge base construction module, a system evolution knowledge base construction module, a business demand analysis module, a historical demand automatic matching module, a query script analysis and automatic generation module, and a query data evaluation module.

[0053] Among them, building a data extraction knowledge base module includes: analyzing historical business data requirements, indexing and saving relevant information according to business requirements, key elements of requirements, extraction scripts and extracted data. Among them, the data extraction knowledge base construction plan: summarize historical data extraction requirements and obtain relevant information, including the data extraction number, the corresponding initiation time, initiation agency, processing personnel, original requirements, key elements of data extraction, extraction scripts, extracted data, etc., among which all information except the extracted data is stored in the database, and the data extracted is saved in the form of files and associated by the data extraction number.

[0054] System evolution knowledge base construction module: Based on the system changes involved in data extraction, the distribution of each business data in each system at any time point is recorded by time slice.

[0055] Business requirements analysis module: Perform natural language analysis on business requirements, separate the business background and extraction reasons from business requirements, locate the extraction targets in business requirements, and then extract the key elements in business requirements.

[0056] Historical demand automatic matching module: automatically matches the N most similar historical extraction demands in the data extraction knowledge base based on cosine similarity.

[0057] Query script parsing and automatic generation module: compares the key elements in historical extraction requirements and current business requirements, generates current business requirement scripts from historical extraction processing scripts, and updates scripts according to the system evolution knowledge base.

[0058] Query data evaluation module: includes the process in which business personnel verify the extracted data and update the data extraction knowledge base based on the verification results.

[0059] The present invention also provides a computer-readable storage medium storing a computer program, which implements the steps in the method for auxiliary data extraction when executed by a processor.

[0060] The present invention also provides an auxiliary data extraction device, which includes an auxiliary data extraction system, or includes a computer-readable storage medium storing a computer program.

[0061] The embodiments of the present invention provide a method, system, medium and device for assisting data extraction, which constructs a data extraction knowledge base based on historical business requirements and scripts; performs natural language analysis on new business requirements and extracts key elements in the requirements; matches historical requirements in the data extraction knowledge base for new data extraction based on similarity, and generates a script corresponding to the data extraction based on the scripts of related requirements and the historical change relationships between the systems involved.

[0062] The present invention generates corresponding data query statements based on the natural language needs of business personnel and returns corresponding data extraction results, thereby realizing the technical problem of automatically converting data extraction business needs from natural language into data extraction scripts, and can obtain data experience in a timely manner, providing convenience for business personnel.

[0063] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0064] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for assisting data extraction, characterized in that: include: Step S1: construct a data extraction knowledge base, which records the logs of historical data queries; Step S2: Dynamically maintain the system evolution knowledge base and record the change details when each system is changed; Step S3: receiving business requirements written in natural language; Step S4: Perform natural language analysis on the business requirements and perform structured processing to extract key elements of the business requirements; Step S5: In the data extraction knowledge base, based on the extracted key elements, the closest N historical extraction requirements are matched according to the cosine similarity; Step S6: Based on the key elements, the differences between each historical extraction requirement and the current business requirement are compared, and the code of the historical extraction requirement is changed accordingly to obtain N query scripts; Step S7: Based on the system evolution knowledge base, compare the system data changes at the historical time point with the current time point, identify the system, database, table, and field involved in the code obtained in step S6 and replace them accordingly based on the data dictionary, update N codes accordingly, and check the rationality of the codes based on the code grammar; Step S8: Execute the updated codes respectively to obtain corresponding data; Step S9: Verify the accuracy of the acquired data and update the data extraction knowledge base.

2. The method for auxiliary data extraction according to claim 1, characterized in that: The log of historical data query recorded by the data extraction knowledge base in step S1 includes: historical data extraction requirements, key elements of the requirements, data extraction scripts and the corresponding relationship between the extracted data.

3. The method for auxiliary data extraction according to claim 1, characterized in that: The system evolution knowledge base in step S2 contains the data dictionary, table structure, data range change records of each system in the enterprise at historical time points and the association relationship between each system.

4. The method for auxiliary data extraction according to claim 1, characterized in that: The key elements in step S4 include: business data information items, related filtering restriction condition information items, sorting condition information items, aggregation and summary condition information items, connection condition information items, and grouping condition information items.

5. The method for auxiliary data extraction according to claim 1, characterized in that: The step S9 of verifying the accuracy of the obtained data includes: If one of the data is accurate, the requirements, key elements and data extraction script corresponding to the data are added to the data extraction knowledge base; If there is no accurate data among the N extracted data, the code with the highest matching degree among the original N codes is selected and manually modified. After the modification, the relevant requirements, key elements and scripts are added to the data extraction knowledge base.

6. A system for assisting data extraction, characterized in that: include: Construct data extraction knowledge base module, system evolution knowledge base construction module, business demand analysis module, historical demand automatic matching module, query script analysis and automatic generation module, query data evaluation module; Among them, the data extraction knowledge base module is constructed: the historical business data requirements are analyzed, and the index is established and relevant information is saved according to the relevant structure including business requirements, key elements of requirements, extraction scripts and extracted data; System evolution knowledge base construction module: Based on the system changes involved in data extraction, the distribution of each business data in each system at any time point is recorded by time slice; Business requirements analysis module: performs natural language analysis on business requirements, strips away the business background and extracts the reason content in the business requirements, locates the extraction target in the business requirements, and then extracts the key elements in the business requirements; Historical demand automatic matching module: automatically matches the most similar N historical extraction demands in the data extraction knowledge base based on cosine similarity; Query script parsing and automatic generation module: compares the key elements of historical extraction requirements with current business requirements, generates current business requirement scripts from historical extraction processing scripts, and updates scripts based on the system evolution knowledge base; Query data evaluation module: includes business personnel verifying the extracted data and updating the data extraction knowledge base based on the verification results.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. An auxiliary data extraction device, comprising the system as claimed in claim 6, or comprising the medium as claimed in claim 7.

Images