Data anomaly positioning method and device, medium, equipment and product

Abstract

The invention relates to the technical field of data analysis, and particularly provides a data exception positioning method and device, a medium, equipment and a product, and the method can comprise the steps: obtaining associated data and a demand document corresponding to exceptional data in a work order; wherein the demand document is used for deducing business logic between the abnormal data and the associated data; the associated data comprises multiple layers of upstream data; performing analog computation on the associated data based on the business logic to obtain verification data; comparing the verification data with the abnormal data, and positioning an abnormal position; wherein the abnormal position is any layer in the multiple layers of upstream data. According to the embodiment of the invention, intelligent traceability of data can be realized, and the precision and efficiency of anomaly positioning are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, in particular to a data abnormality positioning method, device, medium, equipment and product. BACKGROUND

[0002] A data warehouse platform is a technical system integrating data integration, storage, calculation, management and analysis, and the core goal is to build a structured and thematic data model from the heterogeneous data scattered in various business systems of an enterprise after cleaning, conversion and integration, so as to provide unified and reliable data support for enterprise decision analysis, report generation and data mining.

[0003] At present, when monitoring the data quality of a data warehouse platform, it is mainly responsible by business personnel. When the business personnel or an application system finds that a piece of data has a quality problem, the on-site situation is manually filled into a work order system, and a quality management personnel will enter it into a data quality management system after receiving the work order. A technical personnel finds the problem point after analyzing the data, so as to realize abnormality positioning. However, since the data volume of the data warehouse platform is large, the workload of manually monitoring the data quality is large, and the abnormality analysis efficiency is low.

[0004] Therefore, how to provide a technical solution of an efficient data abnormality positioning method becomes a technical problem to be solved urgently. SUMMARY

[0005] Some embodiments of the present application aim to provide a data abnormality positioning method, device, medium, equipment and product, and the technical solution of the embodiments of the present application can improve the efficiency and accuracy of data abnormality positioning, realize timely monitoring and processing of data quality, and reduce the labor cost.

[0006] In a first aspect, some embodiments of the present application provide a data abnormality positioning method, comprising: acquiring associated data and a requirement document corresponding to abnormal data in a work order; wherein the requirement document is used to deduce the business logic between the abnormal data and the associated data; the associated data comprises multi-layer upstream data; performing simulation calculation on the associated data based on the business logic to acquire verification data; comparing the verification data and the abnormal data to locate an abnormal position; wherein the abnormal position is any layer in the multi-layer upstream data.

[0007] Some embodiments of the present application acquire the associated data and the requirement document of the abnormal data in the work order, perform simulation calculation on the associated data by the business logic deduced from the requirement document to obtain verification data, and finally compare the verification data and the abnormal data to locate the abnormal position. The embodiments of the present application can improve the efficiency and accuracy of data abnormality positioning, realize timely monitoring and processing of a large amount of data quality in the data warehouse platform, and reduce the labor and business maintenance cost.

[0008] In some embodiments, before the acquiring the associated data and the requirement document corresponding to the abnormal data in the work order, the method further comprises: receiving the work order and reading the abnormal data; predicting the expected data corresponding to the abnormal data based on other data in the work order except the abnormal data; and verifying that the abnormal data is abnormal by the expected data.

[0009] Some embodiments of the present application predict the expected data of the abnormal data by other data in the work order, thereby verifying that the abnormal data is indeed abnormal, realizing the error evaluation of the abnormal data, and ensuring the effectiveness of subsequent abnormal positioning.

[0010] In some embodiments, the acquiring the associated data and the requirement document corresponding to the abnormal data in the work order comprises: extracting a key field in the abnormal data; retrieving the associated data matching the key field from a meta database; and retrieving the requirement document corresponding to the key field from a document knowledge base; wherein the data in the meta database is stored in a structured manner; and the data in the document knowledge base is stored in a vector manner.

[0011] Some embodiments of the present application acquire the associated data and the requirement document from the meta database and the document knowledge base respectively by the key field in the abnormal data, thereby providing support for subsequent precise positioning of the abnormal data.

[0012] In some embodiments, the simulating calculation of the associated data based on the business logic to acquire the verification data comprises: automatically writing a script and using the script to simulate the data processing process in the business logic to calculate the associated data, thereby obtaining the verification data.

[0013] Some embodiments of the present application simulate the data processing process in the business logic to calculate the associated data by automatically writing a script, thereby realizing the automatic calculation of the verification data and having a high degree of intelligence.

[0014] In some embodiments, the simulating calculation of the associated data based on the business logic to acquire the verification data comprises: using a recursive algorithm to simulate the calculation of the i-th layer upstream data in the multi-layer upstream data by the business logic, thereby obtaining the i-th verification data; wherein i is a positive integer, and when i = 1, the first layer upstream data is the upstream data adjacent to the abnormal data; and the comparing the verification data with the abnormal data to locate the abnormal position comprises: confirming that the i-th verification data is different from the abnormal data, and then taking the i-th layer upstream data as the abnormal position.

[0015] Some embodiments of the present application obtain the check data by sequentially calculating each layer of upstream data through a recursive algorithm, and then compare the check data with the abnormal data to determine which layer of upstream data is the abnormal position, thereby achieving accurate and efficient positioning of data abnormalities.

[0016] In some embodiments, after locating the abnormal position, the method further comprises: determining an abnormal reason of the abnormal position; generating a correction suggestion corresponding to the abnormal reason; wherein the correction suggestion comprises a correction code, a correction scheme, a correction basis or a correction reference document.

[0017] Some embodiments of the present application generate a corresponding correction suggestion after determining the abnormal reason, so as to provide a reference for relevant technical personnel to take timely measures.

[0018] In a second aspect, some embodiments of the present application provide a device for positioning data abnormalities, comprising: an acquisition module configured to acquire associated data corresponding to abnormal data in a work order and a requirement document; wherein the requirement document is used to deduce the business logic between the abnormal data and the associated data; the associated data comprises multiple layers of upstream data; a calculation module configured to simulate calculation of the associated data based on the business logic to obtain check data; and a positioning module configured to compare the check data with the abnormal data to locate an abnormal position; wherein the abnormal position is any layer of the multiple layers of upstream data.

[0019] In a third aspect, some embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, can implement the method of any one of the embodiments of the first aspect.

[0020] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, can implement the method of any one of the embodiments of the first aspect.

[0021] In a fifth aspect, some embodiments of the present application provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor, can implement the method of any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the following will briefly introduce the drawings needed to be used in some embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0023] Figure 1 System diagram of data anomaly positioning provided for some embodiments of the present application; Figure 2 Method flow chart of data anomaly positioning provided for some embodiments of the present application; Figure 3 Method flow chart of data anomaly positioning provided for some embodiments of the present application; Figure 4 Device composition block diagram of data anomaly positioning provided for some embodiments of the present application; Figure 5 Electronic device schematic diagram provided for some embodiments of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in some embodiments of the present application will be described below in combination with the drawings in some embodiments of the present application.

[0025] It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing description, and cannot be understood as indicating or implying relative importance.

[0026] In the data warehouse platform, data problems often occur in the end of statistical results. Take a simple example for illustration. For example, the test scores of students in a school, the full scores of Chinese, mathematics and English are all 100 points, and the highest total score should be 300 points, but there is a statistical data of 301 points, which is obviously a data error. To confirm which subject has entered the wrong score, the data processing and calculation process needs to be verified, the upstream data source needs to be analyzed, and the corresponding responsible person and repair scheme need to be given. However, the correlation of data in the data warehouse platform is much more complex than this example. A data may be calculated and processed from dozens or even hundreds of systems, and many are entered by external systems of the enterprise; to find out which data is problematic and give a reasonable correction scheme is currently through manual way, business personnel and technical personnel cooperate and cross many departments to complete, which is a very time-consuming and laborious process. In general enterprises, a data quality department will be set up to be responsible for data quality processing.

[0027] The existing system is generally composed of a data quality management system, a work order system, and a data bloodline system. When a business personnel or an application system finds that a piece of data has a quality problem, the on-site situation is manually filled into the work order system; the quality management personnel enters the work order into the data quality management system after receiving the work order; the data quality management system contains metadata information of all data, and finds upstream data through the metadata and the bloodline system; then the technical personnel analyzes and finds data layer by layer through the metadata and the bloodline, finally gives the problem point, and the person in charge of the problem point gives a solution, and finally corrects the problem and verifies the problem.

[0028] From the above related technology, it can be seen that in the prior art, many steps of data quality monitoring are processed in an artificial manner, and depend on the understanding of the technical personnel on the business; however, when processing the problem, macroscopic thinking and reasoning cannot be performed, and the entire structure of the data warehouse cannot be understood in a short time, including analyzing and processing the problem through complete bloodlines, complete metadata, and complete various documents. In the process of processing abnormal data, some data is time-sensitive, and complex data problems need to be spent for a long time, which cannot guarantee the timeliness. Moreover, the problem processing is often not completed by one person, and lacks experience sharing. In the process of processing the problem, the relevant personnel (problem processing personnel and technical personnel) of the non-problem business system need to view the actual data, which cannot guarantee the security of the data; moreover, there is a contradiction that the problem data cannot be obtained when sensitive data is encountered. In the process of analyzing the data link, because many departments and data systems are crossed, the intermediate link is easy to be incomplete, which leads to misleading in the process of processing the problem. Sometimes, due to the lack of the ability of the entire business overview, some problems are only corrected at the problem point, and the essence of the problem cannot be found in the business process or processing process, which leads to frequent occurrence of the problem and cannot be radically cured. The analysis report is often not complete, and it is impossible to analyze which other business systems are affected through the root problem, and it is impossible to make a prior prediction.

[0029] In view of this, some embodiments of the present application provide a data abnormality positioning method, which can automatically read abnormal data in a work order after obtaining the work order with a problem, and obtain associated data and requirement documents corresponding to the abnormal data. Then, the associated data is simulated and calculated based on the business logic to obtain verification data; and the abnormal position of the abnormal data is positioned by comparing the verification data and the abnormal data. The embodiments of the present application can realize accurate and efficient positioning of abnormal data in the work order without manual participation throughout the process, ensure the timeliness of data quality monitoring, and at the same time guarantee the security of the data.

[0030] The overall structure of the data abnormality positioning system provided by some embodiments of the present application will be described below with reference to the accompanying Figure 1 The overall structure of the data abnormality positioning system provided by some embodiments of the present application will be described below with reference to the accompanying

[0031] As Figure 1 shown, some embodiments of the present application provide a system diagram of data anomaly positioning. The data anomaly positioning system can include a terminal 100 and a server 200. Among them, the server 200 is pre-deployed with an AI large model (referred to as AI) and a corresponding knowledge base (for example, a metadata database and a document knowledge base). Specifically, the terminal 100 can send a problem ticket to the server 200. The AI large model in the server 200 can read the ticket and obtain the abnormal data existing therein; then the AI large model obtains the associated data and the requirement document corresponding to the abnormal data by calling the knowledge base; then the AI large model automatically generates a script to simulate the calculation of the associated data based on the business logic obtained from the requirement document, and obtains the verification data; finally, the verification data and the abnormal data are compared to locate the abnormal position of the abnormal data.

[0032] Among them, the data anomaly positioning system needs to collect all the metadata in the data warehouse to the system platform in advance; collect the information of the person in charge or the department to which each nightly system in the data warehouse belongs to the system platform; In addition, when positioning, the problem ticket system can be connected to the system platform to automatically obtain data problems and realize problem positioning. In addition, the system platform stores the metadata in a structured manner after vectorization processing to form a metadata database. The document content related to the data is processed by slicing and vectorization to form a document knowledge base. The function name, parameter, comment and other information of the code type, SQL, Python and other codes are processed by vectorization to form a blood relationship knowledge base; the blood relationship knowledge base can store the processing relationship between data.

[0033] In some embodiments of the present application, the terminal 100 can be a mobile terminal, or a non-portable computer terminal, which is not specifically limited in the embodiments of the present application.

[0034] The following will be described in conjunction with the accompanying Figure 2 The implementation process of the data anomaly positioning performed by the server 200 provided by some embodiments of the present application will be described by way of example.

[0035] Please refer to the accompanying Figure 2 , Figure 2 The method flowchart of data anomaly positioning provided by some embodiments of the present application.

[0036] In some embodiments of the present application, before performing the following anomaly positioning method, the data anomaly positioning method can include: receiving a ticket and reading the abnormal data; predicting the expected data corresponding to the abnormal data based on other data in the ticket except the abnormal data; and verifying and confirming that the abnormal data is abnormal through the expected data.

[0037] For example, in the specific embodiments of the present application, the work order system will actively push the work order with data problems to the system platform. After the system platform receives the work order, the AI large model reads the problem description in the work order to obtain the abnormal data. For example, the work order describes which data has unreasonable conditions, so the abnormal data can be determined. Then, the AI large model calls the content in the metadata database of the system platform to locate the business content and business role contained in the work order. Subsequently, the AI large model obtains other correct data (as a specific example of other data) from the work order by writing SQL, and calculates the expected data of the current abnormal data according to a large amount of correct data; the expected data can be the mean or median of the correct data. The expected data can be used to evaluate the abnormal data. If it is confirmed that the abnormal data deviates from the median or mean, it is confirmed that the abnormal data is indeed abnormal, thereby realizing the preliminary evaluation of the abnormal data. When calculating the expected data, the AI large model can generate a calculation formula of the corresponding median or mean, and corresponding code, and execute the code in the sandbox to realize the preliminary evaluation of the abnormal data.

[0038] In some embodiments, S210 can be performed after the above operations are performed; or S210 can be performed directly after receiving the work order, and the embodiments of the present application are not limited herein.

[0039] In some embodiments of the present application, the method for locating data anomalies can include: S210, obtaining associated data and requirement documents corresponding to the abnormal data in the work order; wherein the requirement documents are used to infer the business logic between the abnormal data and the associated data; and the associated data includes multiple layers of upstream data.

[0040] For example, in the specific embodiments of the present application, the AI large model can query the upstream data and related requirement documents involved in the abnormal data in the work order.

[0041] In some embodiments of the present application, S210 can include: extracting key fields in the abnormal data; retrieving the associated data matching the key fields from the metadata database; and retrieving the requirement documents corresponding to the key fields from the document knowledge base; wherein the data in the metadata database is stored in a structured manner; and the data in the document knowledge base is stored in a vector manner.

[0042] For example, in the specific embodiments of the present application, the metadata in the meta database belongs to structured data and is stored in a structured manner, such as: {“system”: “user management system”, “table”: “user information table”, “field”: “user phone”}; when doing vector calculation, only the table information and field information are extracted, for example, vector(“user information table”+ “user phone”). The document class data in the document knowledge base is directly sliced to do vector, such as the traditional RAG (Retrieval-Augmented Generation, retrieval augmented generation).

[0043] In some embodiments, when the AI large model analyzes the work order, it first performs structured processing on the work order text, divides the work order text into multiple segments such as “business background segment”, “abnormal phenomenon segment” and “expected result segment”, and identifies table name, field name, system name, time granularity and statistical caliber and other entity information from each segment. For each identified entity, the AI large model encodes it into a feature vector containing entity type identifier and text representation, and concatenates multiple feature vectors to obtain the feature vector representation of the work order (as a specific example of the key field).

[0044] Subsequently, the server 200 performs joint retrieval in the meta database, the document knowledge base and the blood relationship knowledge base based on the feature vector representation of the work order. Specifically, the table, field and other metadata information in the meta database is encoded into a metadata vector, the document in the document knowledge base is sliced and encoded into a document vector, and the field mapping relationship, conversion rule and task node in the blood relationship knowledge base are encoded into a blood vector. The server 200 calculates the similarity between the feature vector of the work order and the metadata vector, the document vector and the blood vector respectively through the preset joint similarity function, and weights each similarity according to the pre-set weight to obtain the joint similarity score, and selects the most relevant associated data record, demand document segment and blood node according to the joint similarity score, thereby narrowing the data range that needs to be processed in subsequent simulation calculation. The joint similarity function can adopt a weighted cosine similarity function, that is: S = α·cos(v_ticket, v_meta) + β·cos(v_ticket, v_doc) + γ·cos(v_ticket, v_lineage), where v_ticket is the feature vector of the work order, v_meta is the metadata vector, v_doc is the document vector, and v_lineage is the blood vector; α, β, γ are preset weight parameters, which can be flexibly set.

[0045] It should be noted that the joint similarity function in the above joint search can be deployed in the AI large model of the server 200, and the joint similarity calculation is performed using the AI large model. Alternatively, the joint similarity function can also be deployed in the algorithm service (i.e., not inside the AI large model) of the server 200 itself, and the joint similarity calculation is realized through the algorithm service. Specifically, it can be flexibly deployed according to the actual application scenario, and the embodiments of the present application do not make specific limitations here.

[0046] That is, the AI large model obtains the key vector (i.e., the feature vector of the work order) by disassembling the abnormal data in the work order. The key vector can be metadata-related description, requirement class description, and impact range description, etc. The associated data related to the abnormal data is retrieved in the metadata database by taking the metadata-related description in the key vector as an index. For example, which field in the abnormal data has a problem, the related information such as table and system is traced back through the metadata database, and the upstream data (i.e., the blood relationship node) can be obtained from the blood relationship knowledge base through these related information. The upstream data is the data processed by the abnormal data. Through the requirement class description and the impact range description in the key vector and other description information, the related requirement content can be obtained from the document knowledge base, and the more detailed requirement document or other explanation document (as a specific example of the requirement document) of the current field is obtained. Alternatively, the above-mentioned retrieved associated data and the problem list of the abnormal data are stacked together and searched in the document knowledge base, and then the ETL (Extract-Transform-Load) process, field association that may be involved are circled and the search range is narrowed down in the blood relationship knowledge base to obtain the requirement document.

[0047] The AI can preliminarily infer the business logic and the change of the data in the blood relationship knowledge base through the above-mentioned requirement document and the description of the upstream data and the abnormal data.

[0048] That is, the AI will use the document structured features (such as title hierarchy, table, code block) in the work order where the abnormal data is located to extract keywords and sentences, extract structured data such as field mapping relationship, transformation rule, and execution node from the ETL blood relationship knowledge base configuration, and then obtain chapters such as "function module" and "parameter description" from the document knowledge base to grab the configuration details related to the problem and obtain the requirement document.

[0049] S220, simulating calculation on the associated data based on the business logic to obtain verification data.

[0050] For example, in the specific embodiments of the present application, the AI can simulate the upstream data through the data processing process exhibited by the business logic to obtain the verification data.

[0051] In some embodiments of the present application, S220 can include automatically writing a script and using the script to simulate the data processing process in the business logic to calculate the associated data to obtain the verification data.

[0052] In some specific embodiments, automatically writing a script specifically includes that the server first converts the requirement document into a structured business logic representation according to the business rules extracted from the requirement document, and the business logic representation at least includes an input field set, an output field set, a data transformation operation sequence, and a filtering condition. The server selects a target script template matching the business logic type from a preset script template library based on the business logic representation. The script template library has pre-stored script templates for different data processing scenarios such as single-table aggregation, multi-table connection, and window function calculation.

[0053] After selecting the target script template, the AI large model maps the input fields, output fields, and filtering conditions in the business logic representation to the placeholder positions in the target script template according to the table names and field names recorded in the meta database, to generate a target script (referred to as script) containing specific table names, field names, and calculation rules. Before submitting the target script to the sandbox environment for execution, the server can also perform static analysis on the target script to prohibit access to tables or fields unrelated to the current work order and preset sensitive data tables, to ensure the security of script execution. After executing the target script in the sandbox environment, the verification data corresponding to the abnormal data can be obtained.

[0054] For example, in the specific embodiments of the present application, the AI can intelligently write scripts such as SQL and Python, read upstream data, and simulate the data processing process to obtain the verification data. According to the associated data extracted above and the details in the requirement document, a verification script is written. Then, the ETL execution process or rule verification logic is simulated, and the script is automatically run. The abnormal data is compared with the expected result (i.e., verification data) in the work order to verify the accuracy of the abnormal data.

[0055] S230, comparing the verification data and the abnormal data to locate the abnormal position; wherein the abnormal position is any layer in the multi-layer upstream data.

[0056] For example, in the specific embodiments of the present application, the AI can compare the verification data and the abnormal data for consistency to locate which layer in the multi-layer upstream data the abnormal position is in, to obtain the abnormal position.

[0057] In some embodiments of the present application, S220 can include: employing a recursive algorithm, simulating and calculating the i-th layer upstream data in the multi-layer upstream data by the service logic to obtain the i-th check data; wherein i is a positive integer, and when i = 1, the first layer upstream data is the upstream data adjacent to the abnormal data. S230 can include: confirming that the i-th check data is different from the abnormal data, and then taking the i-th layer upstream data as the abnormal position.

[0058] It should be noted that in some embodiments of the present application, before simulating and calculating the multi-layer upstream data by the recursive algorithm, the server can also calculate the abnormal confidence of each layer of upstream data node based on the work order content and the information in the metadata base and the blood relationship knowledge base. Specifically, the server can comprehensively consider the latest change time of the node corresponding task, the complexity of the business rules involved, and the keyword matching situation related to the node in the work order text, etc. factors, and generate an abnormal confidence value through a preset model or scoring function.

[0059] In the recursive backtracking process, the server sorts the upstream data nodes according to the abnormal confidence, and preferentially simulates and calculates the nodes with higher abnormal confidence (i.e. greater than or equal to the confidence threshold), and temporarily does not expand the simulation calculation for the nodes with lower abnormal confidence, thereby reducing the number of simulation calculations to be performed. When the check data of the nodes with higher confidence is compared with the abnormal data and no abnormality is found, the preset confidence threshold can be gradually reduced, and the nodes with lower abnormal confidence can be sequentially simulated and calculated until the abnormal position is determined. The confidence threshold can be gradually reduced by one or more times of the set unit value, which can be selected according to actual conditions.

[0060] For example, in a specific embodiment of the present application, when analyzing the problem point (as a specific example of the abnormal position) where the abnormal data is located, the AI employs a recursive method to analyze the upstream data layer by layer, traces the entire complete data link and stores it in the memory, and lets the problem be repeatedly verified in the entire link.

[0061] For example, taking the first layer upstream data as an example, the simulation upstream computing logic calculates the first layer upstream data to obtain first check data, and the first check data and the abnormal data are matched and analyzed. If they are different, it indicates that the problem point is in the first layer upstream data node. If they are the same, the second layer upstream data and the first layer business upstream data are calculated according to the business logic to obtain second check data. The second check data and the abnormal data are matched and analyzed. If they are different, it indicates that the problem point is in the second layer upstream data node. If they are the same, the third layer upstream data is taken as a starting point for calculation to obtain third check data. The consistency comparison operation is continued to trace back in this way until the location of the problem point is determined.

[0062] Since the data chain involved in the abnormal data is relatively long, it is not a simple upstream and downstream problem. If it is verified that the adjacent upstream is not a problem, it still needs to continue to trace upwards. One problem may be caused by the superposition of multiple problem points. After the most upstream problem point is found, the complete calculation logic complete verification is generated according to the entire problem link.

[0063] In addition to using the recursive method, the method opposite to the recursive method can also be used for tracing back, for example, the original data at the beginning of obtaining the abnormal data is calculated downwards, and the consistency of each layer of calculation results is compared to determine the problem point that exists abnormally. Specifically, it can be flexibly adjusted according to actual conditions, and the embodiments of the present application are not limited specifically herein.

[0064] In some embodiments of the present application, after S230 is performed, the data abnormality positioning method can further include: determining an abnormal reason of the abnormal position; generating a correction suggestion corresponding to the abnormal reason; wherein the correction suggestion includes a correction code, a correction scheme, a correction basis or a correction reference document.

[0065] For example, in specific embodiments of the present application, the AI can also automatically locate the abnormal reason caused by the abnormal position and generate a correction suggestion. Finally, the abnormal position, the abnormal reason and the correction suggestion can be written into a comprehensive report for technical personnel of the responsible department to check and record in the knowledge base.

[0066] The process of generating a correction suggestion is exemplarily described below based on several abnormal conditions.

[0067] In some embodiments, after determining the abnormal position, the server first classifies the abnormal phenomenon (i.e., the abnormal cause) using a rule engine. Specifically, the server classifies the abnormality into several initial categories such as suspected input error, suspected blood relationship configuration or code logic error, suspected demand or rule understanding deviation, and suspected metadata definition problem, according to the difference pattern between the check data and the abnormal data, the distribution of the abnormal data on the time series, and the deviation of the business indicators.

[0068] The server provides the above initial classification results, data link information corresponding to the abnormal position, and local data distribution characteristics as inputs to a pre-trained model to make a fine-grained determination of the abnormal cause and obtain the target abnormal cause type (referred to as abnormal cause). Subsequently, the server retrieves from a pre-set knowledge base the historical problem case most similar to the current problem and its corresponding repair scheme and verification script template according to the target abnormal cause type, and automatically generates a correction suggestion containing correction code and test cases; for input error type problems, the server can also give a reference correction value based on the statistical characteristics of similar data.

[0069] For example, for the case of input error (as a specific example of abnormal cause), if no explicit expected data is given, the reference data (as a specific example of correction suggestion) can be given by reading other correct values in the database. And according to the business description of the current error point (for example, the data is wrong), the AI speculates the cause of the problem (for example, input error), and gives a correction scheme (for example, re-enter). For the case of data blood relationship configuration or code logic error, point out the ETL field mapping missing configuration, conversion rule (as a specific example of abnormal cause), AI can give correction code, and give test cases and execution results of correction code to ensure the availability of the correction code. For the case of demand and rule understanding deviation, specify the demand and rule deviation problem (as a specific example of abnormal cause), and recommend a modification scheme and the corresponding responsible department. For the metadata definition problem (as a specific example of abnormal cause), the corresponding responsible person can be directly informed of the correction suggestion (such as modifying the metadata definition) for correction. In addition, problems that cannot be automatically processed can also be processed manually, etc. It should be noted that the correction suggestion can include problem root, problem correction method, correction basis, reference document, reference code (for correction code), or business problem, etc. Specifically, different correction suggestions can be generated for different abnormal causes to timely handle the abnormality, which is not limited herein.

[0070] The following will be described in conjunction with the accompanying Figure 3 The specific process of data anomaly positioning provided by some embodiments of the present application is exemplarily described.

[0071] Please refer to the accompanyingFigure 3 , Figure 3 A method flowchart of data anomaly positioning is provided for some embodiments of the present application.

[0072] The above process is exemplarily described below.

[0073] S310, receiving a work order and reading abnormal data in the work order.

[0074] S320, predicting expected data corresponding to the abnormal data based on other data in the work order except the abnormal data.

[0075] S330, verifying that the abnormal data is abnormal by the expected data.

[0076] S340, extracting key fields in the abnormal data.

[0077] S350, acquiring multi-layer upstream data and requirement documents from a metadata database, a blood relationship knowledge base and a document knowledge base according to the key fields.

[0078] The upstream data includes N layers, and N is a positive integer.

[0079] S360, performing simulation calculation on the i-th layer upstream data by business logic in the requirement documents to obtain i-th verification data.

[0080] S370, judging whether the verification data and the abnormal data are the same, if yes, confirming that i is less than N, setting i=i+1 and returning to S360, otherwise, executing S380.

[0081] S380, determining that the abnormal position is the i-th layer upstream data.

[0082] S390, determining an abnormal reason of the abnormal position and generating a correction suggestion, and outputting a comprehensive report.

[0083] It can be understood that the specific implementation process of S310-S390 can refer to the method embodiments provided above, and the detailed description is appropriately omitted here to avoid repetition.

[0084] According to some embodiments of the present application, the AI in the present application can obtain global data in the data warehouse, realize rapid reading of data, and give comprehensive problem analysis. In link analysis of abnormal data positioning, the test script on the link node is automatically written, executed, and verified more quickly and accurately. The AI can verify in multiple dimensions, complete the verification of the whole link without the cooperation of other departments, solve the problem across departments, and ensure the safety of data without leakage risk. The AI can record complete problem analysis and processing links, cooperate with a large amount of knowledge, analyze the processing logic or intermediate algorithm problem of data, and not just solve the occurrence point of the problem. Through complete memory in problem processing, the global scope of problem influence is analyzed, and relevant personnel are informed. In the whole process, AI automation is completed, which can effectively ensure the timeliness of data, quickly analyze and solve problems. Finally, the problem solving process is archived for future reference and statistics.

[0085] Reference is made to Figure 4 , Figure 4 A component block diagram of the data abnormality positioning apparatus provided by some embodiments of the present application is shown. It should be understood that the data abnormality positioning apparatus corresponds to the above-mentioned method embodiments, and can perform each step involved in the above-mentioned method embodiments. The specific functions of the data abnormality positioning apparatus can be referred to the description in the foregoing, and the detailed description is appropriately omitted here to avoid repetition.

[0086] Figure 4 The data abnormality positioning apparatus includes at least one software function module stored in the form of software or firmware in the memory or solidified in the data abnormality positioning apparatus. The data abnormality positioning apparatus includes: an acquisition module 410, configured to acquire associated data corresponding to abnormal data in a work order and a requirement document; wherein the requirement document is used to deduce the business logic between the abnormal data and the associated data; and the associated data includes multi-layer upstream data; a calculation module 420, configured to perform simulation calculation on the associated data based on the business logic to acquire verification data; and a positioning module 430, configured to compare the verification data with the abnormal data to locate an abnormal position; wherein the abnormal position is any layer in the multi-layer upstream data.

[0087] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the apparatus described above can refer to the corresponding process in the foregoing method, and will not be described in more detail here.

[0088] Some embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the operations of the method corresponding to any of the above-mentioned embodiments provided by the above-mentioned embodiments.

[0089] Some embodiments of the present application further provide a computer program product, which comprises a computer program, wherein the computer program can realize the operation of the method corresponding to any of the above-mentioned embodiments when executed by a processor.

[0090] As shown in Figure 5 Some embodiments of the present application provide an electronic device 500, which comprises a memory 510, a processor 520 and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 can realize the method of any of the above-mentioned embodiments when reading the program from the memory 510 and executing the program through the bus 530.

[0091] The processor 520 can process digital signals and can comprise various computing structures, such as a complex instruction set computer structure, a reduced instruction set computer structure or a structure implementing a combination of multiple instruction sets. In some examples, the processor 520 can be a microprocessor.

[0092] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of the instructions. These instructions and / or data can include code for implementing some or all of the functions of one or more modules described in the embodiments of the present application. The processor 520 of the embodiments of the present disclosure can be used to execute the instructions in the memory 510 to implement the above-mentioned method. The memory 510 comprises a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memories well known to those skilled in the art.

[0093] The above only describes the embodiments of the present application and does not limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0094] The above only describes the embodiments of the present application and does not limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0095] It is to be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a combination of two or more components. Additionally, the terms "comprise," "comprises," and "comprising," or any variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless otherwise indicated herein, the terms "first," "second," "third," etc., are used herein merely as labels, and are not intended to impose ordinal import.

Claims

1. A method for locating data anomalies, characterized in that, include: Obtain the associated data and requirement document corresponding to the abnormal data in the work order; wherein, the requirement document is used to infer the business logic between the abnormal data and the associated data; the associated data includes multiple layers of upstream data; Based on the business logic, simulated calculations are performed on the associated data to obtain verification data; The verification data and the abnormal data are compared to locate the abnormal position; wherein, the abnormal position is any layer in the multi-layer upstream data.

2. The method as described in claim 1, characterized in that, Before obtaining the associated data and requirement documents corresponding to the abnormal data in the work order, the method further includes: Receive the work order and read the abnormal data; Based on the other data in the work order besides the abnormal data, predict the expected data corresponding to the abnormal data; The expected data verification confirmed that the abnormal data was indeed abnormal.

3. The method as described in claim 1 or 2, characterized in that, The process of obtaining the associated data and requirement documents corresponding to the abnormal data in the work order includes: Extract the key fields from the abnormal data; The system retrieves the associated data matching the key field from the metadata database and retrieves the requirement document corresponding to the key field from the document knowledge base. The data in the metadata database is stored in a structured manner, and the data in the document knowledge base is stored in a vector manner.

4. The method as described in claim 1 or 2, characterized in that, The step of simulating calculations on the associated data based on the business logic to obtain verification data includes: The script is automatically written and used to simulate the data processing in the business logic to calculate the associated data and obtain the verification data.

5. The method as described in claim 1 or 2, characterized in that, The step of simulating calculations on the associated data based on the business logic to obtain verification data includes: A recursive algorithm is used to simulate and calculate the i-th layer of upstream data in the multi-layer upstream data through the business logic to obtain the i-th verification data; where i is a positive integer, and when i=1, the first layer of upstream data is the upstream data adjacent to the abnormal data; The step of comparing the verification data and the abnormal data to locate the abnormal location includes: If the i-th verification data is confirmed to be different from the abnormal data, then the upstream data of the i-th layer is taken as the abnormal position.

6. The method as described in claim 1 or 2, characterized in that, After locating the abnormal position, the method further includes: Determine the cause of the anomaly at the aforementioned location; Generate correction suggestions corresponding to the cause of the anomaly; wherein, the correction suggestions include correction code, correction scheme, correction basis or correction reference document.

7. A device for locating data anomalies, characterized in that, include: The acquisition module is used to acquire the associated data and requirement documents corresponding to the abnormal data in the work order; wherein, the requirement documents are used to infer the business logic between the abnormal data and the associated data; the associated data includes multi-layer upstream data; The calculation module is used to perform simulated calculations on the associated data based on the business logic to obtain verification data; The positioning module is used to compare the verification data and the abnormal data to locate the abnormal position; wherein the abnormal position is any layer in the multi-layer upstream data.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-6.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-6.

Images