Data processing method and device, equipment, medium and program product

Abstract

The invention provides a data processing method which can be applied to the field of financial science and technology and the field of big data technology. The data processing method comprises the following steps: extracting a first primary key from a to-be-processed source table; abnormal data detection is conducted on the first main key, under the condition that the first main key does not comprise first abnormal data, a batch processing instruction is generated, and the first abnormal data represents data, making multiple downstream applications report errors, in the first main key; in response to a received batch processing instruction, calling a batch processing statement to perform batch processing on the source table to obtain a mart table; extracting a second major key from the market table; abnormal data detection is conducted on the second main key, under the condition that the second main key does not comprise second abnormal data, the multiple downstream applications are allowed to read the market table, and the second abnormal data represent data, making the multiple downstream applications report errors, in the second main key. The invention further provides a data processing device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and more specifically to a data processing method, device, equipment, medium and program product. Background Art

[0002] In some data processing systems, data marts integrate data from multiple business systems based on specific rules and requirements to form a unified business data set for use by multiple downstream applications. Some abnormal data in the business system can cause multiple downstream applications to report errors, affecting the normal reading and use of data. When multiple downstream application jobs report errors, the troubleshooting link is usually long, and some indicators are difficult to trace. It is necessary to penetrate upstream layers to locate the cause of the problem, which is difficult to troubleshoot and affects the efficiency of error handling, increasing operation and maintenance costs. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a data processing method, apparatus, device, medium and program product for improving error handling efficiency.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, including: extracting a first primary key from a source table to be processed, the first primary key being used to identify record data of the source table; performing abnormal data detection on the first primary key, and generating a batch processing instruction when the first primary key does not include the first abnormal data, the first abnormal data representing data in the first primary key that causes multiple downstream applications to report errors; in response to receiving the batch processing instruction, calling a batch processing statement to perform batch processing on the source table to obtain a mart table corresponding to the source table; extracting a second primary key from the mart table, the second primary key being used to identify record data of the mart table; performing abnormal data detection on the second primary key, and allowing multiple downstream applications to read the mart table when the second primary key does not include the second abnormal data, the second abnormal data representing data in the second primary key that causes multiple downstream applications to report errors.

[0005] According to an embodiment of the present disclosure, before extracting the first primary key from the source table to be processed, it also includes: receiving an initial source table sent by a data warehouse; determining the access dependencies of multiple downstream applications on each sub-source table in the initial source table, extracting the sub-source tables whose access dependencies exceed a preset threshold, and forming a source table to be processed.

[0006] According to an embodiment of the present disclosure, extracting the first primary key from a source table includes: organizing the source table into a dimension table and a fact table; extracting the first sub-primary key in the dimension table and the second sub-primary key in the fact table to obtain the first primary key; wherein the first sub-primary key is used to identify the record data of the dimension table, the second sub-primary key is used to identify the record data of the fact table, and the first sub-primary key and the second sub-primary key are associated with each other.

[0007] According to an embodiment of the present disclosure, abnormal data detection is performed on the first primary key, and when the first primary key does not include the first abnormal data, generating a batch processing instruction includes: performing abnormal data detection on the first sub-primary key and the second sub-primary key respectively, and when both the first sub-primary key and the second sub-primary key do not include the first abnormal data, generating a batch processing instruction.

[0008] According to an embodiment of the present disclosure, the data processing method also includes: when the first primary key includes first abnormal data, caching the first primary key and generating a first warning message, the first warning message is used to display the detection result of the first primary key to the first user, so that the first user updates the data in the first primary key according to the warning message.

[0009] According to an embodiment of the present disclosure, the data processing method also includes: when the second primary key includes second abnormal data, caching the second primary key and generating a second warning message, the second warning message is used to display the detection result of the second primary key to the second user, so that the second user updates the data in the second primary key according to the second warning message.

[0010] According to an embodiment of the present disclosure, the first abnormal data includes duplicate primary key data; performing abnormal data detection on the first primary key, and generating a batch processing instruction when the first primary key does not include the first abnormal data, includes: performing duplicate checking on the first primary key, and generating a batch processing instruction when the first primary key does not include duplicate primary key data.

[0011] According to an embodiment of the present disclosure, the second abnormal data includes duplicate primary key data; abnormal data detection is performed on the second primary key, and when the second primary key does not include the second abnormal data, multiple downstream applications are allowed to read the mart table, including: performing duplicate checking on the second primary key, and when the second primary key does not include duplicate primary key data, multiple downstream applications are allowed to read the mart table.

[0012] A second aspect of the present disclosure provides a data processing device, including: a first extraction module, used to extract a first primary key from a source table to be processed, the first primary key being used to identify record data of the source table; a first detection module, used to perform abnormal data detection on the first primary key, and to generate a batch processing instruction when the first primary key does not include the first abnormal data, the first abnormal data representing data in the first primary key that causes multiple downstream applications to report errors; a batch processing module, used to call a batch processing statement to perform batch processing on the source table in response to receiving the batch processing instruction, and obtain a mart table corresponding to the source table; a second extraction module, used to extract a second primary key from the mart table, the second primary key being used to identify record data of the mart table; a second detection module, used to perform abnormal data detection on the second primary key, and to allow multiple downstream applications to read the mart table when the second primary key does not include the second abnormal data, the second abnormal data representing data in the second primary key that causes multiple downstream applications to report errors.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0015] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] Based on the above embodiments, by performing abnormal data detection on the source table, batch processing to obtain the mart table after the detection is correct, and then performing abnormal data detection on the mart table, and transmitting it to the downstream application after the detection is correct, the error that appears in the downstream application can be accurately located in the data mart layer, so as to facilitate the rapid processing of abnormal data in the data mart layer in advance, thereby improving the efficiency of error handling. Detecting abnormal data in the primary key of the source table and the mart table, rather than performing abnormal data detection on the entire source table and the mart table, because the primary key can identify the record data in the source table and the mart table, it is possible to quickly and accurately detect abnormal data, thereby improving the efficiency of abnormal data detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0019] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0020] Figure 3 A flowchart schematically shows a data processing method according to another embodiment of the present disclosure;

[0021] Figure 4 A structural block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0022] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] It should be noted that the data processing method and device disclosed herein can be used for data processing applications in the financial field, and can also be used for data processing applications in any field other than the financial technology field. The application field of the data processing method and device disclosed herein is not limited.

[0028] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0029] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0031] In order to process large-scale data, the main focus of existing online analytical processing systems is on data storage and processing efficiency, rather than maintaining the relationship between data through primary keys or foreign keys. This will cause the problem of abnormal primary key data to be hidden, and the problem will eventually be exposed in the Cartesian product association of multiple tables at the downstream application end. The downstream application's references to upstream data tables are intricate, and the downstream application's own processing logic is also complex. After the downstream application reports an error, it takes a lot of time to trace the source of the problem, resulting in increased operating and maintenance costs. In addition, the tracing process requires certain professional skills, which also increases the difficulty of tracing.

[0032] In view of this, an embodiment of the present disclosure provides a data processing method, including: extracting a first primary key from a source table to be processed, the first primary key being used to identify record data of the source table; performing abnormal data detection on the first primary key, and generating a batch processing instruction when the first primary key does not include the first abnormal data, the first abnormal data representing data in the first primary key that causes multiple downstream applications to report errors; in response to receiving the batch processing instruction, calling a batch processing statement to batch process the source table to obtain a mart table corresponding to the source table; extracting a second primary key from the mart table, the second primary key being used to identify record data of the mart table; performing abnormal data detection on the second primary key, and allowing multiple downstream applications to read the mart table when the second primary key does not include the second abnormal data, the second abnormal data representing data in the second primary key that causes multiple downstream applications to report errors.

[0033] Figure 1The application scenario diagram of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0034] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0035] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0037] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The data processing method of the disclosed embodiment is described in detail.

[0041] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0042] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S250, and the transaction processing method can be executed by the server 105.

[0043] In operation S210 , a first primary key is extracted from a source table to be processed, where the first primary key is used to identify record data of the source table.

[0044] In database management, a source table usually contains a large number of data records, which can be identified by a primary key. The first primary key can be a combination of one or more fields in the source table, whose value is unique in the table, and is used to ensure that each record can be accurately distinguished. For example, in a source table containing ID card number information, since each person's ID card number is uniquely determined, the ID card number can be used as the primary key of the table.

[0045] In operation S220, abnormal data detection is performed on the first primary key, and a batch processing instruction is generated when the first primary key does not include first abnormal data, where the first abnormal data represents data in the first primary key that causes multiple downstream applications to report errors.

[0046] Abnormal data can be data that does not conform to the expected format, range, or logic rules, which may cause errors in downstream applications when processing or analyzing data. As an example, the abnormal data of the primary key can be one or more of the following: empty, duplicate, or contain illegal characters. Batch processing instructions can include data conversion, loading to the target table, performing aggregation operations, etc. Data validation tools or scripts can be used to detect abnormal data on the first primary key.

[0047] In operation S230 , in response to receiving the batch processing instruction, a batch processing statement is called to batch process the source table to obtain a mart table corresponding to the source table.

[0048] Batch statements can include data conversion statements, data aggregation statements, data loading statements, etc. Data Mart Table is a table used to store analytical data. It can be designed according to specific business requirements and can contain fields and formats required for analysis. For example, in the field of financial technology, a data mart table can store transaction financial data and user behavior data. Transaction financial data can include order data, account data, etc.

[0049] In operation S240, a second primary key is extracted from the mart table, where the second primary key is used to identify record data of the mart table.

[0050] The process of extracting the second primary key in the mart table can be similar to the process of extracting the first primary key in the source table. In the process of batch processing some source tables into mart tables, the primary key information in the source table can also be the primary key information in the mart table, that is, the second primary key and the first primary key can be the same. In the process of batch processing some source tables into mart tables, the primary key information in the source table and the primary key information in the mart table may change, that is, the second primary key and the first primary key can be different. It can be determined according to different data tables and processing requirements, and the present disclosure does not make specific limitations.

[0051] In operation S250, abnormal data detection is performed on the second primary key. When the second primary key does not include second abnormal data, multiple downstream applications are allowed to read the mart table. The second abnormal data represents data in the second primary key that causes multiple downstream applications to report errors.

[0052] The process of detecting abnormal data for the second primary key may be similar to the process for the first primary key. The type of abnormal data detected in the second primary key may be the same as or different from the type of abnormal data detected in the first primary key.

[0053] By performing abnormal data detection on the source table, batch processing to obtain the mart table after the detection is correct, and then performing abnormal data detection on the mart table again, and transmitting it to the downstream application after the detection is correct, the error that appears in the downstream application can be accurately located in the data mart layer, so that the abnormal data can be quickly processed in the data mart layer in advance, improving the efficiency of error handling. Detecting abnormal data in the primary key of the source table and the mart table, rather than detecting abnormal data on the entire source table and the mart table, because the primary key can identify the record data in the source table and the mart table, it can quickly and accurately detect abnormal data, improving the efficiency of abnormal data detection.

[0054] Based on the above embodiment, before extracting the first primary key from the source table to be processed, the data processing method may further include: receiving an initial source table sent by a data warehouse; determining the access dependency of multiple downstream applications on each sub-source table in the initial source table, extracting the sub-source tables whose access dependency exceeds a preset threshold, and forming the source table to be processed.

[0055] The initial source table sent by the data warehouse may be a table after source processing, and may contain multiple sub-source tables, each of which may correspond to a specific business field or data set. These sub-source tables may be divided into access tables with different dependencies according to the access dependency. The division may be performed by analyzing the access mode and usage of each sub-source table in the initial source table by the downstream application. According to the analysis results, the frequency, importance, and degree of influence of each sub-source table accessed by the downstream application on its business logic are determined. The preset threshold may be determined based on factors such as access frequency, data volume, and business importance. For example, a sub-source table with a frequency higher than a certain set frequency accessed by the downstream application, i.e., a high-frequency dependency table, may be set as a source table to be processed, and the above-mentioned operations S210 to S250 may be performed. A sub-source table with a frequency lower than the set frequency, i.e., a low-frequency access dependency table may be processed in a general manner, for example, a batch process may be directly performed on the low-frequency access dependency table, which is not specifically limited in the present disclosure.

[0056] In this way, the data processing method of this embodiment can focus on source tables with high access frequencies instead of performing abnormal data detection and processing on all source tables. It can reduce the amount of data processing in the data mart and improve data processing efficiency while reducing the difficulty of downstream application error troubleshooting.

[0057] On the basis of the above embodiment, extracting the first primary key from the source table includes: organizing the source table into a dimension table and a fact table; extracting the first sub-primary key in the dimension table and the second sub-primary key in the fact table to obtain the first primary key; wherein the first sub-primary key is used to identify the record data of the dimension table, the second sub-primary key is used to identify the record data of the fact table, and the first sub-primary key and the second sub-primary key are associated with each other.

[0058] For example, the source table can be organized into dimension tables and fact tables through the dimensional model to be suitable for data warehouse and data mart environments. Dimension tables can contain descriptive fields to provide context and classification information. For example, in banking business, it can be card information, contract information, sub-account information, and loan information. Fact tables can contain metric fields to store actual data values, such as loan transaction details and channel transaction flow.

[0059] In the dimension table, one or more fields can be determined as the first sub-primary key to uniquely identify each record in the dimension table, such as an ID number, date, etc. In the fact table, one or more fields can be determined as the second sub-primary key to uniquely identify each record in the fact table. The first sub-primary key in the dimension table can be associated with the second sub-primary key in the fact table through business logic or data relationships. After the association, the data consistency and integrity between the dimension table and the fact table can be ensured.

[0060] In the framework of data processing and analysis, source tables usually contain a large amount of data that needs to be effectively organized and accessed to meet business needs. Processing the source tables to be processed into dimension tables and fact tables can optimize data access and improve query performance.

[0061] On the basis of the above embodiment, abnormal data detection is performed on the first primary key, and when the first primary key does not include the first abnormal data, generating a batch processing instruction includes: performing abnormal data detection on the first sub-primary key and the second sub-primary key respectively, and when both the first sub-primary key and the second sub-primary key do not include the first abnormal data, generating a batch processing instruction.

[0062] The process of detecting abnormal data for the first sub-primary key and the second sub-primary key is similar to the process of detecting abnormal data for the first primary key, and the similar parts are not repeated here. It should be emphasized that when neither the first sub-primary key nor the second sub-primary key contains abnormal data, it indicates that the first primary key does not contain abnormal data. The detection of abnormal data in the first sub-primary key and the second sub-primary key can be completed synchronously or separately. It can be selected and set according to the usage. Synchronous completion can reduce the time of abnormal data detection, and separate completion can reduce the calculation load per unit time.

[0063] Based on the above embodiment, the data processing method may further include: when the first primary key includes first abnormal data, caching the first primary key and generating a first warning message, the first warning message is used to display the detection result of the first primary key to the first user, so that the first user updates the data in the first primary key according to the warning message.

[0064] The first primary key can be cached in a temporary storage area to save abnormal data to be processed or reviewed. The present disclosure does not limit the specific storage area. Through data caching, it can be ensured that the abnormal data will not be deleted or overwritten immediately, so that further analysis and processing can be performed later. The first warning message may include detailed information about the abnormal data, such as the type of abnormality, the location of occurrence, the scope of impact, etc. The first warning message can be sent in the form of text messages, message push, system alarms, etc. The warning message can be used to promptly notify the first user (such as data operation and maintenance personnel) about the abnormal data so that they can take appropriate measures to solve the problem, such as replacing, deleting or adding part of the data, thereby improving the efficiency of abnormal data resolution and user satisfaction.

[0065] Based on the above embodiment, the data processing method may further include: when the second primary key includes second abnormal data, caching the second primary key and generating a second warning message, the second warning message is used to display the detection result of the second primary key to the second user, so that the second user updates the data in the second primary key according to the second warning message.

[0066] The process of caching the second primary key and sending a warning message is similar to the process of caching the first primary key and sending a warning message, and similar content will not be repeated here. It should be noted that the second primary key can be cached in the same or different temporary storage area as the first primary key. The second user and the first user can be the same, or they can be different users responsible for handling different businesses. The second warning message and the first warning message can be sent through the same sending platform to simplify the system design.

[0067] Based on the above embodiment, the first abnormal data may include duplicate primary key data; performing abnormal data detection on the first primary key, and generating a batch processing instruction when the first primary key does not include the first abnormal data, includes: performing duplicate checking on the first primary key, and generating a batch processing instruction when the first primary key does not include duplicate primary key data.

[0068] Duplicate primary key data refers to two or more records in the same table that have the same primary key value. For example, you can use SQL queries, data processing tools or scripts to check for duplicates on the first primary key to identify all records with duplicate primary key values. Errors caused by duplicate primary key data are common in downstream applications and difficult to trace. This embodiment focuses on detecting duplicate primary key data in the primary key, and can promptly find abnormal data in the source table and process it in a targeted manner, reducing multiple errors in subsequent links, thereby improving the processing efficiency of data errors.

[0069] For the duplicate primary keys screened out, the duplicate records can be deleted, related records can be merged, the primary key value can be updated, etc. to ensure the uniqueness of the primary key. However, the present disclosure is not limited thereto.

[0070] Based on the above embodiment, the second abnormal data includes duplicate primary key data; abnormal data detection is performed on the second primary key, and when the second primary key does not include the second abnormal data, multiple downstream applications are allowed to read the mart table, including: duplicate checking is performed on the second primary key, and when the second primary key does not include duplicate primary key data, multiple downstream applications are allowed to read the mart table.

[0071] The process of checking duplicates for the second primary key is similar to that of checking duplicates for the first primary key, and similar contents are not repeated here. Checking duplicates for the second primary key can timely discover duplicate primary keys in the mart table, so that there are no duplicate primary keys in the mart table that enters multiple downstream applications, avoiding multiple errors when downstream applications reference data in the mart table, thereby improving the efficiency of error handling.

[0072] Figure 3 The flowchart of a data processing method according to another embodiment of the present disclosure is schematically shown.

[0073] like Figure 3 As shown, in the data processing method of this implementation, the source tables sent by the data warehouse can be first divided into downstream high-frequency dependent tables and low-frequency access tables. The downstream high-frequency dependent tables are processed using the data processing method provided by the present disclosure, and the primary key of the source table is detected for abnormal data through a pre-processing command, such as checking the duplicate processing of the primary key of the source table, detecting whether the sum and comparison of the amount field of the source table referenced downstream exceeds the length of the reporting task data, etc. If passed, the post-processing command is executed. Then, the primary key of the mart table is detected for abnormal data. If passed, the mart table is sent to downstream applications for use. If the abnormal data detection of the source table or the mart table fails, a warning message A or a warning message B is sent respectively to prompt the job error. The low-frequency access table can be directly batch processed and used by different downstream applications a, b, c, d, e, d, f, etc. after processing.

[0074] For a certain high-frequency dependency table, it can be used by downstream applications a, b, c, d, and e. If abnormal data appears in the high-frequency dependency table, the traditional method may cause errors in downstream applications a, b, c, d, and e, with a total of 5 errors. Since each downstream application may use several dependency tables at the same time, it is difficult to quickly locate when there are a large number of downstream applications and dependency tables. However, by adopting the data processing method of this embodiment, the errors that originally appeared in the application market can be exposed to the regulatory market, and 5 errors from multiple applications can be reduced to 1 error in the regulatory market. In this way, by shielding or exposing the problem at the market level, the downstream multi-application errors are intercepted upstream, reducing errors and improving the efficiency of batch error handling.

[0075] Based on the above data processing method, the present disclosure also provides a data processing device. Figure 4 The device is described in detail.

[0076] Figure 4 The structure block diagram of a data processing device according to an embodiment of the present disclosure is schematically shown.

[0077] like Figure 4 As shown, the data processing device 400 of this embodiment includes a first extraction module 410 , a first detection module 420 , a batch processing module 430 , a second extraction module 440 and a second detection module 450 .

[0078] The first extraction module 410 is used to extract a first primary key from the source table to be processed, and the first primary key is used to identify the record data of the source table. In one embodiment, the first extraction module 410 can be used to perform the operation S210 described above, which will not be described in detail here.

[0079] The first detection module 420 is used to detect abnormal data on the first primary key, and generate a batch processing instruction when the first primary key does not include the first abnormal data, where the first abnormal data represents the data in the first primary key that causes multiple downstream applications to report errors. In one embodiment, the first detection module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0080] The batch processing module 430 is used to call the batch processing statement to batch process the source table in response to receiving the batch processing instruction, and obtain the mart table corresponding to the source table. In one embodiment, the batch processing module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0081] The second extraction module 440 is used to extract the second primary key from the mart table, and the second primary key is used to identify the record data of the mart table. In one embodiment, the second extraction module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0082] The second detection module 450 is used to detect abnormal data on the second primary key. When the second primary key does not include the second abnormal data, multiple downstream applications are allowed to read the market table. The second abnormal data represents the data in the second primary key that causes multiple downstream applications to report errors. In one embodiment, the second detection module 450 can be used to perform the operation S250 described above, which will not be repeated here.

[0083] Based on the above embodiment, the data processing device 400 may further include a partitioning module for receiving an initial source table sent by a data warehouse; determining the access dependency of multiple downstream applications on each sub-source table in the initial source table, extracting the sub-source tables whose access dependency exceeds a preset threshold, and forming a source table to be processed.

[0084] Based on the above embodiment, the data processing device 400 may further include a first warning module, which is used to cache the first primary key and generate a first warning message when the first primary key includes first abnormal data. The first warning message is used to display the detection result of the first primary key to the first user, so that the first user can update the data in the first primary key according to the warning message.

[0085] Based on the above embodiment, the data processing device 400 may further include a second warning module, which is used to cache the second primary key and generate a second warning message when the second primary key includes second abnormal data. The second warning message is used to display the detection result of the second primary key to the second user, so that the second user can update the data in the second primary key according to the second warning message.

[0086] According to an embodiment of the present disclosure, any multiple modules of the first extraction module 410, the first detection module 420, the batch processing module 430, the second extraction module 440, and the second detection module 450 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first extraction module 410, the first detection module 420, the batch processing module 430, the second extraction module 440, and the second detection module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the first extraction module 410, the first detection module 420, the batch processing module 430, the second extraction module 440 and the second detection module 450 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0087] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown.

[0088] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0089] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0090] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0091] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0092] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0093] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present disclosure.

[0094] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0095] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0096] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0097] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0098] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0100] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, characterized in that: include: Extracting a first primary key from a source table to be processed, where the first primary key is used to identify record data of the source table; Performing abnormal data detection on the first primary key, and generating a batch processing instruction when the first primary key does not include first abnormal data, wherein the first abnormal data represents data in the first primary key that causes multiple downstream applications to report errors; In response to receiving the batch processing instruction, calling a batch processing statement to batch process the source table to obtain a mart table corresponding to the source table; Extracting a second primary key from the mart table, where the second primary key is used to identify record data in the mart table; Abnormal data detection is performed on the second primary key, and when the second primary key does not include second abnormal data, the multiple downstream applications are allowed to read the mart table, where the second abnormal data represents data in the second primary key that causes the multiple downstream applications to report errors.

2. The data processing method according to claim 1, characterized in that: Before extracting the first primary key from the source table to be processed, the method further includes: Receive the initial source table sent by the data warehouse; Determine the access dependency of the multiple downstream applications on each sub-source table in the initial source table, extract the sub-source tables whose access dependency exceeds a preset threshold, and form the source table to be processed.

3. The data processing method according to claim 1, characterized in that: The extracting the first primary key from the source table comprises: Arrange the source table into dimension tables and fact tables; Extracting the first sub-primary key in the dimension table and the second sub-primary key in the fact table to obtain the first primary key; The first sub-primary key is used to identify the record data of the dimension table, the second sub-primary key is used to identify the record data of the fact table, and the first sub-primary key and the second sub-primary key are associated with each other.

4. The data processing method according to claim 3, characterized in that: The detecting abnormal data on the first primary key and generating a batch processing instruction when the first primary key does not include the first abnormal data comprises: Abnormal data detection is performed on the first sub-primary key and the second sub-primary key respectively, and the batch processing instruction is generated when neither the first sub-primary key nor the second sub-primary key includes the first abnormal data.

5. The data processing method according to claim 1, characterized in that: Also includes: In the case where the first primary key includes the first abnormal data, the first primary key is cached and a first warning message is generated, where the first warning message is used to display the detection result of the first primary key to a first user, so that the first user updates the data in the first primary key according to the warning message.

6. The data processing method according to claim 1, characterized in that: Also includes: In the case where the second primary key includes the second abnormal data, the second primary key is cached and a second warning message is generated, wherein the second warning message is used to display the detection result of the second primary key to the second user so that the second user updates the data in the second primary key according to the second warning message.

7. The data processing method according to claim 1, characterized in that: The first abnormal data includes duplicate primary key data; and the abnormal data detection on the first primary key, when the first primary key does not include the first abnormal data, generating a batch processing instruction includes: The first primary key is subjected to a duplicate checking process, and when the first primary key does not include duplicate primary key data, the batch processing instruction is generated.

8. The data processing method according to claim 1, characterized in that: The second abnormal data includes duplicate primary key data; the abnormal data detection on the second primary key, and allowing the multiple downstream applications to read the mart table when the second primary key does not include the second abnormal data, includes: The second primary key is checked for duplicates, and when the second primary key does not include duplicate data of the primary key, the multiple downstream applications are allowed to read the mart table.

9. A data processing device, characterized in that: include: A first extraction module, used to extract a first primary key from a source table to be processed, where the first primary key is used to identify record data of the source table; a first detection module, configured to perform abnormal data detection on the first primary key, and generate a batch processing instruction when the first primary key does not include first abnormal data, wherein the first abnormal data represents data in the first primary key that causes multiple downstream applications to report errors; A batch processing module, configured to, in response to receiving the batch processing instruction, call a batch processing statement to batch process the source table to obtain a mart table corresponding to the source table; A second extraction module, used for extracting a second primary key from the mart table, where the second primary key is used for identifying the record data of the mart table; The second detection module is used to perform abnormal data detection on the second primary key, and allow the multiple downstream applications to read the mart table when the second primary key does not include second abnormal data, where the second abnormal data represents data in the second primary key that causes the multiple downstream applications to report errors.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images