Data processing method and device

By processing and converting the metric and detailed data of multiple data sources, the problem of inconsistent data structures is solved, the unified processing of data is realized, and the efficiency of data processing and auditing is improved.

CN114490663BActive Publication Date: 2024-11-19ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210132717.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-14
Publication Date
2024-11-19
Estimated Expiration
2042-02-14

AI Technical Summary

Technical Problem

In the prior art, due to the different data sources of each data processing node, the data structure is not unified, making it difficult to quickly complete audit and data reporting processing, which affects efficiency.

Method used

By obtaining indicator data and detailed data from multiple data sources, processing them according to the storage structure of the indicator database and columnar database, converting them into a unified target data structure, and writing them to the corresponding database to achieve unified data processing .

Benefits of technology

It improves data processing efficiency, avoids the problem of multiple data sources affecting efficiency, facilitates data verification and review, and simplifies subsequent business audits and data reporting processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114490663B_ABST
    Figure CN114490663B_ABST
Patent Text Reader

Abstract

The present specification provides a data processing method and device, wherein the data processing method includes: obtaining indicator data and detail data corresponding to multiple data sources; processing the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data; processing the detail data of each data source according to the storage structure of the column database to obtain target detail data; writing the target indicator data into the indicator database, and writing the target detail data into the column database.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to the field of compliance data management technology, and in particular to data processing methods and devices. Background Art

[0002] With the development of Internet technology, more and more users choose online consumption. Different from traditional physical consumption scenarios, users can complete shopping, reservations, stock trading, and certificate processing through the Internet, which greatly improves the convenience of users. With the deepening of online services, more and more data are involved, and the data associated with different business scenarios is also different. When it is necessary to audit the business, the data needs to be aggregated and reported by each node under the business; in the prior art, due to the different data sources of each node, the reported data has different data structures, and the audit processing cannot be completed quickly. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide a data processing method. This specification also relates to a data processing device, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:

[0005] Obtain indicator data and detailed data corresponding to various data sources;

[0006] Process the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data;

[0007] Process the detailed data of each data source according to the storage structure of the column database to obtain the target detailed data;

[0008] The target indicator data is written into the indicator database, and the target detail data is written into the column database.

[0009] Optionally, the obtaining of indicator data and detail data corresponding to multiple data sources includes:

[0010] Receive data reporting requests submitted for target businesses;

[0011] Sending a data upload request to at least two service nodes associated with the target service according to the data reporting request;

[0012] Receiving service data returned by each service node in response to the data reporting request;

[0013] By classifying the business data, indicator data and detail data corresponding to various data sources are obtained.

[0014] Optionally, the processing of the indicator data of each data source according to the storage structure of the indicator database to obtain the target indicator data includes:

[0015] Determine a conversion rule between the storage structure of each data source and the storage structure of the indicator database;

[0016] The indicator data of each data source is processed according to the conversion rule to obtain the target indicator data having the same storage structure as the indicator database.

[0017] Optionally, the processing of the detailed data of each data source according to the storage structure of the column-based database to obtain target detailed data includes:

[0018] Synchronize the detailed data corresponding to each data source to the intermediate business database;

[0019] Convert the detailed data in the intermediate business database according to the structural conversion rules corresponding to each data source;

[0020] The target detailed data having the same storage structure as the column-based database is generated according to the conversion result.

[0021] Optionally, converting the detailed data in the intermediate business database according to the structural conversion rules corresponding to each data source includes:

[0022] Parsing the detailed data in the intermediate business database to obtain initial sub-data;

[0023] Target sub-data are screened out from the initial sub-data, and the target sub-data are converted according to the structural conversion rules corresponding to each data source.

[0024] Optionally, after the step of writing the target indicator data into the indicator database and the step of writing the target detail data into the column database is performed, the method further includes:

[0025] When the indicator database and the column database are in a stopped state, extracting the indicator data to be reported from the indicator database and extracting the detail data to be reported from the column database according to a preset reporting cycle;

[0026] Obtaining the business data to be reported by integrating the indicator data to be reported and the detail data to be reported;

[0027] The business data to be reported is sent to the reporting node.

[0028] Optionally, before the step of sending the to-be-reported business data to the reporting node is executed, the step further includes:

[0029] Obtaining the indicator detection strategy corresponding to the indicator data to be reported, and the detail detection strategy corresponding to the detail data to be reported;

[0030] Detect the indicator data to be reported according to the indicator detection strategy to obtain the indicator data detection result, and

[0031] Detect the detailed data to be reported according to the detailed detection strategy to obtain the detailed data detection result;

[0032] The indicator data detection result and the detail data detection result are sent to the business adjustment node.

[0033] Optionally, the detecting the indicator data to be reported according to the indicator detection strategy to obtain the indicator data detection result includes:

[0034] Read historical indicator data according to the indicator detection strategy;

[0035] The historical indicator column data is compared with the indicator data to be reported, and the indicator data detection result is determined according to the comparison result.

[0036] Optionally, the detecting the detailed data to be reported according to the detailed detection strategy to obtain the detailed data detection result includes:

[0037] Creating structure detection conditions and rule detection conditions according to the detailed detection strategy;

[0038] The detailed data to be reported is detected based on the structure detection condition and the rule detection condition to obtain the detailed data detection result.

[0039] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:

[0040] A data acquisition module is configured to acquire indicator data and detail data corresponding to various data sources;

[0041] A first processing module is configured to process the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data;

[0042] The second processing module is configured to process the detail data of each data source according to the storage structure of the column database to obtain target detail data;

[0043] The data writing module is configured to write the target indicator data into the indicator database, and write the target detail data into the column database.

[0044] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0045] Memory and processor;

[0046] The memory is used to store computer executable instructions, and the processor is used to implement the steps of the data processing method when executing the computer executable instructions.

[0047] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the data processing method are implemented.

[0048] According to a fifth aspect of the embodiments of this specification, a computer program is provided, which, when executed in a computer, causes the computer to execute the steps of the data processing method.

[0049] The data processing method provided in this specification can, after obtaining indicator data and detail data corresponding to multiple data sources, process the indicator data of each data source according to the storage results of the indicator database to obtain target indicator data with a unified data structure; at the same time, process the detail data of each data source according to the storage structure of the column database to obtain target indicator data with a unified data structure, and finally write the data into the corresponding database respectively, so as to unify the business-related data according to detail and indicator types, avoid the impact of multiple data sources on efficiency when conducting subsequent audits or data reporting on the business, and also facilitate rapid verification or review processing at nodes such as data verification or review, thereby further improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic diagram of a data processing method provided in an embodiment of this specification;

[0051] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;

[0052] Figure 3 It is a processing flow chart of a data processing method applied to a data reporting scenario provided in an embodiment of this specification;

[0053] Figure 4 It is a structural schematic diagram of a data processing device provided in an embodiment of this specification;

[0054] Figure 5 It is a structural block diagram of a computing device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0055] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0056] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0058] First, the terms involved in one or more embodiments of this specification are explained.

[0059] Data Source: The source of data is the device or original media that provides the required data. All the information for establishing a database connection is stored in the data source. Just like you can find a file in the file system by specifying the file name, you can find the corresponding database connection by providing the correct data source name.

[0060] Database: A warehouse that organizes, stores, and manages data according to data structures. It is a collection of large amounts of data that is stored in a computer for a long time, organized, shareable, and uniformly managed.

[0061] In this specification, a data processing method is provided. This specification also relates to a data processing device, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0062] In actual applications, different business scenarios correspond to different numbers of data processing nodes. As the number of data processing nodes increases, different data sources will be faced, that is, the databases storing data in each data processing node may be different, such as relational databases (MySQL, MariaDB) or non-relational databases (Cassandra, MongoDB), etc. When it is necessary to audit the business or report data, the problem of multiple data sources will be faced, which will greatly affect the efficiency of auditing or data reporting. Therefore, an effective solution is urgently needed to solve this problem.

[0063] See also Figure 1 The schematic diagram of the data processing method is as follows: after obtaining the indicator data and detail data corresponding to multiple data sources, the indicator data of each data source can be processed according to the storage results of the indicator database to obtain the target indicator data with a unified data structure; at the same time, the detail data of each data source is processed according to the storage structure of the column database to obtain the target indicator data with a unified data structure, and finally the data are written into the corresponding database respectively, thereby achieving the unification of business-related data according to detail and indicator types, avoiding the impact of multiple data sources on efficiency during subsequent business audits or data reporting, and also facilitating rapid verification or review processing at data verification or review nodes, further improving data processing efficiency.

[0064] Figure 2 A flowchart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0065] Step S202, obtaining indicator data and detail data corresponding to multiple data sources.

[0066] Specifically, multiple data sources specifically refer to the databases corresponding to each data processing node included in the same business project; correspondingly, indicator data specifically refers to the data related to quantity information in the business project, and detail data specifically refers to the data related to business information in the business project.

[0067] For example, a business project is a payment service provided by a payment platform. Sellers can list goods on the payment platform, and buyers can purchase goods provided by sellers through the payment platform. Based on this, in order to support the payment platform to provide payment services to buyers and sellers, the business side of the payment platform will arrange different nodes to manage them. For example, the node that manages goods is responsible for the data related to goods, the node that manages orders is responsible for the data related to orders, the node that manages shipments is responsible for the data related to logistics, the node that manages the information of buyers or sellers is responsible for the data related to personal information, etc. Different nodes will manage different types of data to support the normal development of business projects.

[0068] In order to facilitate data storage and backup, each node usually selects a database that matches the data processed by the current node to store this data. When data reporting or auditing of business projects is required, each node will directly extract data from the database for reporting. At this time, data related to business projects corresponding to multiple data sources will be obtained, and this data will be divided into indicator and detail categories to facilitate subsequent data unification and improve audit or reporting efficiency.

[0069] If the nodes related to the business project correspond to data sources DS1~DSn respectively (where each DS corresponds to a database, and the types of each database are different), the data corresponding to the DS1~DSn data sources will be obtained at this time. Then, this part of the data can be divided according to indicator categories and detail categories, so as to obtain indicator category data and detail category data corresponding to multiple data sources.

[0070] It should be noted that any business project may correspond to multiple data sources. When it is necessary to audit the business project or submit data, you can refer to the relevant description content of this specification, and this embodiment will not be described in detail here.

[0071] Based on this, whether in an audit or in a data reporting scenario, when there is a need for statistical requirements for data related to a business project, there will be the problem of inconsistent data structures due to different data sources, which greatly reduces the statistical efficiency. The data processing method provided in this embodiment performs an initial classification of data from different data sources and then unifies them, thereby effectively improving the data statistical efficiency and thus improving the processing efficiency of downstream businesses.

[0072] Furthermore, in the data reporting scenario, when it is necessary to collect statistics on the data involved in the project, in order to improve the data processing efficiency, each node can be requested to actively send data to obtain data corresponding to multiple data sources. In this embodiment, the specific implementation method is as follows:

[0073] Receive data reporting requests submitted for target businesses;

[0074] Sending a data upload request to at least two service nodes associated with the target service according to the data reporting request;

[0075] Receiving service data returned by each service node in response to the data reporting request;

[0076] By classifying the business data, indicator data and detail data corresponding to various data sources are obtained.

[0077] Specifically, the target business specifically refers to the business project that currently needs to report data; correspondingly, the business node specifically refers to the data processing node that supports the target business to provide services to users, and each business node corresponds to a data source for storing data related to the node; business data specifically refers to all data collected or generated during the operation of the target business.

[0078] Based on this, when the management platform receives a data reporting request for the target business, it means that the management platform needs to perform risk prediction or data statistics on the target business. The data reporting request can include a data statistics cycle to facilitate the data upload of each business node of the target business, which can be completed according to the data statistics cycle; at this time, the business platform can send a data upload request to each business node associated with the target business according to the data reporting request, and at the same time receive the business data returned by each business node in response to the data upload request.

[0079] Since each business node corresponds to at least one data source, the returned business data will correspond to multiple data sources. In order to facilitate subsequent statistics, the business data can be classified according to indicator categories and detail categories at this time, so as to obtain indicator category data corresponding to multiple data sources and detail category data corresponding to multiple data sources, so as to improve the subsequent data processing efficiency.

[0080] Continuing with the above example, receive the customer quantity data of the corresponding data source DS1 returned by the customer data management node; the transaction amount data of the corresponding data source DS2 returned by the transaction data management node; the active user quantity data of the corresponding DS3 returned by the user data management node; the order data of the corresponding DS4 returned by the order data management node; the merchant data of the corresponding data source DS5 returned by the merchant data management node; and the address data of the corresponding data source DS6 returned by the logistics data management node. In order to improve the subsequent reporting efficiency, the above data can be classified according to detail categories and indicator categories to obtain the indicator category data {transaction amount data, customer quantity data, active user quantity data} of the corresponding data sources DS1, DS2 and DS3; and obtain the detail category data {order data, merchant data, address data} of the corresponding data sources DS4, DS5 and DS6.

[0081] In summary, by separating the business data returned by each business node, we can obtain detailed data and indicator data corresponding to multiple data sources, and achieve preliminary data statistics from two dimensions through preprocessing, which facilitates subsequent data unification based on this and effectively improves data processing efficiency.

[0082] Step S204: Process the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data.

[0083] Specifically, after obtaining the indicator data corresponding to multiple data sources as mentioned above, since different data sources correspond to different data structures, if the indicator data corresponding to multiple data sources are directly stored, incompatibility issues may arise, and even data conflicts may lead to data loss. Therefore, in order to facilitate the use of downstream businesses, the indicator data corresponding to different data sources can be uniformly processed according to the storage structure of the indicator database according to current statistical needs to obtain target indicator data with the same data structure.

[0084] Among them, the indicator database specifically refers to a database used to store indicator data, and the database has only one storage structure, so that the data stored in the database will be unified according to the storage structure to facilitate management and persistence; correspondingly, the target indicator data is the indicator data expressed in the same storage structure as the indicator database.

[0085] Furthermore, before writing the indicator data corresponding to multiple data sources into the indicator database, the indicator data of each data source needs to be converted into a data structure that is the same as the storage structure of the indicator database to make the data persistent. The data corresponding to different data sources needs to be implemented according to different rules when performing data conversion. In this embodiment, the specific implementation method is as follows:

[0086] Determine a conversion rule between the storage structure of each data source and the storage structure of the indicator database;

[0087] The indicator data of each data source is processed according to the conversion rule to obtain the target indicator data having the same storage structure as the indicator database.

[0088] Specifically, the conversion rules refer to the conversion method between the storage structure of each data source and the storage structure of the indicator database. Through the conversion rules, the data structure of the corresponding data source can be converted into the same expression as the storage structure of the indicator database, which is convenient for the indicator database to store the converted indicator data.

[0089] Based on this, after obtaining the indicator data corresponding to each data source, the conversion rules between the storage structure of each data source and the storage structure of the indicator database can be read, and then the corresponding indicator data can be processed according to the conversion rules. The indicator data corresponding to the conversion rules can be converted into target indicator data with the same storage structure as the indicator database. Similarly, the indicator data of each data source are processed according to the corresponding conversion rules. When the indicator data corresponding to all data sources are converted into target indicator data, they can be stored in the indicator database.

[0090] Continuing with the above example, after obtaining the indicator data {transaction amount data, customer quantity data, active user quantity data} corresponding to data sources DS1, DS2 and DS3, determine the storage structure of the indicator database, and read the conversion rule P1 between the storage structure of the data source DS1 and the storage structure of the indicator database, the conversion rule P2 between the storage structure of the data source DS2 and the storage structure of the indicator database, and the conversion rule P3 between the storage structure of the data source DS3 and the storage structure of the indicator database; then, convert the customer quantity data according to the conversion rule P1 to obtain the target customer quantity data that conforms to the storage structure of the indicator database; convert the transaction amount data according to the conversion rule P2 to obtain the target transaction amount data that conforms to the storage structure of the indicator database; convert the active user quantity data according to the conversion rule P3 to obtain the target active user quantity data that conforms to the storage structure of the indicator database; wherein, the target transaction amount data, the target customer quantity data and the target active user quantity data all have the same data structure to facilitate writing into the indicator database and subsequent data reporting and processing.

[0091] In summary, by presetting the conversion rules between the storage structure of different data sources and the storage structure of the indicator database, it is possible to unify the data structure of the indicator data corresponding to different data sources and obtain the target indicator data to facilitate downstream business use.

[0092] Step S206: Process the detail data of each data source according to the storage structure of the column-based database to obtain target detail data.

[0093] Specifically, after obtaining the detailed data corresponding to multiple data sources as mentioned above, since different data sources correspond to different data structures, if the detailed data corresponding to multiple data sources are directly stored, incompatibility problems may occur. At the same time, since the detailed data has more dimensions and the business information involved is also richer, in order to facilitate the use of downstream businesses, it is necessary to unify the detailed data corresponding to multiple data sources, and at the same time extract fields or field values ​​that are convenient for downstream businesses to use for storage, so as to obtain the target detailed data.

[0094] Among them, the column-based database specifically refers to a database used to store detailed data, and the database has only one storage structure, so that the data stored in the database will be unified according to the storage structure to facilitate management and persistence; accordingly, the target detailed data is the detailed data expressed in the same storage structure as the column-based database.

[0095] Furthermore, when storing detailed data corresponding to various data sources in a column-based database, in order to improve data storage efficiency and facilitate downstream business use, the detailed data may be processed according to a preset data processing link. In this embodiment, the specific implementation is as follows:

[0096] Step S2062: Synchronize the detailed data corresponding to each data source to the intermediate business database.

[0097] Specifically, the intermediate business database refers to a unified data warehouse used to summarize the detailed data corresponding to each data source, and the detailed data stored in the intermediate business database will retain the original data structure, that is, the intermediate business database can be understood as a data warehouse that temporarily stores the detailed data that needs to be stored in the column database.

[0098] Based on this, since the detailed data is more complex and contains more information related to the target business, if it is directly converted or processed, it may cause additional processing burden. Therefore, in order to improve the subsequent storage efficiency, the detailed data corresponding to each data source can be synchronized to the intermediate business database for temporary storage to facilitate subsequent processing. The intermediate business database will not persist the data. After the data is processed, the storage space can be cleared to reduce resource consumption.

[0099] Step S2064: convert the detailed data in the intermediate business database according to the structural conversion rules corresponding to each data source.

[0100] Specifically, the structure conversion rule refers to the rule for converting the detail data corresponding to each data source into the storage structure of a columnar database, thereby realizing the storage structure of the corresponding columnar database and facilitating the subsequent storage and persistence of the target detail data.

[0101] Based on this, since the detail data stored in the intermediate database corresponds to multiple data sources, and the column database corresponding to each data source may contain more business-related information, and the data processed and used by the downstream business may not necessarily cover all the detail data, when converting the detail data corresponding to each data source, it can be achieved by parsing and screening. In this embodiment, the specific implementation method is as follows:

[0102] Parsing the detailed data in the intermediate business database to obtain initial sub-data;

[0103] Target sub-data are screened out from the initial sub-data, and the target sub-data are converted according to the structural conversion rules corresponding to each data source.

[0104] Specifically, the initial sub-data specifically refers to all the sub-business data that make up the detail data, and correspondingly, the target sub-data specifically refers to the sub-business data required to be converted into the target detail data. Based on this, the detail data corresponding to each data source is written into the intermediate business database according to the data processing link. When all the writing is completed, the detail data can be extracted from the intermediate business database for parsing to obtain the initial sub-business data corresponding to each detail data. After that, the target sub-data is filtered out from the initial sub-business data, and then the target sub-data is converted according to the structural conversion rules corresponding to each data source, so as to obtain the target detail data with the same storage structure as the column database.

[0105] In summary, by filtering the initial sub-business data to obtain the target sub-data that can be converted, other useless data can be filtered out, and the data can be unified to meet the storage requirements of the columnar database, improve the efficiency of subsequent sorting out of the target detail data, and facilitate downstream business use.

[0106] Step S2066: Generate the target detailed data having the same storage structure as the column-based database according to the conversion result.

[0107] Specifically, after the target sub-data are screened out and converted according to the structure conversion rules, target detailed data having the same storage structure as the column-based database can be generated according to the conversion results.

[0108] Continuing with the above example, after obtaining the detailed data {order data, merchant data, address data} corresponding to data sources DS4, DS5, and DS6, the detailed data corresponding to each data source can be stored in the unified data warehouse according to the data processing chain {original data source-> unified data warehouse-> row to column-> unified data source}. When all the data are stored, the detailed data corresponding to each data source can be extracted from the unified data warehouse; the storage structure of the order data corresponding to data source DS4 is {order number, order establishment time, order transaction time, order end time}, and the storage structure of the merchant data corresponding to data source DS5 is {ID , merchant name, merchant address, merchant type}; the storage structure of the address data corresponding to the data source DS6 is {ID, merchant address, shipping address, receiving address}. At this time, the detailed data corresponding to each data source can be converted according to the structural conversion rules between the storage structure of each data source and the storage structure of the column-based database. According to the conversion result, the target detailed data of the corresponding column-based database storage structure {subject, ID, field name, field value} is obtained. By storing the field value and field name, each detailed data can be recorded for subsequent use. The structure of the target detailed data is shown in the following table (1):

[0109] Table (1)

[0110] main body ID Field Name Field Value Merchants 123 Type1 S1 Merchants 123 Type2 S2 Merchants 145 Type1 S3 user 965 Type9 S4 … … … …

[0111] Among them, different IDs represent different merchants or users, and different field names represent different data contents, such as Type 1 corresponds to the merchant address, and the corresponding field value is used to indicate the specific address location, or Type 2 corresponds to the merchant type, and the corresponding field value is used to indicate the specific merchant type. In actual applications, the field name and field value are set according to actual needs, and this embodiment does not make any limitation here.

[0112] In summary, by converting detailed data from multiple data sources into the storage structure of a columnar database, data unification is achieved, which facilitates subsequent management and persistence, and is more convenient for downstream businesses to use, effectively improving data processing efficiency.

[0113] In addition, there is no execution order for step S204 and step S206. Step S204 may be executed first and then step S206, or step S206 may be executed first and then step S204, or step S2041 and step S206 may be executed simultaneously. This embodiment does not make any limitation here.

[0114] Step S208, writing the target indicator data into the indicator database, and writing the target detail data into the column database.

[0115] Specifically, after completing the data unification as mentioned above, target indicator data and target detail data will be obtained. At this time, the target indicator data can be written into the indicator database, and the target detail data can be written into the column database, so that downstream businesses can directly read the data in each database when using it, without the need for unified processing, thereby improving the data reuse rate.

[0116] Furthermore, after the data is written into the database, in order to quickly complete the reporting process, the data reporting process operation can be triggered by detecting the running status of the database. In this embodiment, the specific implementation method is as follows:

[0117] When the indicator database and the column database are in a stopped state, extracting the indicator data to be reported from the indicator database and extracting the detail data to be reported from the column database according to a preset reporting cycle;

[0118] Obtaining the business data to be reported by integrating the indicator data to be reported and the detail data to be reported;

[0119] The business data to be reported is sent to the reporting node.

[0120] Specifically, the reporting cycle refers to the period for sending data to the reporting node, which can be 24 hours, 1 week or one month, and can be set according to actual needs in actual applications; correspondingly, the indicator data to be reported specifically refers to the indicator data in the indicator database that is within the reporting cycle; correspondingly, the detail data to be reported specifically refers to the detail data in the column database that is within the reporting cycle, and the indicator data to be reported and the detail data to be reported are the business data to be reported that need to be sent to the reporting node for reporting and processing.

[0121] Based on this, after the target indicator data and target detail data are continuously written into the indicator database and the column database, it is possible to detect whether the reporting operation is triggered by detecting the running status of the database; when the running status of the indicator database and the column database is stopped, it means that data reporting processing can be performed at this time, then the indicator data to be reported can be extracted from the indicator database according to the preset reporting cycle, and the detail data to be reported can be extracted from the column database at the same time, and then the two can be integrated to obtain the business data to be reported, and finally the business data to be reported can be sent to the reporting node.

[0122] For example, when it is necessary to send business-related data from January to February 2021 to the reporting node, the indicator data to be reported corresponding to the time interval can be extracted from the indicator database according to the reporting cycle, and the detailed data to be reported corresponding to the time interval can be extracted from the column database. These two parts of data are then integrated and sent to the reporting node for processing.

[0123] In actual applications, the reporting node may be a data management node within the business party, or a data management organization, or a data management department of the parent company of the business party, and this embodiment does not impose any limitation thereto.

[0124] In summary, by triggering the reporting operation by detecting the operating status of the database, the resource consumption of setting up the reporting node can be saved, and at the same time, the unified data within the reporting cycle can be reported, which effectively improves the reporting efficiency and the processing efficiency of the reporting node.

[0125] In addition, before sending the data to the reporting node, in order to avoid unclear problems in the reported data, such as missing key values ​​or data loss, and to improve the compliance within the business project, the data to be reported can be tested. In this embodiment, the specific implementation method is as follows:

[0126] Step S2082, obtaining the indicator detection strategy corresponding to the indicator data to be reported, and the detail detection strategy corresponding to the detail data to be reported.

[0127] Specifically, the indicator detection strategy refers to a strategy that can detect the indicator data to be reported and check whether this part of the data is compliant; the corresponding detail detection strategy refers to a strategy that can detect the detail column data to be reported and check whether this part of the data is compliant.

[0128] In actual applications, the indicator detection strategy specifically refers to detecting whether the changes and fluctuations of the indicator data to be reported are compliant compared with the historical indicator data; the detailed indicator strategy specifically refers to detecting whether the detailed data to be reported itself is compliant, and / or whether the business nodes are compliant; among them, detecting whether the data is compliant specifically refers to detecting whether the data meets the requirements of the current business scenario.

[0129] Step S2084, detecting the indicator data to be reported according to the indicator detection strategy to obtain the indicator data detection result, and detecting the detail data to be reported according to the detail detection strategy to obtain the detail data detection result.

[0130] Specifically, after obtaining the indicator detection strategy corresponding to the indicator data to be reported, and the detailed detection strategy corresponding to the detailed data to be reported, in order to ensure the compliance of the business scenario and ensure that the business scenario is more standardized, each part of the data can be tested separately according to the detection strategy, so that the compliance of the business scenario can be reflected through the detection results.

[0131] Furthermore, in the process of testing the reported indicator data according to the indicator detection strategy, in order to improve the detection accuracy, the comparison can be achieved by reading the historical indicator data. In this embodiment, the specific implementation method is as follows:

[0132] Read historical indicator data according to the indicator detection strategy;

[0133] The historical indicator column data is compared with the indicator data to be reported, and the indicator data detection result is determined according to the comparison result.

[0134] Specifically, historical indicator data refers to the indicator data corresponding to each reporting period in the previous one or more reporting periods, which is used to compare the indicator data to be reported; correspondingly, the indicator data detection results refer to the detection results obtained after detecting the indicator data to be reported, so as to analyze whether the indicator data is compliant.

[0135] Based on this, after obtaining the indicator detection strategy, the historical indicator data can be read according to the indicator detection strategy to detect the indicator data to be reported. Furthermore, the historical indicator data can be compared with the indicator data to be reported, and the indicator data detection results corresponding to the indicator data to be reported can be determined by comparison, so as to analyze whether the indicator data to be reported has risks that will affect the business project, so as to respond in time and avoid greater losses.

[0136] In summary, by analyzing the indicator data to be reported by comparing it with historical indicator data, the detection can be completed in combination with the characteristics of the indicator data to be reported, which can not only improve the detection accuracy but also improve the compliance of business scenarios.

[0137] Furthermore, in the process of detecting the detailed data to be reported according to the detailed detection strategy, in order to improve the detection accuracy, it can be achieved by means of detection structure and rules. In this embodiment, the specific implementation method is as follows:

[0138] Creating structure detection conditions and rule detection conditions according to the detailed detection strategy;

[0139] The detailed data to be reported is detected based on the structure detection condition and the rule detection condition to obtain the detailed data detection result.

[0140] Specifically, the structural detection conditions refer to the conditions that can detect the structural dimensions of the detailed data to be reported, that is, to detect whether the detailed data to be reported itself is compliant; correspondingly, the rule detection conditions refer to the conditions that can detect the business dimensions of the detailed data to be reported, that is, to detect whether the business nodes that generate or transmit the detailed data to be reported are compliant.

[0141] Based on this, after obtaining the detail detection strategy, you can create structure detection conditions and rule detection conditions according to the strategy, and then perform structural dimension detection on the detail data to be reported based on the structure detection conditions, and at the same time perform business dimension detection on the detail data to be reported based on the rule detection conditions, so as to obtain the detail data detection results based on the detection results.

[0142] In other words, when conducting risk prediction for the target business, it can be completed at the same time as data reporting, thereby saving data statistics time, reducing resource consumption, achieving multiple purposes in one operation, effectively ensuring compliance of business scenarios, and improving reporting efficiency.

[0143] In summary, by using structural testing and business testing to test the detailed data to be reported, we can not only avoid the impact of the data itself, but also determine the impact at the business level, thereby further improving the stability of business scenarios and improving the compliance of business scenarios.

[0144] Step S2086: Send the indicator data detection result and the detail data detection result to the business adjustment node.

[0145] Specifically, after obtaining the indicator data detection results and the detailed data detection structure as mentioned above, the detection structure can be sent to the adjustment node so that the adjustment node can adjust the business according to the detection results, or update the business, or correct the data to ensure that the data subsequently sent to the reporting node are true and accurate.

[0146] For example, after obtaining the indicator data to be reported from February to March 2021, the historical indicator data from January to February 2021 can be read, and then the two parts of data can be compared to analyze the fluctuations of each indicator. If the volatility is greater than the threshold, it means that the indicator data has changed significantly and there may be business risks. In this case, the indicator data detection structure is determined to have business risks. If the volatility is less than or equal to the threshold, it means that the indicator data has changed little and the probability of business risks is low. In this case, the indicator data detection result is determined to have no business risks.

[0147] At the same time, create structure detection conditions and rule detection conditions, and based on these two conditions, detect the detailed data to be reported, that is, detect whether each detailed data to be reported has missing field values, field names, IDs or subjects, and detect whether each detailed data to be reported violates the reporting specifications or compliance clauses; if there is a missing data structure, it means that there is a problem with the data itself, and the result of the detailed data detection is determined to be a problem with data transmission; if there is a violation of the data, it means that the node that processes the data has a violation, and the result of the detailed data detection is determined to be a problem with the business.

[0148] Finally, the indicator data detection results and the detail data detection results are integrated and sent to the adjustment node. The business items are updated or adjusted through the adjustment node to standardize the business scenarios and ensure compliance.

[0149] The data processing method provided in this specification can, after obtaining indicator data and detail data corresponding to multiple data sources, process the indicator data of each data source according to the storage results of the indicator database to obtain target indicator data with a unified data structure; at the same time, process the detail data of each data source according to the storage structure of the column database to obtain target indicator data with a unified data structure, and finally write the data into the corresponding database respectively, so as to unify the business-related data according to detail and indicator types, avoid the impact of multiple data sources on efficiency when conducting subsequent audits or data reporting on the business, and also facilitate rapid verification or review processing at nodes such as data verification or review, thereby further improving data processing efficiency.

[0150] The following combination Figure 3 , taking the application of the data processing method provided in this specification in the data reporting scenario as an example, the data processing method is further explained. Figure 3 A processing flow chart of a data processing method applied to a data reporting scenario provided in an embodiment of this specification is shown, which specifically includes the following steps:

[0151] Step S302, obtaining indicator data and detail data corresponding to multiple data sources.

[0152] Step S304, determining the conversion rules between the storage structure of each data source and the storage structure of the indicator database.

[0153] Step S306, processing the indicator data of each data source according to the conversion rule to obtain target indicator data having the same storage structure as the indicator database.

[0154] Step S308: Synchronize the detailed data corresponding to each data source to the intermediate business database.

[0155] Step S310: convert the detailed data in the intermediate business database according to the structural conversion rules corresponding to each data source.

[0156] Step S312: Generate target detailed data having the same storage structure as the column-based database according to the conversion result.

[0157] Step S314, write the target indicator data into the indicator database and write the target detail data into the column database.

[0158] Step S316, before data submission, extract the indicator data to be submitted from the indicator database, and extract the detail data to be submitted from the column database.

[0159] Step S318, obtaining the indicator detection strategy corresponding to the indicator data to be reported, and the detail detection strategy corresponding to the detail data to be reported.

[0160] Step S320: Detect the indicator data to be reported according to the indicator detection strategy to obtain the indicator data detection result.

[0161] Step S322, detect the detailed data to be reported according to the detailed detection strategy to obtain the detailed data detection result.

[0162] Step S324, sending the indicator data detection result and the detail data detection result to the business adjustment node.

[0163] Step S326: After the adjustment node returns the adjustment result, the data in the indicator database and the column database are sent to the reporting node.

[0164] The data processing method provided in this specification can, after obtaining indicator data and detail data corresponding to multiple data sources, process the indicator data of each data source according to the storage results of the indicator database to obtain target indicator data with a unified data structure; at the same time, process the detail data of each data source according to the storage structure of the column database to obtain target indicator data with a unified data structure, and finally write the data into the corresponding database respectively, so as to unify the business-related data according to detail and indicator types, avoid the impact of multiple data sources on efficiency when conducting subsequent audits or data reporting on the business, and also facilitate rapid verification or review processing at nodes such as data verification or review, thereby further improving data processing efficiency.

[0165] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0166] The data acquisition module 402 is configured to acquire indicator data and detail data corresponding to various data sources;

[0167] The first processing module 404 is configured to process the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data;

[0168] The second processing module 406 is configured to process the detail data of each data source according to the storage structure of the column database to obtain target detail data;

[0169] The data writing module 408 is configured to write the target indicator data into the indicator database, and write the target detail data into the column database.

[0170] In an optional embodiment, the data acquisition module 402 is further configured to:

[0171] Receive a data reporting request submitted for a target business; send a data upload request to at least two business nodes associated with the target business according to the data reporting request; receive business data returned by each business node in response to the data reporting request; and obtain indicator data and detail data corresponding to multiple data sources by classifying the business data.

[0172] In an optional embodiment, the first processing module 404 is further configured to:

[0173] Determine the conversion rules between the storage structure of each data source and the storage structure of the indicator database; process the indicator data of each data source according to the conversion rules to obtain the target indicator data with the same storage structure as the indicator database.

[0174] In an optional embodiment, the second processing module 406 is further configured to:

[0175] The detailed data corresponding to each data source is synchronized to the intermediate business database; the detailed data in the intermediate business database is converted according to the structural conversion rules corresponding to each data source; and the target detailed data having the same storage structure as the column database is generated according to the conversion result.

[0176] In an optional embodiment, the second processing module 406 is further configured to:

[0177] The detailed data in the intermediate business database is parsed to obtain initial sub-data; target sub-data is screened out from the initial sub-data, and the target sub-data is converted according to the structural conversion rules corresponding to each data source.

[0178] In an optional embodiment, the data processing device further includes:

[0179] The reporting module is configured to extract the indicator data to be reported from the indicator database and the detail data to be reported from the column database according to a preset reporting cycle when the operating status of the indicator database and the column database is stopped; obtain the business data to be reported by integrating the indicator data to be reported and the detail data to be reported; and send the business data to be reported to the reporting node.

[0180] In an optional embodiment, the data processing device further includes:

[0181] The adjustment module is configured to obtain the indicator detection strategy corresponding to the indicator type data to be reported, and the detailed detection strategy corresponding to the detailed type data to be reported; detect the indicator type data to be reported according to the indicator detection strategy to obtain the indicator type data detection result, and detect the detailed type data to be reported according to the detailed detection strategy to obtain the detailed type data detection result; send the indicator type data detection result and the detailed type data detection result to the business adjustment node.

[0182] In an optional embodiment, the adjustment module is further configured to:

[0183] The historical indicator data is read according to the indicator detection strategy; the historical indicator column data is compared with the indicator data to be reported, and the indicator data detection result is determined according to the comparison result.

[0184] In an optional embodiment, the adjustment module is further configured to:

[0185] According to the detailed detection strategy, a structure detection condition and a rule detection condition are created; based on the structure detection condition and the rule detection condition, the detailed data to be reported is detected to obtain the detailed data detection result.

[0186] The data processing device provided in this specification can, after acquiring indicator data and detail data corresponding to multiple data sources, process the indicator data of each data source according to the storage results of the indicator database to obtain target indicator data with a unified data structure; at the same time, process the detail data of each data source according to the storage structure of the column database to obtain target indicator data with a unified data structure, and finally write the data into the corresponding database respectively, thereby achieving unification of business-related data according to detail and indicator types, avoiding the impact of multiple data sources on efficiency during subsequent business audits or data reporting, and also facilitating rapid verification or review processing at data verification or review nodes, further improving data processing efficiency.

[0187] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0188] Figure 5The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0189] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0190] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0191] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.

[0192] The processor 520 is used to execute the following computer executable instructions:

[0193] Obtain indicator data and detailed data corresponding to various data sources;

[0194] Process the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data;

[0195] Process the detailed data of each data source according to the storage structure of the column database to obtain the target detailed data;

[0196] The target indicator data is written into the indicator database, and the target detail data is written into the column database.

[0197] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0198] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which are used when executed by a processor to:

[0199] Obtain indicator data and detailed data corresponding to various data sources;

[0200] Process the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data;

[0201] Process the detailed data of each data source according to the storage structure of the column database to obtain the target detailed data;

[0202] The target indicator data is written into the indicator database, and the target detail data is written into the column database.

[0203] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0204] An embodiment of the present specification further provides a computer program, which, when executed in a computer, enables the computer to execute the steps of the data processing method.

[0205] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the computer program and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the computer program that are not described in detail can be found in the description of the technical scheme of the above data processing method.

[0206] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0207] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0208] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0209] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0210] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Obtain indicator data and detailed data corresponding to various data sources; Process the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data; Synchronize the detail data corresponding to each data source to the intermediate business database, parse the detail data in the intermediate business database to obtain initial sub-data, filter out target sub-data from the initial sub-data, and transform the target sub-data according to the structural transformation rule corresponding to each data source, and generate target detail data with the same storage structure as the column database according to the transformation result, wherein the detail data stored in the intermediate business database retains the original data structure; The target indicator data is written into the indicator database, and the target detail data is written into the column database.

2. According to the data processing method of claim 1, the step of obtaining indicator data and detail data corresponding to multiple data sources comprises: Receive data reporting requests submitted for target businesses; Sending a data upload request to at least two service nodes associated with the target service according to the data reporting request; Receiving service data returned by each service node in response to the data reporting request; By classifying the business data, indicator data and detail data corresponding to various data sources are obtained.

3. The data processing method according to claim 1, wherein the step of processing the indicator data of each data source according to the storage structure of the indicator database to obtain the target indicator data comprises: Determine a conversion rule between the storage structure of each data source and the storage structure of the indicator database; The indicator data of each data source is processed according to the conversion rule to obtain the target indicator data having the same storage structure as the indicator database.

4. The data processing method according to any one of claims 1 to 3, after the step of writing the target indicator data into the indicator database and the step of writing the target detail data into the column database is executed, further comprising: When the indicator database and the column database are in a stopped state, extracting the indicator data to be reported from the indicator database and extracting the detail data to be reported from the column database according to a preset reporting cycle; Obtaining the business data to be reported by integrating the indicator data to be reported and the detail data to be reported; The business data to be reported is sent to the reporting node.

5. The data processing method according to claim 4, before the step of sending the business data to be reported to the reporting node is executed, it also includes: Obtaining the indicator detection strategy corresponding to the indicator data to be reported, and the detail detection strategy corresponding to the detail data to be reported; Detect the indicator data to be reported according to the indicator detection strategy to obtain the indicator data detection result, and Detect the detailed data to be reported according to the detailed detection strategy to obtain the detailed data detection result; The indicator data detection result and the detail data detection result are sent to the business adjustment node.

6. The data processing method according to claim 5, wherein the step of detecting the indicator data to be reported according to the indicator detection strategy to obtain the indicator data detection result comprises: Read historical indicator data according to the indicator detection strategy; The historical indicator data is compared with the indicator data to be reported, and the indicator data detection result is determined according to the comparison result.

7. The data processing method according to claim 5, wherein the step of detecting the detailed data to be reported according to the detailed detection strategy to obtain the detailed data detection result comprises: Creating structure detection conditions and rule detection conditions according to the detailed detection strategy; The detailed data to be reported is detected based on the structure detection condition and the rule detection condition to obtain the detailed data detection result.

8. A data processing device, comprising: A data acquisition module is configured to acquire indicator data and detail data corresponding to various data sources; A first processing module is configured to process the indicator data of each data source according to the storage structure of the indicator database to obtain target indicator data; The second processing module is configured to synchronize the detail data corresponding to each data source to the intermediate business database, parse the detail data in the intermediate business database to obtain initial sub-data, filter out target sub-data from the initial sub-data, and convert the target sub-data according to the structural conversion rule corresponding to each data source, and generate target detail data with the same storage structure as the column database according to the conversion result, wherein the detail data stored in the intermediate business database retains the original data structure; The data writing module is configured to write the target indicator data into the indicator database, and write the target detail data into the column database.

9. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.