Data reporting method, apparatus and device

Abstract

The application provides a data reporting method, device and equipment, comprising: calculating credit investigation business data according to a credit investigation reporting rule through a data side to obtain a result table, the result table comprising data completely corresponding to each data item of a credit investigation report; and assembling the result table according to a format requirement of the credit investigation report through a platform side to obtain the credit investigation report. Thus, efficient and easy-to-maintain credit investigation data reporting is achieved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data reporting method, apparatus, and device. Background Technology

[0002] Currently, microfinance companies, banks, and other financial institutions are required to submit credit information reports (or report files) to the central bank's credit reporting system. These reports typically include details such as loan amount, interest rate, repayment date, and customer name. With the implementation of the Credit Reporting 2.0 standard, requirements have been placed on the timeliness, accuracy, and frequency of credit information report data submission. Therefore, achieving efficient and easily maintainable credit data submission is the technical problem this application aims to solve. Summary of the Invention

[0003] This application provides a data reporting method, apparatus, and equipment that can achieve efficient and easy-to-maintain credit data reporting.

[0004] Firstly, this application provides a data reporting method, comprising: on the data side, calculating credit reporting business data according to credit reporting rules to obtain a result table, the result table including data that completely corresponds to each data item of the credit report; and on the platform side, assembling the result table according to the format requirements of the credit report to obtain the credit report.

[0005] Secondly, this application provides a data reporting device, comprising: a processing module, used to calculate credit reporting business data according to credit reporting rules on the data side to obtain a result table, the result table including data that completely corresponds to each data item of the credit reporting message; and an output module, used to assemble the result table according to the format requirements of the credit reporting message on the platform side to obtain the credit reporting message.

[0006] Thirdly, this application provides an electronic device, including: a processor and a memory, the memory for storing a computer program, and the processor for calling and running the computer program stored in the memory to perform the methods as described in the first aspect or its various implementations.

[0007] Fourthly, this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0008] Fifthly, this application provides a computer program product including computer program instructions that cause a computer to perform the methods as described in the first aspect or its various implementations.

[0009] Sixthly, this application provides a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0010] The technical solution of this application allows for the calculation of credit reporting data on the data side according to credit reporting rules, resulting in a result table that includes data completely corresponding to each data item in the credit report. On the platform side, the result table is assembled according to the format requirements of the credit report to obtain the credit report. In this process, all complex business logic calculations are completed in one go on the data side. The data side can directly generate data completely corresponding to each data item in the credit report. The platform side is only responsible for reading the result table and format conversion, without participating in any credit reporting-related calculations. This makes the data processing flow transparent and controllable, and optimizes the allocation of computing resources. Furthermore, when the reporting criteria are adjusted or data anomalies occur, only the corresponding calculation logic needs to be modified on the data side, such as processing scripts and re-executing the calculations. The platform side can generate credit reports that meet the new requirements without any modifications, improving system maintenance efficiency and problem response speed. Therefore, efficient and easily maintainable credit data reporting can be achieved.

[0011] Other technical features and effects involved in the technical solution of this application will be described in subsequent embodiments, and will not be repeated here to avoid repetition. Attached Figure Description

[0012] The accompanying drawings used in the following description of the embodiments are introduced.

[0013] Figure 1 A flowchart illustrating a data reporting method provided in this application embodiment; Figure 2 A schematic diagram illustrating a data reporting method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating another data reporting method provided in an embodiment of this application; Figure 4 A schematic diagram illustrating another data reporting method provided in an embodiment of this application; Figure 5 A schematic diagram of the data reporting device provided in the embodiments of this application; Figure 6 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0016] In one embodiment, the technical solution of this application can be used in data reporting scenarios, for example, it can be applied to the processing stage of financial institutions reporting credit data to the central bank's credit reporting system, but is not limited thereto.

[0017] For example, during the daily batch processing window in the early morning, the big data platform on the data side extracts incremental business data, namely credit reporting data, generated the previous day from various business systems (such as the core loan system, customer information system, and transaction log system). This includes loan details, repayment plans, transaction logs, and customer information. Based on pre-configured credit reporting rules, such as the Credit Reporting 2.0 reporting rules, the big data platform can process the incremental business data layer by layer through a data warehouse layering model (including first, second, and third data warehouse layers) to generate a result table that completely corresponds to each data item in the credit reporting message (e.g., an XML format message or a fixed-length TXT format message required by the central bank's credit reporting system). Subsequently, the platform directly extracts the processed result table from the third data warehouse layer (which does not require calculation or processing of the credit reporting business data and result table; for example, business calculations are performed according to the credit reporting rules), assembles it according to the format requirements of the central bank's credit reporting system, and generates various credit reporting messages such as account information messages, credit agreement messages, and repayment record messages. Afterwards, the data can be encrypted and compressed before being reported to the central bank's credit reporting system.

[0018] In one embodiment, the solution provided in this application can be executed by any electronic device with data processing capabilities. For example, the electronic device can be a server, specifically a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Alternatively, the electronic device can be a terminal device, specifically a tablet computer, laptop computer, or desktop computer. Furthermore, the electronic device can be a combination of a server and a terminal device, wherein the server and terminal device in the combination can communicate wirelessly or via wired means. This application does not impose specific limitations on the electronic device.

[0019] For example, the data side can be a big data platform (or a big data platform), which can be deployed on a server cluster or a distributed system (e.g., on the distributed system of the aforementioned electronic device or on the server cluster to which the electronic device belongs). The data side can handle the storage and complex computation tasks of massive amounts of data exceeding TB, and can adopt a cluster architecture composed of multiple servers, using distributed computing frameworks (such as Hadoop and Spark) for parallel processing to ensure the efficiency and scalability of data processing for credit reporting business.

[0020] The platform side can be a visualization platform (or visualization platform), which can be deployed on an independent application server or cloud server (e.g., the application server or cloud server to which the aforementioned electronic devices belong). For example, it can be deployed using a single server or a lightweight cluster, or it can share some server resources with the data side, while logically remaining independent of each other.

[0021] In one deployment method, the data side and the platform side can be deployed on different nodes in the same server cluster, and high-speed data interaction can be carried out through the internal network.

[0022] In another deployment method, the data side can be deployed in a big data platform in a private cloud or public cloud, while the platform side can be deployed on a separate application server. The two communicate through a network interface or database connection.

[0023] It should be noted that all technical solutions in this application can be combined in any way to form optional embodiments of this application. To avoid repetition, these will not be elaborated upon.

[0024] Furthermore, the specific embodiments of this application involve data, information, or instructions such as credit reporting business data, result tables, and credit reporting messages. When the embodiments of this application are applied to specific products or technologies, user permission, consent, or authorization is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards.

[0025] The technical solution of this application will be described in detail below: In one embodiment, Figure 1 This is a flowchart illustrating a data reporting method provided in an embodiment of this application. Figure 1 As shown, the method may include S110-S120.

[0026] Before introducing S110-S120, let's first combine... Figure 2This application introduces a data reporting method based on Credit Reporting 2.0. Credit Reporting 2.0 refers to the data reporting standards of the People's Bank of China's credit reporting system, which includes the format, content, and timeliness specifications that financial institutions must adhere to when reporting data to the central bank's credit reporting system.

[0027] In one embodiment, such as Figure 2 As shown, based on the underlying data tables of the business system, the platform's engineering capabilities can be used to process and calculate the data items required for various credit reporting messages, and then all the processed data can be written into a database (such as MySQL). At this time, the model structure of the database table does not fully map to the data items required for the actual production of various credit reporting messages. Finally, the platform extracts the corresponding model data from the database for secondary processing and splicing to produce the message output.

[0028] Specifically, the business system comprises multiple systems, including System 1, System 2, System 3, and System 4, serving as the source of original credit reporting data and storing various basic data generated during the credit business process. For step A, based on the data in the business system, the corresponding data items can be processed and transformed using Structured Query Language (SQL) scripting capabilities, according to the various data model tables provided by the reporting platform's business database, before being written into the reporting platform's business database. The business database contains various detailed tables and intermediate tables required for credit reporting, such as: Credit Reporting - Loan Basic Information Table, Repayment Transaction Record Table, Repayment Plan Tracking Table, Credit Limit Information Table, Personal Basic Information Table, and Historical Data Table, etc. For step B1, based on the processed model data tables in the business database, the daily historical cross-sectional data table can be processed according to the reporting platform's code processing rules to retain the latest cross-sectional status data for convenient use in subsequent processed message data. For step B2, based on the processed model data table in the business database and the historical cross-section table processed in B1, the data can be processed and transformed according to the various data tables of the reporting platform and the code processing rules of the reporting platform, and then the data is stored in the reporting database. For step C, based on the data model table in the reporting database, the reporting platform's engineering code can perform secondary data processing and transformation on multiple data tables according to the data item requirements of each type of message, and the final result set data is stored in the message.

[0029] However, the above process has the following problems: The processing logic is too fragmented, with business logic residing in both SQL and code, leading to maintenance difficulties. Furthermore, the data flow from the business database to the reporting database generates numerous intermediate tables, increasing disk I / O (input / output) and storage costs. Additionally, for a specific field in the final credit report, it may be necessary to search for its calculation logic in steps A, B2, and C, making it difficult to pinpoint field issues. The process for adjusting the definitions of various data items or correcting data related to customer complaints during the reporting process is relatively time-consuming and cumbersome.

[0030] The following provides a detailed description of S110-S120. S110-S120 not only enables efficient and easy-to-maintain credit data reporting but also solves the aforementioned technical problems. It should be noted that the embodiments of S110-S120 can refer to the above embodiments.

[0031] S110: Through the data side, the credit reporting business data is calculated according to the credit reporting rules to obtain a result table. The result table includes data that completely corresponds to each data item in the credit reporting message.

[0032] For example, S110 includes the following steps: based on the first data warehouse layering, obtaining credit reporting business data from various business systems; based on the second data warehouse layering, processing the credit reporting business data at the detail level according to the credit reporting reporting rules to generate an intermediate table; based on the third data warehouse layering, generating a result table based on the intermediate table.

[0033] For example, the first layer of the data warehouse could be the Operational Data Store (ODS). This first layer serves as a unified access layer for raw data, i.e., credit reporting data, used to pull credit reporting data from various business systems (such as core loan systems, customer information systems, transaction log systems, etc.). The first layer can perform simple extraction and loading of credit reporting data without complex processing, thus preserving the original data structure.

[0034] For example, credit reporting data includes, but is not limited to: loan details (recording the loan number, loan amount, interest rate, start and end dates, etc. for each loan), transaction records (recording each repayment and loan transaction), repayment schedule (recording the due date and amount for each period), credit limit information (recording the customer's credit limit and available credit), and user information (recording the customer's name, ID number, contact information, etc.).

[0035] For example, the second data warehouse layer could be the Data Warehouse Detail (DWD) layer. The second data warehouse layer is the core layer for processing credit reporting data, where credit reporting rules can be transformed into specific fields and tags, and all business logic calculations are completed at this layer in one go.

[0036] For example, the third data warehouse layer could be the summary data layer (DWS, Data Warehouse Service). This third layer is the result table layer for the final submitted credit reporting messages, storing all the actual submitted credit reporting data. Its processing method involves extracting the required fields from the intermediate tables of the second data warehouse layer, possibly performing aggregation or format adjustments, eliminating the need for complex business logic calculations. The resulting table structure corresponds one-to-one with the message fields (i.e., data items in the credit reporting message) required by the central bank's credit reporting system. The third data warehouse layer allows the platform to directly extract data from each table in this layer, assemble it according to the message format, and then submit it without secondary processing.

[0037] For example, the intermediate table includes: a reporting detail label table, used to generate (which may be pre-generated) label information based on loan granular records (e.g., one loan corresponds to one record) according to credit reporting rules, wherein the label information includes at least one of the following: This table displays information such as whether the loan agreement needs to be reported (indicating whether the loan agreement meets the reporting conditions, such as whether the amount meets the requirements, whether the status is normal, etc.), whether to report the entire loan or split it (for revolving loans, indicating whether it should be reported as a single account (R1) or split into multiple accounts (R4)), the amount to be reported (the amount to be reported this time calculated according to the rules), the reporting account type (used to indicate which type of account the loan agreement belongs to, such as R1 (revolving loan consolidation), D1 (non-revolving loan), R4 (revolving loan sub-account)), and the repayment method (such as equal principal and interest payments, interest-only payments followed by principal payments, etc.). This table transforms complex reporting rules into simple label fields, which can be directly used in subsequent processing, avoiding redundant judgments.

[0038] The data collection point details table records the loan information that various accounts need to report at preset observation points. Observation points (referring to the dates when the corresponding loan should be reported) include at least one of the following: account opening date (e.g., the loan disbursement date), settlement date (e.g., the loan repayment date), monthly settlement date (e.g., a fixed monthly reporting date), and non-monthly dates (e.g., other special dates requiring reporting). In other words, it can store the loan information that various accounts need to report at preset observation points at both the account and time point granularities (i.e., one account has multiple records at multiple observation points). Therefore, this table can store account types (R1, R4, D1) and the loan information that needs to be observed and reported at that time, solving the problem of what data to report at what time and providing a time dimension basis for subsequent processing.

[0039] The data reporting details table records the detailed fields that all credit reporting accounts need to report at the loan agreement level. For example, the data reporting details table is generated based on the data reporting details label table, the data collection point details table, and processed credit reporting business data. This table can be the final output of the second data warehouse layer, containing all the field values ​​that need to be reported for each loan agreement, providing complete detailed data for the next layer.

[0040] For example, processing credit reporting data at a detailed level according to credit reporting rules to generate intermediate tables may include: For the reporting detail label table, the loan receipt details table is obtained layer by layer from the first data warehouse. This table includes basic fields for each loan receipt such as receipt number, customer number, product type, loan amount, loan date, receipt status, repayment method, interest rate, and term. Each loan receipt is assessed to determine whether it needs to be reported, based on criteria including: whether the receipt status is valid (normal, overdue, settled, etc., excluding written-off or voided receipts); whether the loan amount meets the reporting threshold (e.g., receipts below a certain amount do not need to be reported); and whether the receipt is on an exemption list (e.g., internal testing receipts). Based on product type and business rules, the reporting method for each loan receipt is determined: for revolving loan products, whether to use consolidated reporting (R1) or separate account reporting (R4) is determined. For non-revolving loan products, they are marked as type D1. For loan receipts that need to be split, a splitting identifier is added, and splitting rules are associated (e.g., splitting by contract, splitting by credit limit). Based on the current status of the loan agreement and the reporting rules, calculate the loan amount to be reported this time: for loan agreements in normal status, report the original loan amount; for partially repaid loan agreements, report the remaining principal or the amount under a specific definition. Integrate the above judgments and calculations into tag information, generating one record for each loan agreement, including the following fields: loan agreement number, whether reporting is required (Yes / No), whole report or split report identifier (whole / split), loan amount to be reported, reporting account type (R1 / R4 / D1), repayment method (equal principal and interest, interest-only, etc.), and other auxiliary tags (such as interest rate type, guarantee method, etc.).

[0041] For the detailed collection point table, based on the reporting account type in the reporting detail label table, the loan receipts are categorized into the corresponding account types: R1 account, R4 account, and D1 account. For each account type, the account opening point is determined based on the loan disbursement date of the loan receipt. The information required to be reported at the account opening point includes: basic account opening information, credit limit, and initial loan amount. The settlement status of the loan receipts is monitored, and a settlement point record is generated when the loan receipt status changes to settled. The information required to be reported at the settlement point includes: settlement date, settlement amount, and settlement method (normal settlement / early settlement). For accounts that require monthly reporting, fixed monthly observation points are generated according to the reporting rules. The information required to be reported at the monthly settlement point includes: current balance, overdue status, amount due this month, and amount actually repaid this month. By integrating the above-mentioned various time point information, each account generates a record at each observation point, which includes the following fields: account identifier (associated with a loan receipt or contract), account type (R1 / R4 / D1), observation point type (account opening / settlement / monthly settlement / non-monthly), observation point date, and a summary of the loan receipt information corresponding to the point (such as a list of loan receipt numbers, total amount, etc.).

[0042] For the reported data details table, based on the reported details label table, the following data sources are linked: Basic attributes of the loan documents (amount, interest rate, term, etc.) are obtained from the linked loan document details table; basic information of the borrower (name, ID number, contact information, etc.) is obtained from the linked user information table; and information required to be reported for the loan document at various points in time is obtained from the data collection point details table. The basic information fields of the loan documents are processed, including: loan document number, contract number, customer number, customer name, ID type, ID number, loan disbursement date, due date, original loan amount, current balance, annualized interest rate, and repayment method. Combining the transaction log and repayment plan table, repayment-related information is processed: total amount due (summarizing all amounts due according to the repayment plan), total amount repaid (summarizing all actually repaid amounts according to the transaction log), remaining amount due (total amount due minus total amount repaid), overdue status (determining whether it is overdue based on the current date and repayment plan), overdue days (calculating the current overdue days), and overdue start date (first overdue date). Based on the above data processing, the fields in the reported data details table are obtained. All the processed fields are integrated, and a complete reported data details record is generated for each loan receipt, containing all the fields required for credit reporting, for direct use by the third data warehouse layer.

[0043] In the embodiment of S110 above, the processing of all the above models (one model can correspond to one table) can be realized through a big data batch processing scheduling mechanism. Specifically, according to the dependencies between the models at each layer, the batch processing tasks can be executed sequentially on a daily basis through the scheduling platform: the tasks of the first data warehouse layer are executed only after the tasks of the second data warehouse layer are executed successfully; the tasks of the second data warehouse layer are executed only after the tasks of the third data warehouse layer are executed successfully. Through this scheduling mechanism, the correct order of data processing and the consistency and integrity of the data can be ensured.

[0044] S120: The platform assembles the result table according to the format requirements of the credit report to obtain the credit report.

[0045] For example, S120 includes the following steps: pulling a result table from the data side; converting and splicing the result table according to a preset message template to obtain a credit report message.

[0046] For example, the platform can periodically (e.g., every morning) pull the corresponding result tables directly from the third data warehouse layer (e.g., DWS layer) on the data side. This pull operation includes data querying and reading, and may not involve any business logic calculations.

[0047] For example, the platform, based on the message format required by the central bank's credit reporting system (such as XML or fixed-length TXT), converts and concatenates the retrieved result tables according to a preset message template to generate various credit reporting messages. These credit reporting messages specifically include: Normal messages, such as personal basic information messages, personal credit agreement messages, D1 account messages (non-revolving loan accounts), R1 account messages (revolving loan merged accounts), etc.

[0048] Special messages are used to handle data changes or error correction scenarios, such as credit account identifier change messages, credit account segment correction messages, personal basic information deletion messages, and credit account segment or whole deletion messages.

[0049] After generating the credit report, the platform can also encrypt and compress the credit report to form a final reportable file.

[0050] In the aforementioned S120 embodiment, the platform side is only responsible for data reading, format conversion, and file processing, and does not participate in any calculation logic related to credit reporting. In other words, the platform side code does not contain any calculation logic for People's Bank of China (PBOC) data (which refers to data that needs to be reported to the PBOC credit reporting system, i.e., credit reports, which may include: basic personal information, credit agreement information, account information, repayment records, etc. The values ​​of data items in the credit report can be specific values ​​for each field; for example, the overdue field might have a value of 61-90 days; the amount field might have a value of 1000 yuan). In other words, it does not contain any business calculations for generating data items (such as basic personal information, credit agreement information, account information, repayment records, etc.) that need to be reported to the PBOC credit reporting system. All complex calculations related to PBOC data are pre-completed in the second data warehouse layer on the data side.

[0051] In the above embodiments, the distributed computing capabilities of the big data framework can be leveraged for clustered parallel processing, with horizontally scalable computing power, thus efficiently handling TB-level data processing scenarios. Furthermore, all complex computational logic is completed on the data side in one go, directly producing the final result table (DWS layer). Data does not need to be repeatedly written to the database, reducing disk I / O and storage overhead. The platform-side code is also streamlined, highly efficient, and less prone to errors. In addition, since all computational logic is concentrated in the ETL script on the data side, if there are issues such as adjustments to data definitions or customer complaints, only the computational script needs to be modified and the batch run rerun to update the DWS table data. No changes are required on the platform side, making the data processing process transparent and enabling rapid problem localization. Furthermore, a visualization platform, i.e., the platform side, is introduced, allowing users to configure daily extraction tasks, monitor operational status, and view logs through the interface, reducing maintenance complexity and improving operability.

[0052] In some embodiments, the technical solution of this application further includes the following steps: in response to the reporting caliber adjustment instruction, performing data correction on the data side corresponding to the reporting caliber adjustment, and performing calculations on the corresponding credit reporting business data through the corrected data side to update the result table; not performing data correction on the platform side corresponding to the reporting caliber adjustment, and obtaining a credit reporting message conforming to the adjusted reporting caliber based on the updated result table.

[0053] For example, when the central bank's credit reporting system adjusts its reporting rules (such as adding reporting fields, modifying the calculation logic of existing fields, or adjusting the reporting time point), only the data side needs to be modified accordingly. Specifically, only the corresponding calculation / processing logic / flow in the second data warehouse layer needs to be modified, such as the ETL (Extract, Transform, Load) batch processing script (e.g., modifying the tag generation logic of the reporting detail tag table, modifying the time point generation rules of the collection time point detail table, and modifying the field calculation method of the reporting data detail table). After the modification is completed, the batch processing task is re-executed to update the result table data of the third data warehouse layer. The platform side does not need to make any code modifications or logic adjustments; it still follows the original extraction and assembly logic to pull data from the updated result table to generate credit reports that conform to the new reporting standards.

[0054] In the above embodiments, the response cycle for caliber adjustment can be shortened, and no development resources need to be invested on the platform side.

[0055] In some embodiments, the technical solution of this application further includes the following steps: configuring data quality monitoring of target data through a scheduling platform; performing data quality monitoring on target data; the target data includes at least one of the following: credit reporting business data, result table, and intermediate table.

[0056] For example, by configuring Data Quality Control (DQC) rules in the scheduling platform (e.g., similar to the data side), key data during the data processing can be quality-verified. The monitored objects, i.e., the target data, can include credit reporting business data from the first data warehouse layer, intermediate tables from the second data warehouse layer (such as reporting detail label tables, collection point detail tables, and reporting data detail tables), and result tables from the third data warehouse layer.

[0057] For example, various data quality verification rules can be configured, such as: null value verification (checking whether there are null values ​​in key fields), fluctuation verification (checking whether the fluctuation of data volume or key indicators is within the normal range), enumeration value verification (checking whether the field value is within the specified enumeration range), and association integrity verification (checking whether the association relationship between tables is complete).

[0058] For example, data quality monitoring tasks and batch processing tasks can be integrated and automatically triggered upon completion of each processing task in the aforementioned data warehouse layering. For instance, after the second data warehouse layering task is successfully executed, quality checks on intermediate tables are automatically performed; after the third data warehouse layering task is successfully executed, quality checks on the result tables are automatically performed.

[0059] For example, when data quality monitoring detects anomalies, the scheduling platform can automatically issue alarms (such as emails or SMS messages) to notify relevant personnel to intervene and ensure the accuracy and reliability of the reported data.

[0060] In the above embodiments, the data quality monitoring mechanism can detect problems early in the data processing process, prevent data with quality problems from being reported, and thus ensure the overall data quality of credit reporting.

[0061] In some embodiments, to address the inefficient and error-prone technical problem of requiring manual step-by-step checks and data replenishment when data correction is needed after data quality monitoring, this application provides an automated data correction method, including the following steps: In response to data anomaly alarm commands (e.g., when the aforementioned data quality monitoring determines the existence of target anomaly data, a data anomaly alarm command can be triggered), the lineage of the target anomaly data in each data warehouse layer is analyzed to determine the target source and target processing link of the credit reporting business data corresponding to the target anomaly data; based on the target source and target processing link, the target processing script and target intermediate table related to the target anomaly data in the second data warehouse layer are determined; the target source, target processing script, and target intermediate table are displayed; and correction commands for at least one item in the target source, target processing script, and target intermediate table are received and corresponding corrections are performed. Afterwards, calculation tasks can be triggered step-by-step from the first data warehouse layer to the third data warehouse layer according to the lineage, updating the corresponding intermediate tables and result tables in each data warehouse layer; the platform extracts data from the updated result tables to generate corrected credit reporting messages.

[0062] This embodiment can automate the entire process from data source correction to final message update through automatic parsing of data lineage and automatic triggering of step-by-step reruns. It avoids the tedious operation of manual step-by-step investigation and manual data supplementation, and improves the efficiency and accuracy of data correction.

[0063] In some embodiments, to address potential inconsistencies in credit reporting data during multi-level processing and to improve the reliability of data processing and the accuracy of reported data, this application provides a cross-level consistency verification method, comprising the following steps: After obtaining the intermediate table, extract the summary values ​​of key indicators from the data detail table reported in the second data warehouse layer as the first verification benchmark; after obtaining the result table, extract the summary values ​​of the corresponding key indicators from the result table of the third data warehouse layer as the second verification benchmark; compare the first verification benchmark with the second verification benchmark to determine whether they are consistent; if they are inconsistent, it is determined that there is an inconsistency problem in cross-layer data processing, triggering an alarm and suspending the subsequent message generation process; based on the alarm information, locate the processing link between the second and third data warehouse layers and perform data correction (refer to the above data correction method).

[0064] In some embodiments, the caliber adjustment instruction can be parsed to determine the changed reporting rules and corresponding data items; based on the data lineage of each data warehouse layer, the affected data warehouse layers, intermediate tables, result tables, and ETL task scope can be analyzed; affected sample data can be extracted to construct a regression test dataset; tests can be performed based on the changed rules and sample data to generate a test result table; the test result table can be compared with the expected results to identify abnormal data items and processing steps.

[0065] This embodiment automatically defines the scope of regression testing through change impact analysis, performing targeted testing only on affected data and tasks, thus avoiding the overhead of full regression testing. At the same time, automated test execution and result comparison ensure the safety and reliability of the adjustment of standards.

[0066] In some embodiments, the technical solution of this application will be described again below with reference to schematic diagrams.

[0067] For example, in conjunction with the above embodiments, such as Figure 3 and 4 As shown, the technical solution of this application is divided into a data preparation stage and a message processing output stage according to the processing process; and into a big data architecture processing stage (corresponding to the data side) and a platform processing stage (corresponding to the platform side) according to the role. Among them, the big data architecture processing stage mainly includes data modeling and data ETL processing.

[0068] Specifically, a credit reporting model and a layered data warehouse concept can be established. The raw business data (credit reporting data) collected by various business lines (i.e., business systems) is processed according to the rules of the credit reporting collection specifications (i.e., credit reporting rules) into a "data mart" (a data mart is a thematic data set for credit reporting, including credit data for all scenarios required for credit reporting, such as personal information, credit agreements, account information, etc.). Batch processing capabilities can be used to pre-calculate all data items for reporting and store the results in a table. Then, the back-end platform (i.e., the platform side) can directly extract the relevant content from each result table, embed it according to the format requirements of various message types (i.e., credit reporting message format requirements), store the messages, and encrypt and compress them.

[0069] Generally, the routine reporting content for normal credit reporting (taking personal loans from microfinance companies as an example) includes three main categories: basic personal information, credit agreement information, and account information. Therefore, the model construction is designed around these three categories. Specifically, the ODS layer pulls the original data from various business operations, i.e., credit reporting data, including loan details, transaction records, repayment plans, credit limit information, user information, etc. The DWD layer is crucial for processing credit reporting data; its calculations are mainly used to obtain the following intermediate tables: The reporting details label form is mainly based on the reporting rules, namely the credit reporting rules. It pre-labels the corresponding information at the loan receipt level, mainly covering whether the loan receipt needs to be reported, whether to report the whole transaction or split it, the amount of the loan to be reported, the reporting identifier, the repayment method, etc.

[0070] The data collection point details table mainly stores the loan information that needs to be observed and reported at each collection point for various accounts, such as R1 (consolidated reporting of revolving loans), R4 (sub-accounts of revolving loans), and D1 (non-revolving loans), including information on account opening, settlement, monthly settlement, and non-monthly settlement, which is convenient for subsequent processing and use.

[0071] The data reporting details table mainly stores the information details required for reporting to all credit reporting accounts at the loan granularity level. Based on the above reporting details label table, the data collection point details table, and the original business data, all fields required for credit reporting are calculated.

[0072] The DWS layer model can store all real credit reporting data, and the platform can directly extract the data from each table in this layer to the terminal message.

[0073] All the above models (each intermediate table or result can correspond to one model) are processed by big data batch processing scheduling. Batch processing is run sequentially daily according to the table dependencies. For example, tasks in the DWD layer must wait for the ODS layer tasks to succeed before execution; tasks in the DWS layer must wait for the DWD layer tasks to succeed before execution. Corresponding data quality monitoring (DQC) configurations can be implemented on the scheduling platform. For example, data quality verification rules (such as null value verification and fluctuation verification) can be configured in the scheduling platform, and verification will be automatically executed after batch processing to ensure data accuracy and reliability. The platform periodically retrieves data from the corresponding tables daily and processes the finalized messages according to the message assembly format required by credit reporting requirements. This allows the data side to process all models and credit data calculation logic, while the platform side simply retrieves data, organizes messages, and stores them, without any human data calculation logic.

[0074] The technical solution presented in this application, targeting the credit reporting 2.0 scenario, leverages data modeling capabilities and big data ETL processing workflows to rapidly complete the calculation of various message data items and the generation of message files in scenarios involving massive amounts of data exceeding TB. Simultaneously, it can flexibly support adjustments to the reporting standards for various data items with relatively low manpower costs and efficiently respond to the data processing needs of daily customer complaints.

[0075] Figure 5 This is a schematic diagram of a data reporting device provided in an embodiment of this application. Figure 5 As shown, the data reporting device 500 includes: The processing module 510 is used to calculate the credit reporting business data according to the credit reporting rules on the data side to obtain a result table. The result table includes data that completely corresponds to each data item of the credit report. The output module 520 is used to assemble the result table according to the format requirements of the credit report on the platform side to obtain the credit report.

[0076] In some embodiments, the processing module 510 is configured to: obtain credit reporting business data from various business systems based on the first data warehouse layering; perform detailed processing on the credit reporting business data according to the credit reporting reporting rules based on the second data warehouse layering to generate an intermediate table; and generate a result table based on the intermediate table based on the third data warehouse layering.

[0077] In some embodiments, the intermediate table includes: a reporting detail label table, used to record label information generated according to credit reporting rules based on the loan granularity, the label information including at least one of the following: whether the loan needs to be reported, whole report or split report identifier, loan amount to be reported, reporting account type, repayment method; a collection time point detail table, used to record the loan information that various types of accounts need to report at preset observation time points, the observation time points including at least one of the following: account opening time point, settlement time point, monthly settlement time point, non-monthly time point; and a reporting data detail table, used to record the detail fields that all credit reporting accounts under the loan granularity need to report.

[0078] In some embodiments, the data reporting details table is generated based on the data reporting details label table, the data collection time point details table, and credit reporting business data.

[0079] In some embodiments, the output module 520 is used to pull a result table from the data side; and to perform format conversion and splicing of the result table according to a preset message template to obtain a credit report message.

[0080] In some embodiments, the apparatus 500 further includes an adjustment module, configured to: in response to a reporting caliber adjustment instruction, perform data correction on the data side corresponding to the reporting caliber adjustment, and perform calculations on the corresponding credit reporting business data through the corrected data side to update the result table; not perform data correction on the platform side corresponding to the reporting caliber adjustment, and obtain a credit reporting message conforming to the adjusted reporting caliber based on the updated result table.

[0081] In some embodiments, the apparatus 500 further includes a monitoring module for: configuring data quality monitoring of target data through a scheduling platform; performing data quality monitoring on the target data; the target data includes at least one of the following: credit reporting business data, result table, and intermediate table.

[0082] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 5 The apparatus 500 shown can execute the above-described method embodiments, and the aforementioned and other operations and / or functions of each module in the apparatus 500 are respectively for implementing the corresponding processes in the above-described methods. For the sake of brevity, they will not be described in detail here.

[0083] The apparatus 500 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0084] Figure 6 A schematic diagram of an electronic device provided in an embodiment of this application.

[0085] like Figure 6 As shown, the electronic device 600 may include: The system includes a memory 610 and a processor 620. The memory 610 stores computer programs and transfers the program code to the processor 620. In other words, the processor 620 can retrieve and run the computer program from the memory 610 to implement the methods described in the embodiments of this application.

[0086] For example, the processor 620 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0087] In some embodiments of this application, the processor 620 may include, but is not limited to: General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0088] In some embodiments of this application, the memory 610 includes, but is not limited to: Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0089] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0090] like Figure 6 As shown, the electronic device may also include: Transceiver 630, which can be connected to processor 620 or memory 610.

[0091] The processor 620 can control the transceiver 630 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0092] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0093] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0094] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, the computer can perform all or part of the corresponding processes in the methods of the embodiments of this application, producing the functions achievable by the methods of the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid state disks (SSDs)).

[0095] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0097] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0098] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data reporting method, characterized in that, include: On the data side, the credit reporting business data is calculated according to the credit reporting rules to obtain a result table, which includes data that completely corresponds to each data item of the credit report. The platform assembles the result table according to the format requirements of the credit report to obtain the credit report.

2. The method according to claim 1, characterized in that, The calculation of credit reporting data according to the credit reporting rules yields a result table, including: Based on the first data warehouse layering, the credit reporting business data is obtained from various business systems; Based on the second data warehouse layering, the credit reporting business data is processed at the detailed level according to the credit reporting rules to generate an intermediate table; Based on the third data warehouse layering, the result table is generated from the intermediate table.

3. The method according to claim 2, characterized in that, The intermediate table includes: The reporting detail label table is used to generate label information based on the loan receipt granularity record in accordance with the credit reporting rules. The label information includes at least one of the following: whether the loan receipt needs to be reported, whether the entire loan should be reported or the loan should be split, the loan amount to be reported, the reporting account type, and the repayment method. The data collection time point details table is used to record the loan information that various accounts need to report at preset observation time points. The observation time points include at least one of the following: account opening time point, settlement time point, monthly settlement time point, and non-monthly time point. The data reporting details table is used to record the detailed fields that all credit accounts under the aforementioned loan granularity need to report.

4. The method according to claim 3, characterized in that, The reported data details table is generated based on the reported details label table, the collection time point details table, and the credit reporting business data.

5. The method according to claim 1, characterized in that, The step of assembling the result table according to the format requirements of the credit report to obtain the credit report includes: Retrieve the result table from the data side; The result table is formatted and assembled according to a preset message template to obtain the credit report message.

6. The method according to any one of claims 1-5, characterized in that, Also includes: In response to the reporting caliber adjustment instruction, the data side is corrected to correspond to the reporting caliber adjustment, and the calculation of the corresponding credit business data is performed on the corrected data side to update the result table; No data correction is performed on the platform side corresponding to the adjustment of the reporting criteria, and a credit report that conforms to the adjusted reporting criteria is obtained based on the updated result table.

7. The method according to any one of claims 1-5, characterized in that, Also includes: Configure data quality monitoring for target data through the scheduling platform; Perform the data quality monitoring on the target data; The target data includes at least one of the following: the credit reporting business data, the result table, and the intermediate table.

8. A data reporting device, characterized in that, include: The processing module is used to calculate the credit reporting business data according to the credit reporting rules through the data side and obtain a result table. The result table includes data that completely corresponds to each data item of the credit report. The output module is used to assemble the result table according to the format requirements of the credit report message through the platform side to obtain the credit report message.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images