Methods and apparatus for verifying indicator data
By converting business rules into technical rules, and using these technical rules to process business data and verify report metrics, the problems of high database pressure and time-consuming manual comparison in existing technologies are solved, achieving efficient and secure metric data verification and monitoring.
Patent Information
- Application Number
- CN202211515072.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-30
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-11-30
AI Technical Summary
Existing technologies require frequent access to the business database when verifying report metrics, which puts a lot of pressure on the database and poses a risk of data leakage. Furthermore, manually comparing detailed data discrepancies is time-consuming and labor-intensive, and it is impossible to detect metric data errors caused by changes in business logic in a timely manner.
By converting business rules into technical rules, processing business data using these technical rules, calculating technical indicators and comparing them with report indicators, the system reduces access to the business database, automatically verifies the accuracy of indicator data, and provides monitoring and early warning when business rules change.
It reduced the pressure on the business database, improved data storage security, saved human resources, and enabled timely detection and repair of metric data errors caused by changes in business logic, thus ensuring the accuracy and availability of metric data.
Smart Images

Figure CN118113716B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for verifying indicator data. Background Technology
[0002] As the company's business grows, operations and management personnel need to understand the current state of the business through reports to quickly and intuitively grasp the situation and make decisions. All reports are essentially composed of metrics, which are governed by business rules. After developing the big data processing logic for a metric based on these business rules, it's necessary to verify the accuracy of the resulting report metrics. Currently, verifying these metrics involves executing corresponding database queries in the business database, comparing the results with the detailed data processed by the logic, identifying discrepancies, analyzing the problems in the metric processing logic, fixing the logic, and re-executing until the report metric error is within acceptable limits. However, when the metric processing logic is complex, directly executing the query in the business database puts significant pressure on the database and poses a risk of data leakage. Furthermore, when data discrepancies occur, manual investigation and reverse engineering of the data processing logic are required, which is time-consuming and labor-intensive. Summary of the Invention
[0003] In view of this, embodiments of the present invention provide a method and apparatus for verifying indicator data. This method can quickly convert complex business rules into simple technical rules, process business data according to the technical rules to calculate technical indicators, and compare these technical indicators with report indicators obtained from big data processing logic generated according to the business rules. This eliminates the need to re-execute processing logic in the business database, thereby reducing the pressure on the business database and improving data storage security. Furthermore, it eliminates the need for manual comparison of detailed differences when data discrepancies exist, saving time and manpower. Simultaneously, when online business rules change, monitoring and early warning of report indicators can promptly detect errors in indicator data caused by changes in business logic without adjustments to the big data processing logic, ensuring the accuracy and usability of the indicator data.
[0004] To achieve the above objectives, according to one aspect of the present invention, a method for verifying indicator data is provided, comprising:
[0005] Obtain pre-configured technical rules, which are generated based on business rules;
[0006] The business data is processed according to the aforementioned technical rules to obtain technical indicators;
[0007] The report metrics are verified based on the technical indicators, which are obtained by processing the business data according to the business rules.
[0008] Optionally, the business rule is a business description of the report indicator, and the technical rule is generated by field extraction and algorithm abstraction based on the business description; before obtaining the pre-configured technical rule, the method further includes: saving the mapping relationship between the technical rule and the business rule.
[0009] Optionally, the technical rules include fields and algorithms; processing business data according to the technical rules to obtain technical indicators includes: extracting first field data corresponding to the fields and second field data depending on the fields from the business data according to the fields included in the technical rules; and calculating the first field data and the second field data according to the algorithms included in the technical rules to obtain technical indicators.
[0010] Optionally, the business metrics include business metric identifiers and business metric values, and the report metrics include report metric identifiers and report metric values. Verifying the report metrics based on the technical metrics includes: verifying whether the metric identifiers of the technical metrics and the report metrics are consistent based on the business metric identifiers and the report metric identifiers; if the metric identifiers are consistent, for each business metric identifier, obtaining the report metric value corresponding to the report metric identifier that is consistent with the business metric identifier; verifying whether the metric values of the technical metrics and the report metrics are consistent based on the business metric value corresponding to the business metric identifier and the obtained report metric value; if the metric values are consistent, the technical metrics and the report metrics are verified to be consistent.
[0011] Optionally, after verifying the report indicators according to the technical indicators, the method further includes: if the technical indicators and the report indicators are inconsistent, obtaining the business data and the report data respectively, wherein the report data is data including the report indicators obtained by processing the business data; and comparing the business data and the report data line by line to obtain a comparison result.
[0012] Optionally, acquiring the business data and report data separately includes: mapping extraction logic for the business data and report data according to the technical rules; assembling and transforming the data through code logic to generate data extraction statements according to the extraction logic; and extracting the business data and report data from the business data source and report data source respectively according to the data extraction statements.
[0013] Optionally, before acquiring the business data and report data respectively, the method further includes: calculating the error rate of the technical indicators and the report indicators; and confirming that the error rate exceeds a preset error range.
[0014] Optionally, after comparing the business data and the report data line by line to obtain the comparison result, the method further includes: determining the problem of the processing logic corresponding to the business rule and / or the problem of the technical rule based on the comparison result, wherein the problem is used to adjust the processing logic and / or the technical rule.
[0015] Optionally, after verifying the report indicators based on the technical indicators, the method further includes: monitoring the difference rate of the report indicators when the technical indicators and the report indicators are verified to be consistent, and issuing an indicator warning when the difference rate exceeds a set threshold.
[0016] According to another aspect of the present invention, a verification apparatus for indicator data is provided, comprising:
[0017] The technical rule acquisition module is used to acquire pre-configured technical rules, which are generated based on business rules;
[0018] The technical indicator generation module is used to process business data according to the technical rules to obtain technical indicators;
[0019] The indicator comparison and verification module is used to verify the report indicators based on the technical indicators, which are obtained by processing the business data according to the business rules.
[0020] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for verifying indicator data provided in the embodiments of the present invention.
[0021] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method for verifying indicator data provided in the embodiments of the present invention.
[0022] One embodiment of the above invention has the following advantages or beneficial effects: By acquiring pre-configured technical rules, which are generated based on business rules; processing business data according to the technical rules to obtain technical indicators; and verifying report indicators based on the technical indicators, which are technical solutions obtained by processing business data according to business rules, complex business rules can be quickly converted into simple technical rules. Technical indicators are calculated by processing business data according to the technical rules, and by comparing the technical indicators with report indicators obtained from big data processing logic generated according to business rules, the processing logic does not need to be executed again in the business database, thereby reducing the pressure on the business database and improving data storage security. Furthermore, the embodiment of the present invention compares technical indicators with report indicators to verify the accuracy of the indicator data, and determines whether adjustments to the big data processing logic or technical rules are needed based on the verification results. This eliminates the need for manual comparison of detailed difference data when data discrepancies exist, saving time and manpower. Simultaneously, when online business rules change, monitoring and early warning of report indicators can promptly detect errors in indicator data caused by changes in business logic but failure to adjust the big data processing logic, ensuring the accuracy and availability of indicator data.
[0023] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description
[0024] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:
[0025] Figure 1 This is a schematic diagram of the main steps of the method for verifying indicator data according to an embodiment of the present invention;
[0026] Figure 2 This is a schematic diagram illustrating the process of verifying report indicators based on technical indicators according to an embodiment of the present invention;
[0027] Figure 3 This is a schematic diagram of the indicator monitoring and early warning processing procedure according to an embodiment of the present invention;
[0028] Figure 4 This is an architecture diagram of the indicator verification system according to an embodiment of the present invention;
[0029] Figure 5 This is a schematic diagram of the main modules of the indicator data verification device according to an embodiment of the present invention;
[0030] Figure 6 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;
[0031] Figure 7This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation
[0032] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0033] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.
[0034] In existing technologies, verifying report indicator data requires requesting access to the business database each time, which is cumbersome and poses certain data risks. When the processing logic of indicators generated according to business rules is complex, directly executing the processing logic in the business database can put significant pressure on the database, also posing risks. Furthermore, when data discrepancies occur, manual comparison of detailed differences is required, which is time-consuming and labor-intensive. Even after identifying discrepancies, it's necessary to reverse-engineer the data processing logic, modify the big data processing logic of the report indicators, recalculate the results, and re-compare, further increasing database pressure. When online business rules change, the processing logic of report indicators also needs to be modified promptly; otherwise, it will affect analytical decisions. However, existing technologies cannot determine whether the processing logic of report indicators has been modified.
[0035] To address the aforementioned technical problems in existing technologies, this invention provides a method for verifying indicator data. Through relevant configuration, business rules can be quickly converted into technical rules. Technical indicators are then calculated by processing business data according to these technical rules. The accuracy of the indicator data is verified by comparing these technical indicators with report indicators obtained from big data processing logic generated based on the business rules. Based on the verification results, it is determined whether adjustments to the big data processing logic or technical rules are necessary. Finally, if the indicator data verification is consistent, a comparison-based early warning system is implemented to promptly identify indicator data errors caused by changes in business logic without adjustments to the big data processing logic.
[0036] Figure 1This is a schematic diagram illustrating the main steps of the indicator data verification method according to an embodiment of the present invention. Figure 1 As shown, the verification method for indicator data in this embodiment of the invention mainly includes the following steps S101 to S103.
[0037] Step S101: Obtain pre-configured technical rules, which are generated based on business rules. According to an embodiment of the present invention, a basic configuration module can be set up to maintain and configure data and rules. Here, data refers to business data generated by the business system and detailed data from report analysis and processing. Rules refer to rules for different dimensions of indicator statistics, including business rules and technical rules. Business rules are used to generate big data processing logic for indicator data, and technical rules are derived from business rules. According to an embodiment of the present invention, business rules are business descriptions of report indicators, and technical rules are generated by field extraction and algorithm abstraction based on business descriptions. Furthermore, business rules and technical rules are interrelated; therefore, before obtaining pre-configured technical rules, the mapping relationship between technical rules and business rules can be pre-saved.
[0038] Report metrics are generally generated based on business data, so it's necessary to maintain both business data sources and report data sources. Business data sources typically include MySQL databases, distributed databases like TiDB, cached Redis databases, local files, and big data-related databases such as ClickHouse and Elasticsearch. Report data sources typically include big data-related databases such as ClickHouse, Elasticsearch, Hive, and MySQL. When configuring data sources, corresponding connection information (such as domain name / IP address and port number) and data access permission information (such as username and password) can be maintained for different data sources. For security reasons, in this embodiment, passwords are encrypted using a unified encryption algorithm, ACES, and decrypted on the server side. For each data source, it's also necessary to configure the relevant tables involved in the metrics, including field configuration for processing metrics and report analysis dimension field configuration, such as time interval, region, and category fields.
[0039] Business rules are the business descriptions of report metrics, used to generate big data processing logic. For example, the comprehensive human efficiency metric describes the efficiency of processing documents per person; the average cost per document is the average cost required to process one document. The calculation results of report metrics need to be transformed from business rules into technical rules before the results can be output.
[0040] Technical rules are derived from business rules. They mainly consist of two parts: algorithms and fields. For example, the average efficiency of processing orders per person corresponds to the technical rule sum(duration) / manpower / hours. To calculate the average cost per order, it is sum(manpower cost + operating cost) / sum(order volume). Technical rules can be abstracted into algorithms, and the data variables used in the algorithms are the fields.
[0041] In embodiments of the present invention, a rule conversion mapping tool is implemented to facilitate the conversion of business rules and technical rules, enabling automatic mapping conversion. If automatic mapping is not currently possible, or if errors are not automatically handled, custom mapping can be supported.
[0042] Step S102: Process business data according to technical rules to obtain technical indicators. As mentioned earlier, technical rules include two key elements: fields and algorithms. Processing business data according to technical rules to obtain technical indicators may specifically include: extracting the first field data corresponding to the field and the second field data that the field depends on from the business data according to the fields included in the technical rules; and calculating the first field data and the second field data according to the algorithms included in the technical rules to obtain the technical indicators. According to the technical rules, the calculation method for each indicator can be obtained. When calculating the indicator, the data required may include the first field data corresponding to the field and the second field data that the field depends on. For example, when calculating the order quantity, it may depend on the time field data.
[0043] According to an embodiment of the present invention, when processing business data to obtain technical indicators in accordance with technical rules, a dedicated account for the business database can be applied for for unified management, so as to avoid the trouble and data risks caused by applying for business database permissions every time.
[0044] Step S103: Verify the report metrics based on technical indicators. The report metrics are obtained by processing business data according to business rules. The report metrics, along with the report data including the report metrics, are obtained by generating big data processing logic based on business rules to process the business data.
[0045] According to one embodiment of the present invention, business indicators include business indicator identifiers and business indicator values, and report indicators include report indicator identifiers and report indicator values. Verifying report indicators based on technical indicators includes: verifying whether the indicator identifiers of the technical indicators and report indicators are consistent based on the business indicator identifiers and report indicator identifiers; if the indicator identifiers are consistent, for each business indicator identifier, obtaining the report indicator value corresponding to the report indicator identifier that is consistent with the business indicator identifier, and verifying whether the indicator values of the technical indicators and report indicators are consistent based on the business indicator value corresponding to the business indicator identifier and the obtained report indicator value; if the indicator values are consistent, the technical indicators and report indicators are verified to be consistent. Correspondingly, if the indicator identifiers of the technical indicators and report indicators are inconsistent, then the technical indicators and report indicators are verified to be inconsistent; if the indicator identifiers of the technical indicators and report indicators are consistent, but the indicator values corresponding to the indicator identifiers are inconsistent, then the technical indicators and report indicators are still verified to be inconsistent.
[0046] According to one embodiment of the present invention, after verifying the report indicators based on technical indicators, the method further includes: if the verification of technical indicators and report indicators is inconsistent, acquiring business data and report data respectively, wherein the report data is data including report indicators obtained by processing business data; and comparing the business data and report data line by line to obtain a comparison result. When the verification of technical indicators and report indicators is inconsistent, it indicates that there is an error in the processing result. Therefore, it is necessary to acquire business data and report data obtained from big data processing respectively, and compare them line by line to specifically determine the inconsistent records.
[0047] However, not all inconsistencies require a line-by-line comparison of business data and report data. The cause of the error can be analyzed to determine if the inconsistency stems from issues with the big data processing logic or technical rules. If so, and the error range is acceptable to the business, the metric calculation is considered accurate. In embodiments of this invention, before acquiring business data and report data, the method further includes: calculating the error rate of technical and report metrics; and confirming that the error rate exceeds a preset error range. For example, under normal circumstances, inconsistencies in the verification results of technical and report metrics due to delays in big data synchronization of business data or errors in real-time data metrics do not require line-by-line comparison. Typically, the error rate of technical and report metrics caused by delays in big data synchronization of business data or errors in real-time data metrics is not significant. Therefore, an error range can be set. When the error rate of technical and report metrics does not exceed this range, it is unnecessary to acquire business data and report data for line-by-line comparison.
[0048] Figure 2This is a schematic diagram illustrating the process of verifying report indicators based on technical specifications according to an embodiment of the present invention. For example... Figure 2 As shown, firstly, technical indicators are calculated from business data in the business data source according to technical rules; then, consistency verification is performed between the technical indicators and the report indicators; if the verification is inconsistent, the error rate between the technical indicators and the report indicators is calculated, and the verification result is determined based on whether the error rate exceeds the set error range, indicating whether the error is controllable or requires further investigation. In other words, the verification result can include three scenarios: consistent verification, controllable error, and the need for further investigation.
[0049] According to one embodiment of the present invention, when the technical indicators and report indicators are inconsistent, business data and report data are obtained separately for line-by-line comparison. Obtaining business data and report data separately includes: mapping extraction logic for business data and report data according to technical rules; assembling and converting the data according to code logic to generate data extraction statements based on the extraction logic; and extracting business data and report data from the business data source and report data source respectively according to the data extraction statements. For example, the average document processing efficiency per person requires three parts in the business data and report data: duration (start and end time of document processing); manpower (personnel handling the document); and hours (personnel attendance data for document processing). Specifically, for example, the technical rules are: condition: createTime>20220401; statistical function: count(); dimension: dept; corresponding data table: aaa. The data extraction statement output by assembling and converting the data according to code logic is: Select dept, count(1) from aaa where createTime>20220401 group by dept. Next, connect to the business data source and report data source, and execute the data extraction statement respectively to extract the business data and report data, and write the extracted data to a CSV file and return it. CSV, short for Comma-Separated Values, is sometimes called character-separated values because the separator character can be anything other than a comma. Its file stores tabular data (numbers and text) in plain text format.
[0050] The extracted data is compared with business data and big data processed report data. Difference fields in each row are identified, and potential logical issues causing these discrepancies are provided for developers' reference. In one embodiment of this invention, the comparison results obtained by comparing business data and report data row by row are shown in Table 1 below.
[0051] Table 1
[0052]
[0053] As shown in Table 1, the comparison results show the inconsistent records and the possible reasons for the inconsistencies, which will facilitate further adjustments later.
[0054] According to one embodiment of the present invention, after comparing business data and report data line by line to obtain comparison results, the method further includes: determining, based on the comparison results, problems in the processing logic and / or technical rules corresponding to the business rules, whereby these problems are used to adjust the processing logic and / or technical rules. Based on the possible causes of the problems given in the data comparison results, it can be checked whether there are problems with the technical rules for business data statistics. If it is a problem with the technical rules, the technical rules can be adjusted, and the processing results of the business data and report data can be obtained again for comparative analysis. When it is necessary to adjust the technical rules, the problem usually involves formula issues, such as a greater than sign being written as a less than sign, less than or equal to, or only less than, etc. In this case, the formulas of the technical rules need to be adjusted and then verified again.
[0055] If the issue stems from the big data processing logic corresponding to the business rules, the processing logic needs to be modified, and the business data needs to be processed again to obtain report data. This report data should then be compared with the technical metrics until the comparison results are consistent. Additionally, to address the data latency issue with real-time data, both business data and report data can be retrieved at the corresponding time points to minimize the impact of data latency.
[0056] According to another embodiment of the present invention, after verifying the report indicators based on technical indicators, the method further includes: monitoring the difference rate of the report indicators when the technical indicators and report indicators are consistent, and issuing an indicator warning when the difference rate exceeds a set threshold. After verifying the accuracy of the indicators, it is necessary to monitor the difference rate of the report indicators so that an indicator warning can be issued when the difference rate exceeds a set threshold. For example, if the normal average daily order volume is around 100,000, but suddenly reaches 300,000 one day, the difference rate of the indicator is large, indicating an anomaly. In this case, it is not advisable to wait until the data is reviewed to discover the problem; it is necessary to be aware in advance that the data may be abnormal and then analyze the cause so that the problem can be identified and resolved as early as possible, thereby avoiding the use of incorrect indicator data that could influence decision-making.
[0057] Figure 3 This is a schematic diagram illustrating the indicator monitoring and early warning processing procedure of an embodiment of the present invention. For example... Figure 3 As shown, when an abnormality is detected, the first step is to determine whether the business rules have been changed. If so, the problem in the business system is fixed, the big data processing logic is adjusted, and the big data processing logic is validated with indicator data. Otherwise, it indicates that the big data processing logic is abnormal, so the big data processing logic is fixed and the indicator data is validated.
[0058] Figure 4 This is an architecture diagram of the indicator verification system according to an embodiment of the present invention. Figure 4 As shown, the indicator verification system of this invention can be divided into three layers: a basic layer, an application layer, and a business layer. The basic layer is used for basic configuration, mainly including data source configuration, data table and field configuration, business rule configuration, and technical rule configuration. The application layer is used to perform technical indicator calculation, indicator data comparison, processing logic, technical rule correction, and indicator alerts based on the basic layer. The business layer is used to demonstrate the following based on the functions of the application layer: indicator accuracy, indicator difference points, logic correction, and indicator alerts.
[0059] Figure 5 This is a schematic diagram of the main modules of the indicator data verification device according to an embodiment of the present invention. Figure 5 As shown, the indicator data verification device 500 of this embodiment mainly includes a technical rule acquisition module 501, a technical indicator generation module 502, and an indicator comparison and verification module 503.
[0060] The technical rule acquisition module 501 is used to acquire pre-configured technical rules, which are generated based on business rules;
[0061] The technical indicator generation module 502 is used to process business data according to the technical rules to obtain technical indicators;
[0062] The indicator comparison and verification module 503 is used to verify the report indicators based on the technical indicators, wherein the report indicators are obtained by processing the business data according to the business rules.
[0063] According to one embodiment of the present invention, the business rule is a business description of the report indicator, and the technical rule is generated by field extraction and algorithm abstraction based on the business description; the indicator data verification device 500 further includes a mapping relationship storage module (not shown in the figure), which is used to store the mapping relationship between the technical rule and the business rule before obtaining the pre-configured technical rule.
[0064] According to another embodiment of the present invention, the technical rules include fields and algorithms; the technical indicator generation module 502 can also be used to: extract first field data corresponding to the fields and second field data dependent on the fields from business data according to the fields included in the technical rules; and calculate the first field data and the second field data according to the algorithm included in the technical rules to obtain technical indicators.
[0065] According to another embodiment of the present invention, the business indicator includes a business indicator identifier and a business indicator value, and the report indicator includes a report indicator identifier and a report indicator value; the indicator comparison and verification module 503 can also be used to: verify whether the indicator identifiers of the technical indicator and the report indicator are consistent based on the business indicator identifier and the report indicator identifier; if the indicator identifiers are consistent, for each business indicator identifier, obtain the report indicator value corresponding to the report indicator identifier that is consistent with the business indicator identifier, and verify whether the indicator values of the technical indicator and the report indicator are consistent based on the business indicator value corresponding to the business indicator identifier and the obtained report indicator value; if the indicator values are consistent, the technical indicator and the report indicator are verified to be consistent.
[0066] According to another embodiment of the present invention, the indicator data verification device 500 further includes a data detail comparison module (not shown in the figure), which is used to: after verifying the report indicators according to the technical indicators, if the technical indicators and the report indicators are inconsistent, acquire the business data and the report data respectively, wherein the report data is data including report indicators obtained by processing the business data; and compare the business data and the report data line by line to obtain a comparison result.
[0067] According to another embodiment of the present invention, when the data detail comparison module (not shown in the figure) acquires the business data and the report data respectively, it can be specifically used to: map the extraction logic of the business data and the report data according to the technical rules; generate data extraction statements according to the extraction logic through code logic assembly and conversion; and extract the business data and the report data from the business data source and the report data respectively according to the data extraction statements.
[0068] According to another embodiment of the present invention, the data detail comparison module (not shown in the figure) can also be used to: calculate the error rate of the technical indicators and the report indicators before acquiring the business data and report data respectively; and confirm that the error rate exceeds a preset error range.
[0069] According to another embodiment of the present invention, the indicator data verification device 500 further includes a logic adjustment module (not shown in the figure), which is used to: after comparing the business data and the report data line by line to obtain the comparison result, determine the problems of the processing logic corresponding to the business rule and / or the technical rule according to the comparison result, wherein the problems are used to adjust the processing logic and / or the technical rule.
[0070] According to another embodiment of the present invention, the indicator data verification device 500 further includes an indicator monitoring and early warning module (not shown in the figure), which is used to: after verifying the report indicator according to the technical indicator, if the technical indicator and the report indicator are consistent, monitor the difference rate of the report indicator, and issue an indicator early warning if the difference rate exceeds a set threshold.
[0071] According to the technical solution of this invention, by acquiring pre-configured technical rules (generated based on business rules), processing business data according to the technical rules to obtain technical indicators, and verifying report indicators based on these technical indicators (which are obtained by processing business data according to business rules), complex business rules can be quickly converted into simple technical rules. Technical indicators are calculated by processing business data according to these technical rules, and the technical indicators are compared with report indicators obtained from big data processing logic generated according to business rules. This eliminates the need to re-execute processing logic in the business database, reducing pressure on the business database and improving data storage security. Furthermore, by comparing technical indicators with report indicators, the accuracy of the indicator data is verified, and adjustments to the big data processing logic or technical rules are determined based on the verification results. This eliminates the need for manual comparison of detailed difference data when data discrepancies exist, saving time and manpower. Simultaneously, when online business rules change, monitoring and early warning of report indicators can promptly detect errors in indicator data caused by changes in business logic without adjustments to the big data processing logic, ensuring the accuracy and availability of the indicator data.
[0072] Figure 6 An exemplary system architecture 600 is shown, in which the verification method or apparatus for indicator data of embodiments of the present invention can be applied.
[0073] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0074] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, database applications, search applications, instant messaging tools, data processing applications, etc. (for example only).
[0075] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0076] Server 605 can be a server providing various services, such as a backend management server supporting websites browsed by users using terminal devices 601, 602, and 603 (for example only). The backend management server can retrieve pre-configured technical rules from received data such as indicator data verification requests; these technical rules are generated based on business rules; process the business data according to the technical rules to obtain technical indicators; verify report indicators according to the technical indicators, which are obtained by processing the business data according to the business rules; and feed back the processing results (e.g., indicator data verification results – for example only) to the terminal devices.
[0077] It should be noted that the verification method for indicator data provided in this embodiment of the invention is generally executed by server 605, and correspondingly, the verification device for indicator data is generally set in server 605.
[0078] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0079] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing terminal devices or servers of the present invention. Figure 7 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.
[0080] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0081] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.
[0082] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.
[0083] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0085] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a technical rule acquisition module, a technical indicator generation module, and an indicator comparison and verification module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the technical rule acquisition module can also be described as "a module for acquiring pre-configured technical rules."
[0086] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to include: acquiring pre-configured technical rules, the technical rules being generated according to business rules; processing business data according to the technical rules to obtain technical indicators; and verifying report indicators according to the technical indicators, the report indicators being obtained by processing the business data according to the business rules.
[0087] According to the technical solution of this invention, by acquiring pre-configured technical rules (generated based on business rules), processing business data according to the technical rules to obtain technical indicators, and verifying report indicators based on these technical indicators (which are obtained by processing business data according to business rules), complex business rules can be quickly converted into simple technical rules. Technical indicators are calculated by processing business data according to these technical rules, and the technical indicators are compared with report indicators obtained from big data processing logic generated according to business rules. This eliminates the need to re-execute processing logic in the business database, reducing pressure on the business database and improving data storage security. Furthermore, by comparing technical indicators with report indicators, the accuracy of the indicator data is verified, and adjustments to the big data processing logic or technical rules are determined based on the verification results. This eliminates the need for manual comparison of detailed difference data when data discrepancies exist, saving time and manpower. Simultaneously, when online business rules change, monitoring and early warning of report indicators can promptly detect errors in indicator data caused by changes in business logic without adjustments to the big data processing logic, ensuring the accuracy and availability of the indicator data.
[0088] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for verifying indicator data, characterized in that, include: Obtain pre-configured technical rules, which are generated based on business rules. The business rules are business descriptions of report metrics. The technical rules are generated by extracting fields and abstracting algorithms from the business descriptions using a rule transformation and mapping tool. The rule transformation and mapping tool supports automatic mapping and custom mapping and saves the mapping relationship between the technical rules and the business rules. The business data is processed according to the aforementioned technical rules to obtain technical indicators; The report indicators are verified based on the technical indicators, which are obtained by processing the business data according to the business rules. The verification results include three situations: verification consistency, error controllability, and need for further investigation. If the technical indicators and the report indicators are inconsistent, calculate the error rate between the technical indicators and the report indicators. If the error rate is confirmed to exceed a preset error range, the verification result is determined to require further investigation. Obtain the business data and report data respectively. The report data is data including report indicators obtained by processing the business data. Compare the business data and the report data line by line to obtain the comparison result.
2. The method according to claim 1, characterized in that, The technical rules include fields and algorithms; The business data is processed according to the aforementioned technical rules to obtain technical indicators, including: Based on the fields included in the technical rules, extract the first field data corresponding to the field and the second field data that the field depends on from the business data; The technical indicators are obtained by calculating the first field data and the second field data according to the algorithm included in the technical rules.
3. The method according to claim 1, characterized in that, The business metrics include business metric identifiers and business metric values, and the report metrics include report metric identifiers and report metric values; Validating the report metrics based on the aforementioned technical indicators includes: Based on the business indicator identifier and the report indicator identifier, verify whether the indicator identifiers of the technical indicator and the report indicator are consistent; If the indicator identifiers are consistent, for each business indicator identifier, obtain the report indicator value corresponding to the report indicator identifier that is consistent with the business indicator identifier. Based on the business indicator value corresponding to the business indicator identifier and the obtained report indicator value, verify whether the indicator values of the technical indicator and the report indicator are consistent. If the indicator values are consistent, the technical indicators and the report indicators are verified to be consistent.
4. The method according to claim 1, characterized in that, The business data and report data are obtained respectively, including: The extraction logic for the business data and the report data is mapped according to the technical rules; The data extraction statement is generated based on the extraction logic by assembling and transforming the code logic. According to the data extraction statement, the business data and the report data are extracted from the business data source and the report data source, respectively.
5. The method according to claim 1, characterized in that, After comparing the business data and the report data line by line to obtain the comparison results, the process also includes: Based on the comparison results, problems in the processing logic and / or the technical rules corresponding to the business rules are determined, and these problems are used to adjust the processing logic and / or the technical rules.
6. The method according to claim 1 or 3, characterized in that, After verifying the report metrics based on the aforementioned technical indicators, the process also includes: If the technical indicators and the report indicators are verified to be consistent, the difference rate of the report indicators is monitored, and an indicator warning is issued if the difference rate exceeds a set threshold.
7. A device for verifying indicator data, characterized in that, include: The technical rule acquisition module is used to acquire pre-configured technical rules, which are generated based on business rules. The business rules are business descriptions of report indicators. The technical rules are generated by extracting fields and abstracting algorithms from the business descriptions using a rule conversion and mapping tool. The rule conversion and mapping tool supports automatic mapping and conversion as well as custom mapping, and saves the mapping relationship between the technical rules and the business rules. The technical indicator generation module is used to process business data according to the technical rules to obtain technical indicators; The indicator comparison and verification module is used to verify the report indicators based on the technical indicators. The report indicators are obtained by processing the business data according to the business rules. The verification results include three situations: verification consistency, error controllability, and need for further investigation. The data detail comparison module is used to calculate the error rate between the technical indicators and the report indicators when they are inconsistent. If the error rate exceeds a preset error range, the verification result is determined to require further investigation. The module also acquires the business data and report data, where the report data is data including report indicators obtained by processing the business data. The module compares the business data and the report data line by line to obtain the comparison result.
8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.
9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.
Citation Information
Patent Citations
A business data verification method and device
CN109558134A
Data checking method, device and equipment and computer readable storage medium
CN112035447A
Page data verification method and device, medium and electronic equipment
CN112241417A
