Data reporting method, device, equipment, medium and product

Abstract

The invention discloses a data reporting method, device and equipment, a medium and a product, and relates to the technical field of data processing, and the method comprises the steps: determining initial business data, processing the initial business data through employing a preset business data delivery rule, obtaining at least one to-be-processed task, and sending the at least one to-be-processed task to a server; delivering the at least one to-be-processed task to a to-be-processed message queue; determining a target to-be-processed task from at least one to-be-processed task contained in the to-be-processed message queue based on a received data processing instruction; and processing the target to-be-processed task by using a preset data quality inspection rule to obtain to-be-reported data, and sending the to-be-reported data to a data receiving device, the to-be-reported data being business data qualified in quality inspection in the target to-be-processed task. According to the invention, the reporting efficiency of the service data can be improved, the probability of data accumulation is reduced, quality inspection is carried out on the service data, and the reporting quality of the service data is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data reporting method, device, equipment, medium and product. Background Art

[0002] Data reporting is a process of sending collected data to the data recipient through a certain process and method. For example, departments or branches within an enterprise report business data to the headquarters or data center, and the enterprise reports specific data to relevant management departments in accordance with laws and regulations.

[0003] Data reporting is used in every aspect of enterprise management, and data reporting performance is one of the issues that enterprises are most concerned about. At present, the way to report data is to determine the start time of the last data reporting work, sort the data between the start time and the current time using the Auto-Increment ID, and query in the sorted results according to the business type, associate other fields that need to be queried one by one, and finally report the query results. However, the efficiency of obtaining query results by querying in the Auto-Increment ID sorting structure according to the business type is low, the data reporting rate is slow, and it is easy to cause data accumulation, and the comprehensiveness and real-time nature of data reporting are limited. Summary of the invention

[0004] The present invention provides a data reporting method, device, equipment, medium and product, which can improve the reporting efficiency of business data while ensuring the reporting quality of business data and reduce the probability of data accumulation.

[0005] According to one aspect of the present invention, a data reporting method is provided, the method comprising:

[0006] Determine initial business data, process the initial business data using a preset business data delivery rule to obtain at least one task to be processed, and deliver the at least one task to be processed to a message queue to be processed;

[0007] Based on the received data processing instruction, determining a target to-be-processed task from at least one to-be-processed task included in the to-be-processed message queue;

[0008] The target pending task is processed using the pre-set data quality inspection rules to obtain the data to be reported, and the data to be reported is sent to the data receiving device, wherein the data to be reported is the business data in the target pending task that has passed the quality inspection.

[0009] According to another aspect of the present invention, a data reporting device is provided, and the data reporting device is used to implement the data reporting method in any embodiment of the present invention, and the device includes:

[0010] A data acquisition module is used to determine initial business data, process the initial business data using a preset business data delivery rule, obtain at least one task to be processed, and deliver the at least one task to be processed to a message queue to be processed;

[0011] A task determination module, configured to determine a target to-be-processed task from at least one to-be-processed task included in the to-be-processed message queue based on the received data processing instruction;

[0012] The data reporting module is used to process the target pending task using pre-set data quality inspection rules, obtain the data to be reported, and send the data to be reported to the data receiving device, wherein the data to be reported is the business data that has passed the quality inspection in the target pending task.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and a memory communicatively coupled to the at least one processor;

[0015] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the data reporting method in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data reporting method in any embodiment of the present invention when executed.

[0017] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the data reporting method of any embodiment of the present invention is implemented.

[0018] The data reporting method of the present invention includes: determining initial business data, processing the initial business data using pre-set business data delivery rules to obtain at least one task to be processed, and delivering at least one task to be processed to a message queue to be processed; based on the received data processing indication, determining a target task to be processed from at least one task to be processed contained in the message queue to be processed; using pre-set data quality inspection rules, processing the target task to be processed to obtain data to be reported, and sending the data to be reported to a data receiving device, wherein the data to be reported is business data that has passed the quality inspection in the target task to be processed. The technical solution of the present invention uses business data delivery rules to screen and sort the initial business data, obtains at least one task to be processed and delivers it to the pending message queue, reduces the amount of business data that needs to be reported, improves data delivery efficiency, and reduces the probability of data accumulation; it also matches the tasks that need to be reported from the pending message queue according to the data processing instructions, detects them using data quality inspection rules, filters out abnormal data that cannot be calibrated, adjusts abnormal data that can be calibrated, and then reports the business data after quality inspection to the data receiving device, while improving the data reporting efficiency and ensuring the data reporting quality to a great extent. It solves the problems of the method of querying and determining and reporting the query results in the self-incrementing ID sorting structure according to the business type, the low data query efficiency, the slow data reporting rate, the easy accumulation of data, and the limited comprehensiveness and real-time performance of data reporting.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 It is a flow chart of a data reporting method provided by the present invention;

[0022] Figure 2 It is a schematic diagram of a delivery process of a task to be processed provided by the present invention;

[0023] Figure 3 It is a schematic diagram of the consumption and reporting process of a task to be processed provided by the present invention;

[0024] Figure 4It is a flowchart of another data reporting method provided by the present invention;

[0025] Figure 5 It is a structural schematic diagram of a data reporting device provided by the present invention;

[0026] Figure 6 It is a structural schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", "initial", "intermediate", "target", "candidate", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Data reporting can be local management departments reporting local business data to national management departments. The reported data should cover key information and key information comparison as much as possible, such as subject information, winning bid information, transaction volume information, data coverage comparison of each transaction link, etc., to ensure data upload coverage, data reporting timeliness and comprehensiveness, and report standardized, high-quality and high-real-time data to national management departments. However, the method of querying information and uploading in the self-incrementing ID sorting structure is inefficient. Once the data generation speed of the aggregation location is much faster than the speed of data reporting to the national platform, it will cause data accumulation, affect the efficiency of data reporting, and fail to meet the timeliness requirements of data reporting of the national platform. Secondly, this method only reports the aggregated data to the national platform intact. Due to the uneven data quality of the data source system, low-quality data or duplicate data may be reported to the national platform, which cannot meet the requirements of comprehensiveness and accuracy of data reporting of the national platform.

[0030] Figure 1 This is a flow chart of a data reporting method provided by the present invention. This embodiment can be applied to the data reporting work with high efficiency, high quality, low cost and high real-time performance. The method can be executed by the data reporting device provided by the present invention. The device can be implemented in the form of hardware and / or software. In a specific embodiment, the device can be integrated in an electronic device. The following embodiments will be described by taking the device integrated in an electronic device as an example. Figure 1 , the method specifically comprises the following steps:

[0031] S101, determining initial business data, processing the initial business data using a pre-set business data delivery rule, obtaining at least one task to be processed, and delivering the at least one task to be processed to a message queue to be processed.

[0032] The initial business data can be understood as the business data generated by the system of the prefecture-level unit. The pre-set business data delivery rules can be understood as the scheme for pre-processing the initial business data (including but not limited to data screening and data merging). It can be a screening rule for data with high priority or importance, high usage rate, and more attention from the information receiving device. For example, extract the winning bid data in time period A of the initial business data, extract the transaction volume information in time period B of the initial business data, extract the data coverage comparison of each transaction link in time period C of the initial business data, and merge and sort all the winning bid data in time period D by region. The pending tasks can be understood as the screening and merging results of the initial business data. The initial business data can be sorted and merged into at least one pending task. Each pending task can include at least one business data. For example, the initial business data includes A, B, C and D. The business data delivery rules indicate to merge A and B, delete C, and retain D. Then the pending tasks include two tasks, one consisting of A and B, and the other consisting of D. The pending message queue can be understood as a distributed stream processing platform for log aggregation, stream processing, event source, real-time analysis and other fields. For example, the Apache Kafka message queue has the characteristics of high throughput, scalability and fault tolerance. The queue has the characteristics of partitioning, batch processing, disk input / output optimization, zero replication, message compression, asynchronous input / output, high availability architecture, high-performance network stack, data persistence strategy, configurable throughput and resource isolation. It is mainly used to build real-time data pipelines and streaming applications, aiming to process and transmit large amounts of real-time data in a reliable and scalable way, improve the efficiency of data transmission and processing, and reduce the probability of data accumulation.

[0033] The advantage of this setting is that it can extract information from the initial business data, reduce the data processing volume of subsequent information processing equipment, and ensure data reporting efficiency.

[0034] In one embodiment, S101 may specifically include: determining initial business data based on operating data of at least one external system; using pre-set data conversion rules to process the initial business data to obtain intermediate business data, wherein the data volume of the intermediate business data is equal to or less than the data volume of the initial business data, and the intermediate business data includes at least one sub-intermediate business data; using business data delivery rules to process the intermediate business data to obtain at least one task to be processed, wherein the task to be processed includes at least one sub-intermediate business data.

[0035] The present invention will acquire data in units of prefecture-level cities, process data in units of provincial level, and report the data processing results to national units. The external system can be understood as a data sending device at the prefecture-level city, and the provincial level includes multiple prefecture-level cities. Therefore, the number of external systems is multiple, and the operation data of the external system can be understood as the business data of each prefecture-level city, including but not limited to application operation information, application test logs, system or unit subject information, bid winning information, transaction volume information, data coverage comparison of each transaction link, etc. The data conversion rule can be understood as a data format conversion rule, and the intermediate business data can be understood as the initial business data after the data format is converted into a predetermined format. Due to the data format requirements in different regions and the formats of data generated by different devices, there are certain differences. Therefore, the data format in the initial business data is not unified. In order to speed up the efficiency of data identification, analysis and merging, and improve the standardization and neatness of data, the present invention will set the data format conversion rule in advance, process the data in the initial business data that is not standardized in format, and obtain the intermediate business data with a unified format, which is convenient for subsequent data processing. Sub-intermediate business data can be understood as a piece of business data, for example, business data in region A, business data in region B, business data in region C, operation log of application A, operation log of application B, etc. Intermediate business data is only the initial business data after format standardization, and the data flow is still large. Business data delivery rules can be understood as a plan for data screening and data merging of intermediate business data, including but not limited to data screening according to priority or importance, usage rate, and attention of information receiving devices, and merging according to data type, data generation time, and data association relationship. For example, extract the winning bid data of time period A in the initial business data, extract the transaction volume information of time period B in the initial business data, extract the data coverage comparison of each transaction link in time period C in the initial business data, and merge and organize all winning bid data in time period D by region.

[0036] The advantage of this setting is that it can standardize the business data with a higher probability of being reported, thereby reducing the workload of data processing while ensuring that data reporting is comprehensive and accurate.

[0037] Figure 2 It is a schematic diagram of the delivery process of a pending task provided by the present invention, showing a completion process of data delivery, which can be specifically summarized into two parts. The first part is: the data aggregation service receives business data pushed by multiple external systems and stores it in the business database. The business database includes multiple business libraries, which can realize the partitioning and / or classification storage of information. It is a parallel data access library that can effectively improve the access efficiency of data. The second part is: using the command parameter startup Options=latest-offset, monitor the binary log (binlog log) of the business database, determine the data changes in the business database, and transmit the data changes to the data exchange service. The data exchange service processes it, organizes it into pending tasks and delivers it to the pending message queue.

[0038] The data exchange service of the present invention can be a stream-based data integration tool Flink CDC (ChangeData Capture), which can use database log capture technology to achieve real-time data synchronization, and has end-to-end data integration framework, easy-to-use interface, whole database synchronization, automatic synchronization of table structure changes, data conversion, streaming pipeline, exactly once semantics, community support, integration with Flink and other features, which can achieve real-time data capture and synchronization, and is suitable for data synchronization, real-time analysis and data distribution and other scenarios. The data processing work at the data exchange service includes but is not limited to data format conversion, screening of non-essential data, extraction of important data, and merging of data of the same type or related data. Message queue to be processed Kafka message queue.

[0039] It is worth noting that the data exchange service can perform data processing and message delivery tasks according to the business delivery requirements configured by the user. Data processing and delivery work supports data callback, including full data callback and partial data callback. Full data callback refers to the callback to obtain data outside the log retention time, and partial data callback refers to the callback to obtain data within the log retention time. The storage time of the log is generally half a year. The data not saved in the log will be saved to the disk or memory stick. When the callback time point is within the binlog log retention time, you can use the timestamp mode and set the startupTimestamp Millis (start synchronization timestamp) parameter. The Flink CDC data exchange service monitors the binlog log of the business library and captures the database changes after the "start synchronization timestamp". When the callback time point is before the binlog log retention time, you can use the initial mode and set the sync Time (start synchronization time) parameter. The Flink CDC data exchange service will first capture the data in full, and discard the data before the "start synchronization time", and directly capture the database changes after the "start synchronization time" for data synchronization. The advantage of this setting is that it provides multiple ways to obtain data, expands the usage scenarios of data reporting, and improves the universality of data reporting.

[0040] The present invention adopts Flink CDC data exchange service + Kafka message queue in data delivery. Flink CDC is essentially a set of data source connectors. It uses "Change Data Capture (CDC)" to ingest and change data from different databases. Flink CDC supports real-time consumption logs and stream processing. For example, MySQL's binlog log fully records the data changes in the database. The binlog file can be used as the data source of the stream. Therefore, Flink CDC supports full + incremental synchronization of data, which can ensure data consistency and real-time performance. Secondly, Flink CDC data exchange service is a back-end service program developed based on Java language. Its main functions include: 1) real-time monitoring of MySQL's binlog log, capturing database changes, and converting data formats; 2) delivering processed data to Kafka message queue.

[0041] The present invention mainly uses the flink-connector-mysql-cdc dependency package of Flink CDC to monitor and synchronize the binlog log of the data source in real time, and uses the flink-connector-kafka dependency package to connect the Kafka message queue middleware for message delivery. The specific process includes: 1) introducing Maven dependencies and third-party dependency packages; 2) creating a data synchronization class, writing a program entry main method, as the execution entry of the data synchronization program; 3) writing a startup mode setting method for setting the startup mode of the MySQL CDC user (realizing a data call-back resynchronization mechanism); 4) writing a data conversion class, implementing a deserialization method, for converting the data format to json format (a lightweight data exchange format), capturing database operation types (add / update / delete), and setting the operation type parameters of the reporting national interface; 5) writing a message delivery class, implementing a reflection method, and delivering data to the Kafka message queue; 6) packaging the written program into a jar package, setting startup parameters, using Exchange Data Cdc Main as the startup class, running the jar package, and starting the Flink job.

[0042] S102: Based on the received data processing instruction, determine a target to-be-processed task from at least one to-be-processed task included in the to-be-processed message queue.

[0043] The data processing instruction can be understood as a consumption instruction of a task, which is equivalent to a processing instruction of a pending task in a pending message queue. The data processing instruction includes but is not limited to a task name, task time, task identification information, task key information, etc. The target pending task can be understood as a task that needs to be consumed in a pending message queue. In general, the data processing instruction can be issued by a user based on an interactive device, or can be configured in advance and triggered at a fixed time, which is not limited in this embodiment.

[0044] Exemplarily, the pending message queue includes Task 1, Task 2 and Task 3, Task 1 is named A3, Task 2 is named A2, Task 3 is named A1, and the data processing instruction is to consume Task A2, then the target pending task is determined to be Task 2.

[0045] The advantage of this setting is that it can target consumption tasks according to data consumption instructions, improving the accuracy of data processing. It is worth noting that if there is no task to be consumed in the message queue, a data call indication is generated, which is used to prompt the data exchange service to re- and / or supplement the generation and delivery of tasks, ensuring the automation and integrity of data reporting.

[0046] In one embodiment, S102 may specifically include: determining whether the data acquisition time in the data processing indication meets the log data storage conditions; if the data acquisition time meets the log data storage conditions, matching at least one pending task based on the data acquisition time and the data acquisition identifier in the data processing indication, and determining the target pending task based on the matching result; if the data acquisition time does not meet the log data storage conditions, updating at least one pending task based on the data acquisition time and the data acquisition identifier, matching at least one updated pending task using the data acquisition time and the data acquisition identifier, and determining the target pending task based on the matching result.

[0047] In general, the tasks delivered are all information stored in the log, that is, data within half a year. If the time difference between the data acquisition time in the data processing instruction and the current moment is less than or equal to half a year, it proves that only data within half a year needs to be acquired, and the business data in the current queue is sufficient to perform data reporting work. The target pending tasks can be directly matched in the queue's pending tasks according to the data acquisition identifier. If the time difference between the data acquisition time in the data processing instruction and the current moment is greater than half a year, it proves that data from half a year ago needs to be acquired. It is believed that the data acquisition time does not meet the log data storage conditions, and the tasks in the current queue are insufficient to perform data reporting work. Data needs to be acquired from the disk or memory bar. The data acquisition process can refer to the above-mentioned full data callback method, update the queue according to the acquired business data and data processing instructions (essentially updating the pending tasks), and re-match the target pending tasks.

[0048] Exemplarily, the queue of pending messages includes Task 1, Task 2, and Task 3. Task 1 is named A3, Task 2 is named A2, Task 3 is named A1, and the data processing instruction is A2 task consuming time period X. If the time difference between the start time of time period X and the current time is less than or equal to half a year, then the target pending task is determined to be Task 2. If the time difference between the start time of time period X and the current time is greater than half a year, then the data is re-acquired, Task 2 is updated, and the updated Task 2 is determined to be the target pending task. The advantage of such a setting is that the comprehensiveness and integrity of the uploaded data can be guaranteed.

[0049] Furthermore, at least one pending task is updated based on the data acquisition time and the data acquisition identifier, including: searching in the storage location of the initial business data based on the data acquisition time and the data acquisition identifier to obtain first business data to be updated; determining whether the processing level indicated by the data processing meets the single processing condition; if the processing level meets the single processing condition, processing the first business data to be updated using the business data delivery rule to obtain at least one updated pending task; if the processing level does not meet the single processing condition, processing the first business data to be updated and the initial business data using the business data delivery rule to obtain at least one updated pending task.

[0050] Among them, the first business data to be updated can be understood as the business data that meets the data acquisition time, that is, the business data after the data acquisition time indicated by the data acquisition time. The processing level is used to indicate the importance of the data processing indication. The single processing condition is used to indicate whether it is necessary to suspend the remaining data processing work and use all data bandwidth to perform the current data processing work. If the processing level of the data processing indication meets the single processing condition, it proves that the current data delivery and consumption work is more urgent and urgent, and the remaining data delivery work will be stopped, and all data delivery resources will be used to process the current data delivery work, that is, the business data delivery rules are used to process the first business data to be updated, and at least one updated task to be processed is obtained. If the processing level of the data processing indication does not meet the single processing condition, it proves that the current data delivery work is not an urgent task, and the delivery work of this part of the data will be carried out on the basis of maintaining the original data processing work, that is, the business data delivery rules are used to process the first business data to be updated and the initial business data to obtain at least one updated task to be processed.

[0051] It is worth noting that the initial business data is real-time data and is not static. If the delivery work has a higher priority, for example, to test the performance of an application to be launched, the rest of the tasks in the queue can be cleared, and only the running data of the application can be retained. All data delivery resources can be used to process the data delivery work of the application. The advantage of this setting is that it can effectively improve the delivery efficiency and information reporting rate of a single task.

[0052] S103: using the pre-set data quality inspection rules, processing the target task to be processed, obtaining the data to be reported, and sending the data to be reported to the data receiving device.

[0053] Among them, the data quality inspection rules can be understood as the data qualification inspection plan configured in advance, the data to be reported is the business data that has passed the quality inspection in the target task to be processed, and the data receiving device can be understood as a national-level data receiving unit.

[0054] Specifically, the data quality inspection rules include at least one of the data matching quality inspection rules and the data unification quality inspection rules. The data matching quality inspection rules are used to detect whether the data type and content in the business data match, for example, whether the content corresponding to the name is 2-5 Chinese characters, whether the content corresponding to the ID card is 17 Roman numerals + 1 Roman numeral / letter, and the data unification quality inspection rules are used to detect whether there is a conflict association in the business data, for example, the same data content corresponds to two different names, and there are conflicting fields in the same data. The data quality inspection rules can filter out abnormal data, and the data to be reported is data without abnormalities, which can be data that has been quality-checked correctly, or data that has been quality-checked correctly + data that has errors but has been calibrated. The advantage of such a setting is that the accuracy of the reported data can be ensured.

[0055] Optionally, before using pre-set data quality inspection rules to process the target task to be processed, it also includes: using the data quality inspection rules to determine whether there is a first anomaly in the business data of the target processing task, wherein the first anomaly includes at least one of data duplication, missing data key information, data format error and data information deviation; if there is a first anomaly in the business data of the target processing task, determining a first exception handling plan based on the anomaly type of the business data with the first anomaly, using the first exception handling plan to process the business data with the first anomaly, and updating the target task to be processed according to the data processing result.

[0056] Among them, the first exception handling scheme includes a duplicate data handling scheme, a missing information handling scheme, a format specification scheme and a deviation data handling scheme. The duplicate data handling scheme includes but is not limited to deleting redundant data in duplicate data and merging duplicate data, which is used to handle data duplication exceptions. The missing information handling scheme includes missing data marking and missing data supplementation, which is used to handle data key information missing exceptions. The format specification scheme includes processing abnormal data in a preset format and processing abnormal data in a universal data format. The deviation data handling scheme includes deleting deviation data and marking deviation data. The purpose of this setting is to adjust the abnormal data, convert it into qualified data as much as possible, and record unqualified data, generate data warnings, and prompt staff to handle data exceptions in a timely manner. Improve the data qualification rate and standardization degree of the target tasks to be processed, filter out unqualified data, reduce the amount of data in data quality inspection work, and improve the efficiency of data quality inspection work.

[0057] In one embodiment, S103 may specifically include: based on data quality inspection rules, determining whether there is a second anomaly in the business data of the target task to be processed, the second anomaly being at least one of a matching anomaly and a uniformity anomaly, the matching anomaly being used to indicate that the degree of match between the identification information and the content information of the business data is lower than a preset matching threshold, and the uniformity anomaly being used to indicate that there is an abnormal association between the identification information or the content information of the business data; if there is a second anomaly in the business data of the target task to be processed, determining a second anomaly handling scheme based on the anomaly type of the business data with the second anomaly, adjusting the business data with the second anomaly using the second anomaly handling scheme, and determining first candidate data and abnormal warning data based on the data adjustment result, wherein the first candidate data is data in which the abnormality adjustment is successfully made in the business data with the second anomaly, and the abnormal warning data is data in which the abnormality adjustment fails in the business data with the second anomaly; determining the data to be reported based on the first candidate data and the second candidate data, wherein the second candidate data is business data in which there is no second anomaly in the target task to be processed.

[0058] Among them, the preset matching threshold is a quantitative indicator used to quantify the correlation between the identification information and content information of the business data, and is the data support for matching and screening work. For example, when the content corresponding to the name is 6 or more Chinese characters or other forms of text, it is considered that the matching degree between the identification information and the content information of the business data is lower than the preset matching threshold. If the data indicates that business data 1 is generated by A and that business data 1 is generated by B, it is considered that there is an abnormal correlation between the identification information or content information of the business data. The second exception handling scheme is the processing rule for the second exception. The second exception handling scheme includes a matching adjustment scheme and a correlation adjustment scheme. The matching adjustment scheme is used to adjust data with matching anomalies, and the correlation adjustment scheme is used to adjust data with uniformity anomalies. It is worth noting that not all abnormal data can be calibrated. Abnormalities that are not set in the rules cannot be processed. Only abnormal reports can be recorded and displayed. The data that is successfully calibrated is the first candidate data, and the data that is unsuccessfully calibrated is abnormal warning data. For example, if the data indicates that business data 1 is generated by A and also indicates that business data 1 is generated by B, no adjustment can be made. If the data indicates that business data 1 is generated by A at time 1 and also indicates that business data 2 is generated by A at time 1, they can be summarized and it is considered that A generated both business data 1 and business data 2 at time 1.

[0059] Further, the process of the first candidate data and abnormal warning data includes: based on the data matching quality inspection rule, determining whether there is matching abnormal data for the target task to be processed, wherein the matching degree between the identification information of the matching abnormal data and the content information is lower than the preset matching threshold; if there is matching abnormal data for the target task to be processed, adjusting the matching abnormal data based on the preset matching abnormality response plan, and determining the third candidate data and the first abnormal data based on the data adjustment result, wherein the third candidate data is the business data with successful matching abnormality adjustment, and the first abnormal data is the business data with failed matching abnormality adjustment; based on the data uniformity quality inspection rule, determining whether there is uniform abnormal data for the target task to be processed, wherein the identification information or content information of the uniform abnormal data has abnormal association; if there is uniform abnormal data for the target task to be processed, adjusting the uniform abnormal data based on the preset uniform abnormality response plan, and determining the fourth candidate data and the second abnormal data based on the data adjustment result, wherein the fourth candidate data is the business data with successful uniform abnormality adjustment, and the second abnormal data is the business data with failed uniform abnormality adjustment. Specifically, the third candidate data and the fourth candidate data together constitute the first candidate data, and the first abnormal data and the second abnormal data together constitute the abnormal warning data.

[0060] It is worth noting that when reporting and recording reported data, the present invention will determine whether the data is new data. If it is new data, a new data reporting task will be established. If it is not new data, data update work will be performed based on historical data.

[0061] Figure 3 It is a schematic diagram of the consumption and reporting process of a task to be processed provided by the present invention, showing a completion process of data consumption and reporting. The process can be specifically summarized into two parts. The first part is: the data consumption and reporting service consumes the data in the message queue and sends it to the data quality inspection service, so that the data quality inspection service performs quality inspection on the data according to the data quality inspection rules set in advance. The second part is: the data consumption and reporting service sends the data that has passed the quality inspection of the data quality inspection service to the data receiving platform. At the same time, the interactive interface of the data quality inspection service can also display the results and abnormal data to prompt the staff to handle the abnormality.

[0062] The data consumption and reporting service of the present invention is a backend service program developed based on the Java language, and its main functions are: 1) consuming the data synchronized by Flink cdc in the Kafka message queue; 2) calling the data quality inspection service to perform quality inspection on the aggregated data; 3) reporting the data that has passed the quality inspection to the national platform. The specific implementation process is: 1) writing a data change monitoring method for the data consumer class, monitoring the topic of the Kafka message queue, and consuming the data synchronized by flink cdc; 2) calling the data quality inspection interface to perform quality inspection on the consumed data; 3) writing a data reporting program to report the data that has passed the quality inspection to the national platform.

[0063] The data inspection service of the present invention is a backend service program developed based on Java language, and its main functions are: 1) to allow operation administrators to dynamically configure data quality inspection rules; 2) to provide an open data quality inspection interface to perform quality inspection on data before reporting. The specific implementation process is: 1) to write the compilation method of the quality inspection rule parsing class, and add data quality inspection rules with custom names; 2) to write an open data quality inspection interface for data consumption and reporting services to call, and perform quality inspection on data before reporting.

[0064] In the present invention, data reporting work at different stages is performed by different devices. For example, business data is reported by external systems, rules are configured in advance by staff, information is detected by data quality inspection services, and information is received by data receiving platforms. Different execution entities have different levels. Specifically, Figure 4 It is a flow chart of another data reporting method provided by the present invention, which is a complete data reporting process, including data acquisition, data preprocessing, data delivery, data consumption, data quality inspection and data reporting. The data reporting process specifically includes:

[0065] 1. The staff configures the data quality inspection rules and sends them to the provincial data quality inspection service so that the provincial quality inspection service can store the data quality inspection rules.

[0066] 2. Provincial quality inspection service will provide staff with feedback on configuration results.

[0067] The configuration results are fed back through the interactive interface, and the configuration results include but are not limited to the rule configuration status and rule storage status.

[0068] 3. The municipal external system pushes data to the provincial data aggregation service.

[0069] It is worth noting that the number of external systems that have data interaction relationships with the provincial data aggregation service is at least one.

[0070] 4. The provincial data aggregation service saves the data to the provincial business database.

[0071] 5. The provincial business database feeds back the push results (i.e. data preservation results) to the provincial data aggregation service.

[0072] 6. The provincial data aggregation service will feed back and push results to the municipal external system.

[0073] 7. The provincial data exchange service monitors the log changes of the business database and extracts data.

[0074] It is noteworthy that the monitoring work of the present invention supports data call back and re-extraction.

[0075] 8. Provincial data exchange services perform data conversion and type determination on the extracted data.

[0076] The operation type includes at least one of adding, updating and deleting.

[0077] 9. The provincial data exchange service delivers data to the provincial message queue.

[0078] 10. The provincial message queue returns the delivery result to the provincial data exchange service.

[0079] 11. Provincial data consumption and reporting services consume data in provincial message queues.

[0080] 12. The provincial data consumption and reporting service sends the data to the provincial data quality inspection service for data quality inspection.

[0081] 13. The provincial data quality inspection service will feedback the quality inspection results to the provincial data consumption and reporting service.

[0082] 14. Provincial data consumption and reporting services determine the data reporting type.

[0083] Among them, data reporting types include but are not limited to addition and update.

[0084] 15. Provincial data consumption and reporting services report data to the national data receiving platform.

[0085] 16. The national data receiving platform will feedback the reporting results to the provincial data consumption and reporting service.

[0086] 17. Provincial data consumption and reporting services save data reporting results.

[0087] 18. The provincial data consumption and reporting service sends an indication to the provincial consumption queue to confirm data consumption.

[0088] The advantages of the technical solution of the present invention are: 1) It solves the problem of repeated reporting of the same data caused by data call-back and re-reporting, and meets the data reporting accuracy requirements of the national platform; 2) It adds a data quality inspection link, intercepts data that does not meet the quality requirements, and feeds back to the data source system for rectification, thereby improving the quality of reported data and meeting the comprehensiveness and accuracy requirements of data reporting of the national platform; 3) It refines data in multiple links, improves data reporting efficiency, realizes quasi-real-time reporting, solves the problem of easy data accumulation, and meets the timeliness requirements of data reporting of the national platform.

[0089] The technical solution of this embodiment uses the business data delivery rules to screen and sort the initial business data, obtains at least one task to be processed and delivers it to the pending message queue, reduces the amount of business data that needs to be reported, improves the data delivery efficiency, and reduces the probability of data accumulation; it also matches the tasks that need to be reported from the pending message queue according to the data processing instructions, detects them using the data quality inspection rules, filters out abnormal data that cannot be calibrated, adjusts the abnormal data that can be calibrated, and then reports the business data after quality inspection to the data receiving device, which improves the data reporting efficiency and greatly guarantees the data reporting quality. It solves the problems of the method of querying and determining and reporting the query results in the self-incrementing ID sorting structure according to the business type, low data query efficiency, slow data reporting rate, easy to cause data accumulation, and limited comprehensiveness and real-time performance of data reporting.

[0090] Figure 5 Schematic diagram of the structure of a data reporting device provided by the present invention. Figure 5 As shown, the device includes: a data acquisition module 201, a task determination module 202 and a data reporting module 203.

[0091] The data acquisition module 201 is used to determine the initial business data, process the initial business data using the pre-set business data delivery rules, obtain at least one task to be processed, and deliver the at least one task to be processed to the message queue to be processed;

[0092] The task determination module 202 is used to determine a target to-be-processed task from at least one to-be-processed task included in the to-be-processed message queue based on the received data processing instruction;

[0093] The data reporting module 203 is used to process the target task to be processed using pre-set data quality inspection rules, obtain the data to be reported, and send the data to be reported to the data receiving device, wherein the data to be reported is the business data that has passed the quality inspection in the target task to be processed.

[0094] Optionally, the data acquisition module 201 is specifically used to determine initial business data based on operating data of at least one external system; use pre-set data conversion rules to process the initial business data to obtain intermediate business data, wherein the data volume of the intermediate business data is equal to or less than the data volume of the initial business data, and the intermediate business data includes at least one sub-intermediate business data; use business data delivery rules to process the intermediate business data to obtain at least one task to be processed, wherein the task to be processed includes at least one sub-intermediate business data.

[0095] Optionally, the data reporting module 203 is also used to determine whether there is a first anomaly in the business data of the target processing task before processing the target task to be processed using pre-set data quality inspection rules, wherein the first anomaly includes at least one of data duplication, missing data key information, data format error and data information deviation; if there is a first anomaly in the business data of the target processing task, a first exception handling plan is determined based on the anomaly type of the business data with the first anomaly, the business data with the first anomaly is processed using the first exception handling plan, and the target task to be processed is updated according to the data processing result.

[0096] Optionally, the data reporting module 203 is specifically used to determine whether there is a second anomaly in the business data of the target task to be processed based on data quality inspection rules, wherein the data quality inspection rules include at least one of data matching quality inspection rules and data unification quality inspection rules, and the second anomaly is at least one of a matching anomaly and a uniformity anomaly, the matching anomaly is used to indicate that the matching degree between the identification information and the content information of the business data is lower than a preset matching threshold, and the uniformity anomaly is used to indicate that there is an abnormal association between the identification information or the content information of the business data; if there is a second anomaly in the business data of the target task to be processed, a second anomaly handling scheme is determined based on the anomaly type of the business data with the second anomaly, the business data with the second anomaly is adjusted using the second anomaly handling scheme, and first candidate data and abnormal warning data are determined based on the data adjustment result, wherein the first candidate data is data in which the abnormality adjustment is successfully performed in the business data with the second anomaly, and the abnormal warning data is data in which the abnormality adjustment fails in the business data with the second anomaly; based on the first candidate data and the second candidate data, the data to be reported is determined, wherein the second candidate data is business data in which no second anomaly exists in the target task to be processed.

[0097] Optionally, the task determination module 202 is specifically used to determine whether the data acquisition time in the data processing indication meets the log data storage conditions; if the data acquisition time meets the log data storage conditions, then matching is performed in at least one pending task based on the data acquisition time and the data acquisition identifier in the data processing indication, and determining the target pending task based on the matching result; if the data acquisition time does not meet the log data storage conditions, then updating at least one pending task based on the data acquisition time and the data acquisition identifier, matching is performed in at least one updated pending task using the data acquisition time and the data acquisition identifier, and determining the target pending task based on the matching result.

[0098] Optionally, the task determination module 202 is specifically used to search in the storage location of the initial business data based on the data acquisition time and the data acquisition identifier to obtain the first business data to be updated; determine whether the processing level indicated by the data processing meets the single processing condition; if the processing level meets the single processing condition, use the business data delivery rule to process the first business data to be updated to obtain at least one updated task to be processed; if the processing level does not meet the single processing condition, use the business data delivery rule to process the first business data to be updated and the initial business data to obtain at least one updated task to be processed.

[0099] The data reporting device provided in this embodiment can execute the data reporting method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] Figure 6 : is a schematic diagram of the structure of an electronic device provided by the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0101] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (also called random access memory, RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0102] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0103] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a data reporting method.

[0104] In some embodiments, the data reporting method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data reporting method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data reporting method in any other appropriate manner (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0107] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0109] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server) or a computing system that includes middleware components (e.g., an application server) or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein) or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0110] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0111] In one embodiment, the present invention further includes a computer program product. The computer program product includes a computer program. When the computer program is executed by a processor, the data reporting method of any embodiment of the present invention is implemented.

[0112] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0114] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data reporting method, characterized in that: include: Determine initial business data, process the initial business data using a preset business data delivery rule to obtain at least one task to be processed, and deliver the at least one task to be processed to a message queue to be processed; Based on the received data processing instruction, determining a target to-be-processed task from the at least one to-be-processed task included in the to-be-processed message queue; The target task to be processed is processed using pre-set data quality inspection rules to obtain the data to be reported, and the data to be reported is sent to a data receiving device, wherein the data to be reported is the business data that has passed the quality inspection in the target task to be processed.

2. The method according to claim 1, characterized in that The determining of the initial business data, and processing the initial business data using a preset business data delivery rule to obtain at least one task to be processed, includes: Determining the initial business data based on operation data of at least one external system; Using a preset data conversion rule, the initial business data is processed to obtain intermediate business data, wherein the data volume of the intermediate business data is equal to or less than the data volume of the initial business data, and the intermediate business data includes at least one sub-intermediate business data; The intermediate business data is processed using the business data delivery rule to obtain the at least one task to be processed, wherein the task to be processed includes at least one sub-intermediate business data.

3. The method according to claim 1, characterized in that: Before processing the target task to be processed using the pre-set data quality inspection rules, the following steps are also included: Determine, by using the data quality inspection rule, whether the business data of the target processing task has a first anomaly, wherein the first anomaly includes at least one of data duplication, missing key data information, data format error, and data information deviation; If the business data of the target processing task has the first exception, a first exception handling scheme is determined based on the exception type of the business data with the first exception, the business data with the first exception is processed using the first exception handling scheme, and the target task to be processed is updated according to the data processing result.

4. The method according to claim 1, characterized in that: The method of using the pre-set data quality inspection rules to process the target task to be processed and obtain the data to be reported includes: Based on the data quality inspection rule, determine whether the business data of the target task to be processed has a second anomaly, wherein the data quality inspection rule includes at least one of a data matching quality inspection rule and a data uniformity quality inspection rule, and the second anomaly is at least one of a matching anomaly and a uniformity anomaly, wherein the matching anomaly is used to indicate that the matching degree between the identification information and the content information of the business data is lower than a preset matching threshold, and the uniformity anomaly is used to indicate that the identification information or the content information of the business data has an abnormal association; If the business data of the target task to be processed has the second anomaly, a second anomaly handling scheme is determined based on the anomaly type of the business data having the second anomaly, the business data having the second anomaly is adjusted using the second anomaly handling scheme, and first candidate data and abnormal warning data are determined based on the data adjustment result, wherein the first candidate data is data in which the abnormality adjustment is successfully performed in the business data having the second anomaly, and the abnormal warning data is data in which the abnormality adjustment is failed in the business data having the second anomaly; The data to be reported is determined based on the first candidate data and the second candidate data, wherein the second candidate data is business data in the target task to be processed that does not contain the second anomaly.

5. The method according to claim 1, characterized in that The determining, based on the received data processing instruction, a target to-be-processed task from the at least one to-be-processed task included in the to-be-processed message queue comprises: Determining whether the data acquisition time in the data processing instruction meets the log data storage condition; If the data acquisition time meets the log data storage condition, matching is performed on the at least one to-be-processed task based on the data acquisition time and the data acquisition identifier in the data processing instruction, and determining the target to-be-processed task according to the matching result; If the data acquisition time does not meet the log data storage conditions, the at least one pending task is updated based on the data acquisition time and the data acquisition identifier, and the data acquisition time and the data acquisition identifier are used to match the at least one updated pending task, and the target pending task is determined based on the matching result.

6. The method according to claim 5, characterized in that The updating of the at least one pending task based on the data acquisition time and the data acquisition identifier comprises: Searching the storage location of the initial business data based on the data acquisition time and the data acquisition identifier to obtain the first business data to be updated; determining whether the processing level of the data processing instruction meets the single processing condition; If the processing level meets the single processing condition, the first to-be-updated business data is processed using the business data delivery rule to obtain the updated at least one to-be-processed task; If the processing level does not meet the single processing condition, the first to-be-updated business data and the initial business data are processed using the business data delivery rule to obtain the updated at least one to-be-processed task.

7. A data reporting device, characterized in that: Used to implement the data reporting method described in any one of claims 1 to 6, the data reporting device comprises: A data acquisition module, used to determine initial business data, process the initial business data using a preset business data delivery rule, obtain at least one task to be processed, and deliver the at least one task to be processed to a message queue to be processed; A task determination module, configured to determine a target to-be-processed task from the at least one to-be-processed task included in the to-be-processed message queue based on the received data processing instruction; The data reporting module is used to process the target task to be processed using pre-set data quality inspection rules, obtain the data to be reported, and send the data to be reported to the data receiving device, wherein the data to be reported is the business data that has passed the quality inspection in the target task to be processed.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data reporting method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data reporting method described in any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data reporting method according to any one of claims 1 to 6.

Images