A data processing method and device suitable for quasi-real-time service

Abstract

The present invention relates to the field of financial technology (Fintech), and discloses a data processing method and device suitable for quasi-real-time services, which obtains target source data from a data source through a first channel as first data; obtains target source data through a second channel as second data; compares the first data of a first time period with the second data of the time period to determine whether the two meet a preset consistency standard; if they meet, continues to compare the first data of the second time period with the second data of the time period. By comparing the first data and the second data of each time period, the scheme promptly finds out whether the first data as external data is abnormal. If normal, the response time of reliable quasi-real-time services provided by a distributed database to the outside is greatly shortened. In the later stage, the target source data of the time period does not need to be processed twice by the second channel, which avoids repeated writing of the data of the time period in the distributed database, saving writing overhead.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of financial technology (Fintech), and in particular, to a data processing method and device suitable for quasi-real-time services. Background Art

[0002] With the development of computer technology, more and more technologies (such as big data, cloud computing or blockchain) are being applied in the financial field. Traditional finance is gradually transforming into financial technology, and big data technology is no exception. However, due to the security and real-time requirements of the financial and payment industries, higher requirements are also placed on big data technology.

[0003] With the development of the Internet and the richness of application scenarios, the amount of data related to financial business is increasing. Generally speaking, a large amount of data related to financial business relies on relational databases, but relational databases support limited application scenarios and cannot support quasi-real-time services for external systems in many scenarios.

[0004] In recent years, with the emergence of a large number of open source software, it has become easier to process massive amounts of data in near real time, and therefore it has supported more and more near real-time services. Among them, Hadoop, a distributed system infrastructure developed by the Apache Software Foundation, has components in its ecosystem that support both batch processing and near real-time processing of massive amounts of data at a low cost, and has become an essential infrastructure for financial technology companies. Figure 1 As shown in FIG. 1 , a typical data processing method for quasi-real-time services provided by the prior art is shown, and the method is mainly executed by a series of components in the Hadoop ecosystem. Figure 1 :

[0005] T-day data: Kafka continuously receives and stores source data. Spark Streaming periodically obtains source data from Kafka, performs calculations based on business logic, and writes the calculated data to HBase.

[0006] Data on T+1 day: After the source data is extracted through Sqoop, it is batch processed by Hive and then written into HBase, overwriting the data of the previous day in HBase.

[0007] In the above solution, on the one hand, due to the inherent characteristics of open source components such as Kafka and Spark Streaming, the components cannot meet the requirements of data integrity. In addition, the above open source components need to cooperate with each other. Since various exceptions are prone to occur in the actual cooperation process, the adverse result of not being able to ensure data integrity is brought about; on the other hand, since Sqoop and Hive are based on the MR mechanism and cannot support quasi-real-time operations, the integrity of the T-day data can only be guaranteed by fully covering the T-day data on T+1 day.

[0008] In summary, the existing technology for providing quasi-real-time services for financial data has the problem of insufficient reliability. Summary of the invention

[0009] The present invention provides a data processing method and device suitable for quasi-real-time service, so as to solve the problem of insufficient reliability of existing quasi-real-time service.

[0010] In a first aspect, an embodiment of the present invention provides a data processing method suitable for quasi-real-time services, the method comprising: obtaining target source data as first data from a data source through a first channel at a first time granularity, and storing the first data in a distributed database; the distributed database is used to provide quasi-real-time services; obtaining the target source data as second data from the data source through a second channel at a second time granularity; the second time granularity is greater than the first time granularity; comparing first data of a first time period with second data of the first time period to determine whether the first data of the first time period and the second data of the first time period meet a preset consistency standard; if so, continuing to compare the first data of the second time period with the second data of the second time period.

[0011] Based on this scheme, for each time period, the target source data of the time period is obtained through the first channel and the second channel respectively, and used as the first data and the second data respectively. By comparing the first data and the second data of the time period, it is timely discovered whether the first data as the external data is abnormal, and when it is determined that the first data of the previous time period is normal, the first data and the second data of the next time period are continuously compared. It can be seen that the prior art covers the data of day T in full on day T+1. On the one hand, the waiting time is long - when the first data is abnormal, the distributed database can only provide reliable quasi-real-time services to the outside world on day T+1. On the other hand, the indiscriminate full coverage of the prior art increases the write operation to the distributed database, increases the pressure on the distributed database, and even affects the response of the distributed database to the quasi-real-time service. In this scheme, by refining the time granularity of the comparison, the response time of the distributed database to provide reliable quasi-real-time services to the outside world is shortened. At the same time, when the comparison is consistent, it is not necessary for the second channel to perform secondary processing on the target source data of the time period, which avoids repeated writing of the data of the time period in the distributed database and saves the write overhead of the distributed database.

[0012] In a possible implementation method, if there is no match, the data in the first data of the first time period that is inconsistent with the second data of the first time period is used as the third data; the third data is reacquired from the data source through the first channel, and the distributed database is updated according to the third data, and the step of comparing the first data of the first time period with the second data of the first time period is returned to execute.

[0013] Based on this solution, when comparing the first data and the second data of each time period, if it is determined that the two do not meet the preset consistency standard, the data supplementation measures are taken in a timely manner: the third data is re-acquired from the data source through the first channel and updated to the distributed database. The third data here refers to the data in the first data of the time period that is inconsistent with the second data. After the data is supplemented, the data of the time period (the original second data and the updated first data) will be compared again until the two (the original second data and the updated first data) meet the preset consistency standard. On the one hand, this method ensures the reliability and timeliness of the distributed database in providing quasi-real-time services to the outside world; on the other hand, for the existing technical solutions that require full coverage of the first day's data on the second day, this method can largely avoid the problem of unnecessary waste of distributed database storage resources.

[0014] In a possible implementation method, before comparing the first data of the first time period with the second data of the first time period, it also includes: determining the data type of the first data of the first time period; wherein the data type is used to indicate the importance of the data; determining the comparison rule of the first data of the first time period according to the data type of the first data of the first time period; wherein the level of the comparison rule is positively correlated with the importance of the data; comparing the first data of the first time period with the second data of the first time period, including: comparing the first data of the first time period with the second data of the first time period according to the comparison rule.

[0015] Based on this solution, the data type of the first data is first determined, and a comparison rule for comparing the first data and the second data is determined based on the data type, which helps to quickly compare the first data in the first period with the second data in the first period, thereby obtaining feedback results and facilitating the implementation of subsequent processes. In practical applications, the accuracy requirements for different types of data are different, so this solution can more flexibly meet the needs of practical applications, while ensuring accuracy while meeting the needs.

[0016] In a possible implementation method, determining the data type of the first data in the first time period includes: if the first data in the first time period includes data of multiple data types, determining the data type of the most important data as the data type of the first data in the first time period.

[0017] Based on this solution, when there are many data types included in the first data of the first period, the data type of the most important data in the first data of the first period is used as the data type of the first data of the first period. It can be understood that the higher the importance of the data, the stricter the comparison rules used in the comparison process. Therefore, by using the data type of the most important data as the data type of the first data of the first period, it means that the most strict comparison rules are used to compare the first data of the first period and the second data of the first period in the later stage. Through this method, the comprehensiveness of the comparison of the first data of the first period and the second data of the first period is guaranteed, and the problem of missed detection of data caused by using lower comparison rules is avoided.

[0018] In a possible implementation method, the comparison rule includes at least one of the following: comparison of the number of data items, comparison of key fields, and comparison of all fields.

[0019] Based on this solution, according to different comparison rules, the first data of the first time period and the second data of the first time period can be compared. After the comparison is consistent, it is determined that the quasi-real-time service provided by the distributed database is reliable. Different comparison rules can be set according to actual needs, including data number comparison rules, key field comparison rules and full field comparison rules. The comparison rules are flexible and diverse, and can meet a variety of scenarios for comparing data.

[0020] In one possible implementation method, target source data is obtained from a data source through a first channel at a first time granularity as first data, including: obtaining the target source data from the data source through a distributed publishing message subscription tool at the first time granularity; calculating the target source data through a real-time stream data processing tool to obtain the first data; obtaining the target source data from the data source through a second channel at a second time granularity as second data, including: obtaining the target source data from the data source through a data transfer tool at the second time granularity; calculating the target source data to obtain the second data.

[0021] Based on this solution, the first data is obtained by obtaining the target source data from the data source through the distributed publishing and subscription system tool according to the first time granularity, and calculating the obtained target source data through the real-time stream data processing tool; the second data is obtained by obtaining the target source data from the data source through the data transfer tool according to the second time granularity, and calculating the target source data. For the target source data of the same time period, when it is obtained by two different channels (the first channel and the second channel), the first data and the second data are formed respectively. The first channel cannot fully guarantee the accuracy of the first data due to its inherent characteristics, while the second channel can 100% guarantee the accuracy of the second data. Then, by comparing the first data and the second data of the same time period, it can be determined whether the first data stored in the distributed database is accurate data or normal data, and after determining that the first data is accurate data or normal data, it can be guaranteed that the quasi-real-time service provided by the distributed database is reliable.

[0022] In a possible implementation method, reacquiring the third data from the data source through the first channel includes: sending the third data in the data source to the distributed publishing and message subscription tool through a notification tool.

[0023] Based on this solution, when the data in the first data of the first time period that is inconsistent with the second data of the first time period is determined, and the determined inconsistent data is used as the third data, the notification tool can send the third data to the distributed publishing and message subscription tool in a targeted manner, that is, re-acquire the third data through the first channel. Compared with re-acquiring the third data from the second channel, the main difference between this method and re-acquiring the third data from the second channel is that: since the second channel only supports batch processing of data, if the second channel is used to re-acquire the third data, it is necessary to process all the second data, and by overwriting the first data in the distributed database with the processed second data, this greatly wastes the write overhead to the distributed database; if the third data is re-acquired through the first channel, it can be targeted, rather than necessarily processed, for the target source data corresponding to the entire first data, thereby avoiding the waste of write overhead to the distributed database to a certain extent.

[0024] In a second aspect, an embodiment of the present invention provides a data processing device suitable for quasi-real-time services, the device comprising: a first data acquisition unit, used to acquire target source data as first data from a data source through a first channel at a first time granularity, and store the first data in a distributed database; the distributed database is used to provide quasi-real-time services; a second data acquisition unit, used to acquire the target source data as second data from the data source through a second channel at a second time granularity; the second time granularity is greater than the first time granularity; a comparison unit, used to compare first data of a first time period with second data of the first time period, to determine whether the first data of the first time period and the second data of the first time period meet a preset consistency standard; the comparison unit is also used to continue comparing the first data of the second time period with the second data of the second time period if they meet.

[0025] Based on this scheme, for each time period, the target source data of the time period is obtained through the first channel and the second channel respectively, and used as the first data and the second data respectively. By comparing the first data and the second data of the time period, it is timely discovered whether the first data as the external data is abnormal, and when it is determined that the first data of the previous time period is normal, the first data and the second data of the next time period are continuously compared. It can be seen that the prior art covers the data of day T in full on day T+1. On the one hand, the waiting time is long - when the first data is abnormal, the distributed database can only provide reliable quasi-real-time services to the outside world on day T+1. On the other hand, the indiscriminate full coverage of the prior art increases the write operation to the distributed database, increases the pressure on the distributed database, and even affects the response of the distributed database to the quasi-real-time service. In this scheme, by refining the time granularity of the comparison, the response time of the distributed database to provide reliable quasi-real-time services to the outside world is shortened. At the same time, when the comparison is consistent, it is not necessary for the second channel to perform secondary processing on the target source data of the time period, which avoids repeated writing of the data of the time period in the distributed database and saves the write overhead of the distributed database.

[0026] In a possible implementation method, the comparison unit is also used to: if there is no match, use the data in the first data of the first time period that is inconsistent with the second data of the first time period as third data; re-acquire the third data from the data source through the first channel, update the distributed database according to the third data, and return to execute the step of comparing the first data of the first time period with the second data of the first time period.

[0027] Based on this solution, when comparing the first data and the second data of each time period, if it is determined that the two do not meet the preset consistency standard, the data supplementation measures are taken in a timely manner: the third data is re-acquired from the data source through the first channel and updated to the distributed database. The third data here refers to the data in the first data of the time period that is inconsistent with the second data. After the data is supplemented, the data of the time period (the original second data and the updated first data) will be compared again until the two (the original second data and the updated first data) meet the preset consistency standard. On the one hand, this method ensures the reliability and timeliness of the distributed database in providing quasi-real-time services to the outside world; on the other hand, for the existing technical solutions that require full coverage of the first day's data on the second day, this method can largely avoid the problem of unnecessary waste of distributed database storage resources.

[0028] In a possible implementation method, the device also includes a determination unit, and before comparing the first data of the first time period with the second data of the first time period, the determination unit is used to: determine the data type of the first data of the first time period; wherein the data type is used to indicate the importance of the data; determine the comparison rule of the first data of the first time period according to the data type of the first data of the first time period; wherein the level of the comparison rule is positively correlated with the importance of the data; the comparison unit is specifically used to: compare the first data of the first time period with the second data of the first time period according to the comparison rule.

[0029] Based on this solution, the data type of the first data is first determined, and a comparison rule for comparing the first data and the second data is determined based on the data type, which helps to quickly compare the first data in the first period with the second data in the first period, thereby obtaining feedback results and facilitating the implementation of subsequent processes. In practical applications, the accuracy requirements for different types of data are different, so this solution can more flexibly meet the needs of practical applications, while ensuring accuracy while meeting the needs.

[0030] In a possible implementation method, the determining unit is specifically configured to: if the first data in the first time period includes data of multiple data types, determine the data type of the data with the highest importance as the data type of the first data in the first time period.

[0031] Based on this solution, when there are many data types included in the first data of the first period, the data type of the most important data in the first data of the first period is used as the data type of the first data of the first period. It can be understood that the higher the importance of the data, the stricter the comparison rules used in the comparison process. Therefore, by using the data type of the most important data as the data type of the first data of the first period, it means that the most strict comparison rules are used to compare the first data of the first period and the second data of the first period in the later stage. Through this method, the comprehensiveness of the comparison of the first data of the first period and the second data of the first period is guaranteed, and the problem of missed detection of data caused by using lower comparison rules is avoided.

[0032] In a possible implementation method, the comparison rule includes at least one of the following: comparison of the number of data items, comparison of key fields, and comparison of all fields.

[0033] Based on this solution, according to different comparison rules, the first data of the first time period and the second data of the first time period can be compared. After the comparison is consistent, it is determined that the quasi-real-time service provided by the distributed database is reliable. Different comparison rules can be set according to actual needs, including data number comparison rules, key field comparison rules and full field comparison rules. The comparison rules are flexible and diverse, and can meet a variety of scenarios for comparing data.

[0034] In a possible implementation method, the first data acquisition unit is specifically used to: obtain the target source data from the data source through a distributed publishing message subscription tool according to the first time granularity; calculate the target source data through a real-time stream data processing tool to obtain the first data; the second data unit is specifically used to: obtain the target source data from the data source through a data transfer tool according to the second time granularity; calculate the target source data to obtain the second data.

[0035] Based on this solution, the first data is obtained by obtaining the target source data from the data source through the distributed publishing and subscription system tool according to the first time granularity, and calculating the obtained target source data through the real-time stream data processing tool; the second data is obtained by obtaining the target source data from the data source through the data transfer tool according to the second time granularity, and calculating the target source data. For the target source data of the same time period, when it is obtained by two different channels (the first channel and the second channel), the first data and the second data are formed respectively. The first channel cannot fully guarantee the accuracy of the first data due to its inherent characteristics, while the second channel can 100% guarantee the accuracy of the second data. Then, by comparing the first data and the second data of the same time period, it can be determined whether the first data stored in the distributed database is accurate data or normal data, and after determining that the first data is accurate data or normal data, it can be guaranteed that the quasi-real-time service provided by the distributed database is reliable.

[0036] In a possible implementation method, the comparison unit is specifically configured to: send the third data in the data source to the distributed publishing and message subscription tool through a notification tool.

[0037] Based on this solution, when the data in the first data of the first time period that is inconsistent with the second data of the first time period is determined, and the determined inconsistent data is used as the third data, the notification tool can send the third data to the distributed publishing and message subscription tool in a targeted manner, that is, re-acquire the third data through the first channel. Compared with re-acquiring the third data from the second channel, the main difference between this method and re-acquiring the third data from the second channel is that: since the second channel only supports batch processing of data, if the second channel is used to re-acquire the third data, it is necessary to process all the second data, and by overwriting the first data in the distributed database with the processed second data, this greatly wastes the write overhead to the distributed database; if the third data is re-acquired through the first channel, it can be targeted, rather than necessarily processed, for the target source data corresponding to the entire first data, thereby avoiding the waste of write overhead to the distributed database to a certain extent.

[0038] In a third aspect, an embodiment of the present invention provides a computing device, including:

[0039] A memory for storing program instructions;

[0040] The processor is used to call the program instructions stored in the memory and execute any method as described in the first aspect according to the obtained program.

[0041] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute any method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 A schematic diagram of a typical data processing method for quasi-real-time services provided by the prior art;

[0044] Figure 2 A data processing method suitable for quasi-real-time services provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a data processing method for a quasi-real-time service provided by an embodiment of the present invention;

[0046] Figure 4A data processing device suitable for quasi-real-time services is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] First, the terms involved in the embodiments of the present invention are briefly introduced. The following is only an example introduction.

[0049] (1) Spark Streaming: A framework that enables high-throughput, fault-tolerant, real-time streaming data processing.

[0050] (2) HBase: A highly reliable, high-performance, column-oriented, scalable distributed storage system.

[0051] (3) Hive: A data warehouse tool used for data extraction, transformation, and loading. It can store, query, and analyze large-scale data stored in Hadoop.

[0052] (4) Kafka: A high-throughput distributed publish-subscribe messaging system.

[0053] (5) Sqoop: An open source tool mainly used to transfer data between Hadoop and traditional databases.

[0054] like Figure 2 As shown, a data processing method applicable to quasi-real-time services provided by an embodiment of the present invention includes the following steps:

[0055] Step 201, obtaining target source data as first data from a data source through a first channel at a first time granularity, and storing the first data in a distributed database; the distributed database is used to provide quasi-real-time services;

[0056] Step 202: acquiring the target source data as second data from the data source through a second channel at a second time granularity; the second time granularity is greater than the first time granularity;

[0057] Step 203, comparing the first data of the first time period with the second data of the first time period to determine whether the first data of the first time period and the second data of the first time period meet a preset consistency standard;

[0058] Step 204: If they match, continue to compare the first data of the second time period with the second data of the second time period.

[0059] Based on this scheme, for each time period, the target source data of the time period is obtained through the first channel and the second channel respectively, and used as the first data and the second data respectively. By comparing the first data and the second data of the time period, it is timely discovered whether the first data as the external data is abnormal, and when it is determined that the first data of the previous time period is normal, the first data and the second data of the next time period are continuously compared. It can be seen that the prior art covers the data of day T in full on day T+1. On the one hand, the waiting time is long - when the first data is abnormal, the distributed database can only provide reliable quasi-real-time services to the outside world on day T+1. On the other hand, the indiscriminate full coverage of the prior art increases the write operation to the distributed database, increases the pressure on the distributed database, and even affects the response of the distributed database to the quasi-real-time service. In this scheme, by refining the time granularity of the comparison, the response time of the distributed database to provide reliable quasi-real-time services to the outside world is shortened. At the same time, when the comparison is consistent, it is not necessary for the second channel to perform secondary processing on the target source data of the time period, which avoids repeated writing of the data of the time period in the distributed database and saves the write overhead of the distributed database.

[0060] In the above step 201, the data source may be a traditional relational database, in which source data of various businesses are stored. Generally speaking, traditional relational databases do not have the performance of providing quasi-real-time services to the outside, while distributed databases usually have the performance of providing quasi-real-time services to the outside. Therefore, the source data in the relational database may be transferred to the distributed database, so that the distributed database can provide quasi-real-time services to the outside to meet customer needs. For example, a commonly used distributed database may be HBase in the Hadoop ecosystem, that is, HBase can support the provision of quasi-real-time services to the outside.

[0061] According to the following method: target source data is obtained from a data source through a first channel at a first time granularity as first data, and the first data is stored in a distributed database, so that source data in a relational database can be transferred to a distributed database.

[0062] It can be understood that, since the target source data in the relational database is transferred to the distributed database, and the distributed database provides quasi-real-time services to the outside, the first time granularity here is used to represent real-time, that is, the source data in the relational database is transferred to the distributed database in real time: the source data related to various businesses recorded in real time in the relational database, that is, the target source data, can be transferred to the distributed database in real time through the first channel, thereby realizing the quasi-real-time service provided by the distributed database to the outside.

[0063] It should be noted that the first data is not completely equivalent to the target source data itself. In addition to the target source data itself, the first data may also include derivative data generated by data processing of the target source data; as for what kind of data processing needs to be done on the target source data, it is flexibly set by business personnel based on accurate understanding of customer needs, and there is no specific limitation.

[0064] How to convert the target source data recorded in the relational database into the first data as external data and stored in the distributed database can be achieved through the first channel:

[0065] Optionally, target source data is obtained from a data source through a first channel at a first time granularity as the first data, including: obtaining the target source data from the data source through a distributed publishing message subscription tool at the first time granularity; and calculating the target source data through a real-time stream data processing tool to obtain the first data.

[0066] In the embodiment of the present invention, the components in the Hadoop ecosystem cooperate with each other to provide quasi-real-time services to the outside world, which will not be described in detail below.

[0067] For example, Kafka, a distributed publishing and subscription tool, receives target source data from a relational database in real time, where the target source data consists of various fields and the values ​​corresponding to the fields; Spark Streaming, a real-time stream data processing tool, periodically obtains target source data from Kafka. When some fields in the target source data require business logic calculations, the target source data corresponding to these fields are calculated to obtain the first data, and the calculated first data is written to HBase. In addition, for fields in the target source data that do not require business logic calculations, the target source data corresponding to these fields can be directly used as the first data and written to HBase.

[0068] In the above step 202, the target source data is obtained from the data source through a second channel at a second time granularity as the second data; the second time granularity is greater than the first time granularity.

[0069] Since the first time granularity in the embodiment of the present invention is used to represent real-time performance, the second time granularity in the embodiment of the present invention can be 1 hour, that is, the target source data is obtained once every 1 hour, and the obtained target source data is used as the second data; the second time granularity in the embodiment of the present invention can also be other durations slightly longer than 1 hour, but it should not be too long, because if the second time granularity is too long, it will greatly cause the distributed database to have a long delay when providing quasi-real-time services to the outside world, thereby affecting the customer experience, which is not desirable. The second time granularity in the embodiment of the present invention can also be other durations slightly shorter than 1 hour, but it should not be too short, because if the second time granularity is too short, it is not practical in actual application, but a waste of computing resources. The second time granularity is not specifically limited in the embodiment of the present invention, but is only used as an example. The second time granularity of 1 hour is used as an example for explanation in the embodiment of the present invention, and will not be repeated below.

[0070] How to convert the target source data recorded in the relational database into the second data can be achieved through the second channel:

[0071] Optionally, obtaining the target source data as the second data from the data source through a second channel at a second time granularity includes: obtaining the target source data from the data source through a data transfer tool at the second time granularity; and calculating the target source data to obtain the second data.

[0072] For example, when the second time granularity is 1 hour, the data transfer tool Sqoop will extract the target source data stored in the relational database at a frequency of obtaining data once an hour, that is, the amount of data obtained by the data transfer tool Sqoop each time is the target source data recorded in the relational database within the past hour; then, for the fields in the target source data that require business logic calculations, the target source data corresponding to these fields are calculated through the Spark component, and the data warehouse tool Hive stores the calculated second data; for the fields in the target source data that do not require business logic calculations, the target source data corresponding to these fields are directly used as the second data and stored in the data warehouse tool Hive. Through the above process, the target source data in the relational database can be transferred periodically and in batches.

[0073] In the above step 203, the first data of the first time period is compared with the second data of the first time period to determine whether the first data of the first time period and the second data of the first time period meet a preset consistency standard.

[0074] For example, when the second time granularity is 1 hour, in the embodiment of the present invention, the first time period can be set to be 1 hour, or the first time period can be set to be another integral multiple of 1 hour, without limitation. As an example, the embodiment of the present invention is described by taking the first time period as 1 hour as an example.

[0075] Before comparing the first data of the first period with the second data of the first period, the target source data is periodically obtained from Kafka through the real-time stream data processing tool Spark Streaming, for example, the target source data within the hour from 9 am to 10 am is obtained, and the target source data within the period is processed to obtain the first data; at the same time, the target source data within the hour from 9 am to 10 am, which is stored in the relational database, is extracted through the data transfer tool Sqoop. According to actual needs, the target source data is processed or not processed to obtain the second data, which will be stored in the data warehouse tool Hive. Since the second data is stored in the data warehouse tool Hive, the data warehouse tool Hive will then compare the first data of the period with the second data of the period.

[0076] Optionally, before comparing the first data of the first time period with the second data of the first time period, it also includes: determining a data type of the first data of the first time period; wherein the data type is used to indicate the importance of the data; determining a comparison rule for the first data of the first time period according to the data type of the first data of the first time period; wherein the level of the comparison rule is positively correlated with the importance of the data; comparing the first data of the first time period with the second data of the first time period, includes: comparing the first data of the first time period with the second data of the first time period according to the comparison rule.

[0077] Relational databases can be used to store various types of data, such as log data, customer personal information data, transaction details data, and other types of data. Considering that the importance of various types of data is not completely consistent: some types of data are relatively important, while others are relatively less important. Therefore, it is of great practical significance to use different comparison rules for data of different importance.

[0078] For example, there are three types of data stored in a relational database, which can include logs, customer personal information, and transaction details. After long-term business processing, business personnel found that the importance of these three types of data is not the same, which can be specifically manifested as follows: the importance of transaction details data is the greatest, the importance of customer personal information data is medium, and the importance of log data is not very important. Therefore, the importance of transaction details data can be set to "very important", the importance of customer personal information data can be set to "generally important", and the importance of log data can be set to "not very important".

[0079] Determining the data type of the first data in the first time period means determining the importance of the first data in the first time period. The data type here is used to indicate the importance of the data.

[0080] For example, when the first data in the first time period is transaction details data, its importance is "very important", which means that the data type of the first data in the first time period is "very important"; if the first data in the first time period is customer personal information data, its importance is "generally important", which means that the data type of the first data in the first time period is "generally important"; if the first data in the first time period is log data, its importance is "not very important", which means that the data type of the first data in the first time period is "not very important".

[0081] The above idea is to classify the data type of the first data in the first period into different levels, and the levels can be defined as "very important", "generally important" and "not very important" in descending order. Of course, the data types can also be classified according to actual needs, which is not limited by the present invention.

[0082] If it is determined that the data type of the first data in the first period is "very important", then the comparison rule corresponding to the data type can be determined and set as comparison rule 1; if it is determined that the data type of the first data in the first period is "generally important", then the comparison rule corresponding to the data type can be determined and set as comparison rule 2; if it is determined that the data type of the first data in the first period is "not very important", then the comparison rule corresponding to the data type can be determined and set as comparison rule 3. Among them, comparison rule 1, comparison rule 2 and comparison rule 3 are different from each other.

[0083] It should be noted that the correspondence between data types and matching rules is pre-set. That is, when the data type is "very important", the matching rule corresponding to it is matching rule 1, not matching rule 2 or matching rule 3; when the data type is "generally important", the matching rule corresponding to it is matching rule 2, not matching rule 1 or matching rule 3; when the data type is "not very important", the matching rule corresponding to it is matching rule 3, not matching rule 1 or matching rule 2.

[0084] It is understandable that when the data types are sorted from high to low in terms of level as "very important", "generally important" and "not very important", the comparison rules are also sorted from high to low in terms of level as: comparison rule 1, comparison rule 2 and comparison rule 3. Here, comparison rule 1, comparison rule 2 and comparison rule 3 are all general concepts, which are described in detail below, but it is understandable that comparison rule 1 is the most stringent, comparison rule 2 is the second most stringent, and comparison rule 3 is the broadest.

[0085] Finally, the first data of the first time period and the second data of the first time period are compared according to the determined comparison rule. If the data type of the first data of the first time period is determined to be "very important", and its comparison rule is further matched to be comparison rule 1, then only the first data of the first time period and the second data of the first time period need to be compared according to comparison rule 1.

[0086] It should be noted that in the process of determining the data type of the first data in the first period, it can be understood that the matching relationship between the data and the data type will be set in advance. Therefore, when the first data in the first period is received, the data type of the first data in the first period can be easily determined based on the preset matching relationship between the data and the data type. For example, when the first data in the first period is a log, its data type can be considered "not very important"; if the first data in the first period is the customer's personal information, its data type can be considered "generally important"; if the first data in the first period is transaction details data, its data type can be considered "very important".

[0087] In the actual process of comparing the first data of the first period with the second data of the first period, the first data is usually classified as a single data type, for example, the data type of the first data of the first period is any one of "very important", "generally important" and "not very important". Then, the first data of the first period and the second data of the first period only need to be compared according to the comparison rule that matches the data type of the first data of the first period.

[0088] However, in actual application, the first data of the first period may include data of multiple data types. For this problem, the data type of the first data of the first period can be determined in the following manner:

[0089] Optionally, determining the data type of the first data in the first time period includes: if the first data in the first time period includes data of multiple data types, determining the data type of data with the highest importance as the data type of the first data in the first time period.

[0090] For example, the first data of the first period is relatively complex. When referring to the matching relationship between the pre-set data and the data type, it is found that the first data of the first period is as follows: part of the first data of the first period meets the data type of "very important". For the convenience of description, the part of the data that meets the data type of "very important" is recorded as data A; at the same time, part of the first data of the first period meets the data type of "not very important". For the convenience of description, the part of the data that meets the data type of "very important" is recorded as data B. For this situation, the embodiment of the present invention proposes to use the data type of the data with the highest degree of importance as the data type of the first data of the first period. Since the importance of data A is greater than that of data B, the embodiment of the present invention proposes to use the data type of data A as the data type of the first data of the first period, that is, the data type of the first data of the first period is "very important". Therefore, when the first data of the first period and the second data of the first period are compared later, the comparison rule corresponding to the data type of "very important" is used to compare the first data of the period with the second data.

[0091] When comparing the first data of the first time period with the second data of the first time period, the comparison will be performed according to different comparison rules.

[0092] Optionally, the comparison rule includes at least one of the following: comparison of the number of data items, comparison of key fields, and comparison of all fields.

[0093] Due to the different data types, that is, the different importance of the data, naturally, when comparing the first data of the first period with the second data of the first period, the comparison rules used are also different. In other words, one data type corresponds to one comparison rule.

[0094] Therefore, in the aforementioned example, for the data type of "very important", it means that the data has the greatest importance and the highest level, so when the first data of the first period meets the data type of "very important", when comparing the first data of the first period with the second data of the first period, the comparison rule of full field comparison can be adopted, and the comparison rule of full field comparison here is equivalent to the aforementioned comparison rule 1;

[0095] The data type of "general importance" means that the importance of the data is medium and the level is low. Therefore, when the first data of the first period meets the data type of "general importance", when comparing the first data of the first period with the second data of the first period, the comparison rule of key field comparison can be used. The comparison rule of key field comparison here is equivalent to the aforementioned comparison rule 2;

[0096] For the data type of "not very important", it means that the data is not very important and has the lowest level. Therefore, when the first data of the first time period meets the data type of "not very important", when comparing the first data of the first time period with the second data of the first time period, the comparison rule of comparing the number of data items can be used. The comparison rule of comparing the number of data items here is equivalent to the aforementioned comparison rule 3.

[0097] In the process of comparing the first data of the first time period with the second data of the first time period, if the comparison rule of comparing the number of data items is used, the preset consistency standard can be understood as: the number of data items of the first data of the first time period is the same as the number of data items of the second data of the first time period.

[0098] For example, when it is determined that the data type of the first data in the first time period is "not very important", the corresponding comparison rule is the comparison of the number of data items: if and only if the number of data items of the first data in the first time period is the same as the number of data items of the second data in the first time period, it can be determined that the comparison result of the first data in the first time period and the second data in the first time period is consistent; otherwise, it is determined that the comparison result of the first data in the first time period and the second data in the first time period is inconsistent.

[0099] The comparison rule of key field comparison includes two meanings: the comparison of the key field itself and the comparison of the value corresponding to the key field. Therefore, in the process of comparing the first data of the first period with the second data of the first period, if the comparison rule of key field comparison is used, the preset consistency standard can be understood as: the key field of the first data of the first period and the value corresponding to the key field are completely consistent with the key field of the second data of the first period and the value corresponding to the key field.

[0100] For example, when it is determined that the data type of the first data in the first period is "generally important", the corresponding comparison rule is key field comparison. During the comparison process, the following two situations may occur:

[0101] (1) There are five key fields in the second data of the first period, namely A1, A2, A3, A4 and A5. At the same time, there are five fields in the first data of the first period, namely A1, A2, A3, A4 and A6. Since field A6 is different from key field A5, it can be determined that the comparison result between the first data of the first period and the second data of the first period is inconsistent.

[0102] (2) There are five key fields in the second data of the first period, namely A1, A2, A3, A4 and A5. At the same time, there are five fields in the first data of the first period, namely A1, A2, A3, A4 and A5. This means that the preliminary comparison result between the first data of the first period and the second data of the first period is consistent. Next, continue to compare the values ​​corresponding to the key fields. If and only if the values ​​corresponding to the five fields of each record of the first data of the first period match the values ​​corresponding to the five key fields of the record at the same position of the second data of the first period, it can be determined that the comparison result between the first data of the first period and the second data of the first period is consistent. Otherwise, the comparison result is still inconsistent.

[0103] In the process of comparing the first data of the first time period with the second data of the first time period, if the comparison rule of the full field comparison is used, the preset consistency standard can be understood as: the full field of the first data of the first time period and the value corresponding to the full field are completely consistent with the full field of the second data of the first time period and the value corresponding to the full field.

[0104] For example, when it is determined that the data type of the first data of the first time period is "very important", the corresponding comparison rule is full-field comparison: if and only if all the fields of each record of the first data of the first time period and the values ​​corresponding to all the fields match all the fields of the record at the same position of the second data of the first time period and the values ​​corresponding to all the fields one by one, it can be determined that the comparison result of the first data of the first time period and the second data of the first time period is consistent; otherwise, the comparison result is inconsistent.

[0105] In the above step 204 , if they match, the first data of the second time period is compared with the second data of the second time period.

[0106] For example, let's assume that the hour from 9:00 a.m. to 10:00 a.m. is the first period. By comparing the first data and the second data of this period, if the comparison result is consistent, then in the second period, we only need to continue to compare the first data of the second period and the second data of the second period. For example, let's assume that the hour from 10:00 a.m. to 11:00 a.m. is the second period. Since the comparison result of the first period is consistent, we only need to compare the first data and the second data of the hour from 10:00 a.m. to 11:00 a.m.

[0107] It should be noted that, in the embodiment of the present invention, only the second time period of 10 am to 11 am is used as an example for description. The description of the second time period can refer to the above description of the first time period, which is not repeated here.

[0108] When comparing the first data of the first time period with the second data of the first time period, the comparison result may be inconsistent. For such a problem, the embodiment of the present invention provides the following method:

[0109] Optionally, if there is no match, the data in the first data of the first time period that is inconsistent with the second data of the first time period is used as the third data; the third data is reacquired from the data source through the first channel, and the distributed database is updated according to the third data, and the step of comparing the first data of the first time period with the second data of the first time period is returned to execute.

[0110] As in the above example, let the hour from 9 am to 10 am be the first time period, and by comparing the first data and the second data of this time period, the comparison result obtained is not in conformity; let the hour from 10 am to 11 am be the second time period, then in the second time period, not only will the first data and the second data of this time period be received, but the first data of the first time period whose comparison result is not in conformity will also be supplemented: the data in the first data of the first time period that is inconsistent with the second data of the first time period will be used as the third data; the third data will be re-acquired from the data source through the first channel, and the distributed database will be updated according to the third data. Therefore, in the second time period from 10 am to 11 am, in addition to comparing the first data and the second data of this time period, the first data of the first time period from 9 am to 10 am and the second data of the first time period will be compared again, until the comparison result obtained after comparing the first data of the first time period with the second data of the first time period is in conformity.

[0111] How to retrieve the third data can be done according to the following method:

[0112] Optionally, reacquiring the third data from the data source through the first channel includes: sending the third data in the data source to the distributed publishing and message subscription tool through a notification tool.

[0113] In an embodiment of the present invention, the notification tool may be an application (Application, abbreviated as APP) that transmits information with the data warehouse tool Hive, the relational database, and the distributed publishing and message subscription tool Kafka. When the data warehouse tool Hive determines that the comparison result of the first data of the first period and the second data of the first period do not match, the data warehouse tool Hive sends a request message to the notification tool APP to request the data in the first data of the first period that is inconsistent with the second data of the first period, that is, the third data; when the notification tool APP receives the request message, it will re-acquire the third data from the relational database and send the third data to the distributed publishing and message subscription tool Kafka.

[0114] like Figure 3 FIG. 1 is a schematic diagram of a data processing method for a quasi-real-time service provided by an embodiment of the present invention.

[0115] refer to Figure 3 , let the hour from 9:00 a.m. to 10:00 a.m. be the first period.

[0116] First, on the one hand, the distributed publishing and subscription tool Kafka obtains the target source data of the period from the data source in real time, and the real-time stream data processing tool Spark Streaming periodically calculates the business logic of the target source data in Kafka. For example, at 10:00 a.m., Spark Streaming calculates the target source data of the past hour and stores the calculated first data in the distributed database HBase. On the other hand, the data transfer tool Sqoop can also transfer the target source data stored in the data source in the past hour to the data warehouse tool Hive at 10:00 a.m.

[0117] Next, the data warehouse tool Hive compares the first data and the second data in the first period from 9:00 a.m. to 10:00 a.m., which can be shown as follows:

[0118] According to the pre-set matching relationship between data and data types, if it is determined that the first data meets the data type of "not very important", Hive will compare the first data and the second data of the time period according to the matching rule corresponding to the data type of "not very important". For example, if the first data of the time period is a log, the first data and the second data of the time period can be compared according to the matching rule of the number of data entries;

[0119] According to the pre-set matching relationship between data and data types, if it is determined that the first data meets the data type of "general importance", Hive will compare the first data and the second data of the time period according to the matching rule corresponding to the data type of "general importance". For example, if the first data of the time period is customer personal information, the first data and the second data of the time period can be compared according to the matching rule of key field matching;

[0120] According to the pre-set matching relationship between data and data types, if it is determined that the first data meets the data type of "very important", Hive will compare the first data and the second data of the period according to the matching rule corresponding to the data type of "very important". For example, if the first data of the period is transaction details, the first data and the second data of the period can be compared according to the matching rule of full field matching.

[0121] After the data warehouse tool Hive compares the first data of the first period with the second data of the first period, the comparison result will be recorded. For example, the comparison result can be recorded in a relational database management system, abbreviated as MYSQL. If the comparison result of the first period is consistent, a log with the words "data source, 9:00-10:00, success" will be recorded in MYSQL; if the comparison result of the first period is inconsistent, a log with the words "data source, 9:00-10:00, failure" will be recorded in MYSQL, and a list of data that failed to be compared will be generated at the same time: if it is a comparison rule of full field comparison and key field comparison, it will record which data or which serial numbers failed to be compared; if it is a comparison rule of data number comparison, it will record that all data in the period failed to be compared.

[0122] After the data warehouse tool Hive compares the first data of the first time period with the second data of the first time period and records the comparison result in MYSQL, the application APP will read the comparison result in MYSQL: if the comparison result is consistent, the APP will not take any action; if the comparison result is inconsistent, the APP will read the content of the data list that failed to be compared. If the data list that failed to be compared records which one or which sequence numbers of data failed to be compared, the APP will extract the data corresponding to the sequence numbers that failed to be compared from the source data, transfer it to Kafka, and then Spark Streaming will calculate it, re-store it in the distributed database HBase, and write it to the sequence number corresponding to the time period; if the data list that failed to be compared records that all data in the time period failed to be compared, the APP will re-extract all data in the time period from the source data, transfer it to Kafka, and then Spark Streaming will calculate it and re-store it in the distributed database HBase.

[0123] Finally, in an embodiment of the present invention, assuming that the comparison result of the first time period is not consistent, and assuming that the one hour from 10 a.m. to 11 a.m. is the second time period, then the data warehouse tool Hive will not only compare the first data and the second data of the second time period, but will also compare the first data and the second data of the first time period (9 a.m. to 10 a.m. is the first time period) again, because only when the comparison result of the first data of the first time period and the second data of the first time period is consistent, can the first data about the first time period provided by the distributed database HBase to the outside be reliable data, thereby improving the reliability of quasi-real-time services.

[0124] Next, the solution of the present invention will be described in a specific example.

[0125] Assume that Bank A uses relational database B to record the transfer records of its customers, and further assume that all transfers are made to the outside world by bank cards issued by Bank A. Let 9:00-10:00 am on March 6, 2020 be the first period. On the one hand, Kafka can obtain the transfer records stored in relational database B in real time, and Spark Streaming can calculate the transfer records in Kafka regularly, such as calculating the transfer records of the first period at 10:00 am, and store the calculated first data in the distributed database HBase; on the other hand, Sqoop can periodically send the transfer records stored in relational database B for a period of time that has just passed, that is, the second data (the calculation of the target source data is not considered here, that is, the second data is equivalent to the target source data), to the data warehouse tool Hive, where the period of time that has just passed refers to the 1 hour that has just passed, that is, 9:00-10:00 am. The second data stored in the data warehouse tool Hive can be shown in Table 1, and the specific details can be referred to Table 1.

[0126]

[0127] Next, the data warehouse tool Hive compares the first data and the second data from 9:00 am to 10:00 am.

[0128] Referring to Table 1, there are 10 transaction details in total. Since the data within this period are transaction details, they can be classified as the "very important" data type. Therefore, the data warehouse tool Hive can compare the first data and the second data according to the comparison rules of the full field comparison.

[0129] The comparison rule of the full field comparison requires not only that the fields in the first data of the first period must be consistent with the fields in the second data of the first period, but also that the values ​​corresponding to the fields must be completely consistent.

[0130] It can be understood that, for any one of the 10 transaction details, if its field in the first data of the first time period does not match the field in the second data of the first time period, it can be directly concluded that the comparison result of the first data of the first time period and the second data of the first time period is inconsistent; if its field in the first data of the first time period matches the field in the second data of the first time period, it only means that at the field level, the comparison result of the first data of the first time period and the second data of the first time period is inconsistent. As for the final determination of whether the comparison result of the first data of the first time period and the second data of the first time period is consistent or not, it is also necessary to determine whether the values ​​corresponding to the fields of each of the 10 transaction details match.

[0131] It can be directly concluded that the comparison result of the first data of the first time period and the second data of the first time period is inconsistent.

[0132] Assume that for the transaction details with sequence number 1, the fields involved in the second data are: transaction time, transaction account name, transaction card number, counterparty card number, transaction amount, transaction card balance, accounting currency and transaction country or region abbreviation, and the corresponding values ​​of each field are: 10:01:45, A, 6022*21, 6028*21, 100, 1000, RMB and CHN.

[0133] In the comparison process, for the transaction details with sequence number 1, the fields involved in the first data are required to include at least the transaction time, transaction account name, transaction card number, counterparty card number, transaction amount, transaction card balance, accounting currency and transaction country or region abbreviation. When the fields of the first data can completely match the fields of the second data, it can only be preliminarily indicated that the comparison result of the first data of the first period and the second data of the first period is consistent at the field level; if there is at least one field in the first data that does not match the field of the second data, it means that the comparison result of this period is inconsistent, and the inconsistent comparison result is recorded in MYSQL.

[0134] After the preliminary comparison is passed, for the transaction details with serial number 1, the values ​​corresponding to the fields in the first data are required to be: 10:01:45, A, 6022*21, 6028*21, 100, 1000, RMB and CHN. When the values ​​corresponding to the fields of the first data can completely match the values ​​corresponding to the fields of the second data, it means that for the transaction details with serial number 1, the comparison result in this period is compliant; if there is at least one field corresponding to the value of the field corresponding to the first data that does not match the value corresponding to the field of the second data, it means that for the transaction details with serial number 1, the comparison result in this period is not compliant. For the comparison of data with other serial numbers, you can refer to the comparison process of data with serial number 1, which will not be repeated here.

[0135] When the data warehouse tool Hive has finished comparing all the transaction data of 10 serial numbers in a certain order, if for the data of serial numbers 1 to 10, the value corresponding to each field in the first data is completely consistent with the value corresponding to the same field in the second data, then the comparison result of this period is finally confirmed to be consistent, and this consistent comparison result is recorded in MYSQL; if there is at least one serial number of data, and the value corresponding to a certain field of the first data does not match the value corresponding to the same field of the second data, then the comparison result of this period is finally confirmed to be inconsistent, and this inconsistent comparison result is recorded in MYSQL, and it is necessary to record in the list of data that did not match which serial number or which serial numbers of data have a inconsistent comparison result.

[0136] After the data warehouse tool Hive finishes comparing the first data and the second data in the first period, it will send a message to notify the APP to prompt the APP to check the comparison result. If the comparison result is consistent, the APP will not take any action; if the comparison result is inconsistent, the APP will read the data list to supplement the data for this period.

[0137] When the APP reads data but does not list, if the field of the first data does not completely match the field of the second data, the APP will extract the target source data for the hour from 9:00 to 10:00 in relational database B, store it in Kafka, and write it to HBase again after Spark Streaming calculation; if one or some sequence numbers of the first data appear, and the values ​​corresponding to some fields of them do not match the values ​​corresponding to the fields of the second data, the APP will extract the source data of the abnormal sequence numbers during the hour from 9:00 to 10:00 in relational database B, store it in Kafka, and write it to HBase again after Spark Streaming calculation.

[0138] Based on the same concept, the embodiment of the present invention also provides a device for detecting anomalies in real-time data streams, such as Figure 4 As shown, the device comprises:

[0139] A first data acquisition unit 401 is used to acquire target source data as first data from a data source through a first channel at a first time granularity, and store the first data in a distributed database; the distributed database is used to provide quasi-real-time services;

[0140] A second data acquisition unit 402 is used to acquire the target source data as second data from the data source through a second channel at a second time granularity; the second time granularity is greater than the first time granularity;

[0141] A comparison unit 403 is used to compare the first data of the first time period with the second data of the first time period to determine whether the first data of the first time period and the second data of the first time period meet a preset consistency standard;

[0142] The comparing unit 403 is further configured to continue comparing the first data of the second time period with the second data of the second time period if they match.

[0143] Furthermore, for the device, the comparison unit 403 is also used to: if there is no match, use the data in the first data of the first time period that is inconsistent with the second data of the first time period as third data; re-acquire the third data from the data source through the first channel, and update the distributed database according to the third data, and return to execute the step of comparing the first data of the first time period with the second data of the first time period.

[0144] Furthermore, the device also includes a determination unit 404. Before comparing the first data of the first time period with the second data of the first time period, the determination unit 404 is used to: determine the data type of the first data of the first time period; wherein the data type is used to indicate the importance of the data; determine the comparison rule of the first data of the first time period according to the data type of the first data of the first time period; wherein the level of the comparison rule is positively correlated with the importance of the data; and the comparison unit 403 is specifically used to: compare the first data of the first time period with the second data of the first time period according to the comparison rule.

[0145] Further, for the device, the determination unit 404 is specifically configured to: if the first data in the first time period includes data of multiple data types, determine the data type of the data with the highest importance as the data type of the first data in the first time period.

[0146] Furthermore, for the device, the comparison rules include at least one of the following: comparison of the number of data items, comparison of key fields, and comparison of all fields.

[0147] Further, for the device, the first data acquisition unit 401 is specifically used to: obtain the target source data from the data source through a distributed publishing message subscription tool according to the first time granularity; calculate the target source data through a real-time stream data processing tool to obtain the first data; the second data acquisition unit 402 is specifically used to: obtain the target source data from the data source through a data transfer tool according to the second time granularity; calculate the target source data to obtain the second data.

[0148] Further, for the device, the comparison unit 403 is specifically used to: send the third data in the data source to the distributed publishing message subscription tool through a notification tool.

[0149] The embodiment of the present invention further provides a computing device, which may be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), a memory, an input / output device, etc. The input device may include a keyboard, a mouse, a touch screen, etc. The output device may include a display device, such as a liquid crystal display (LCD), a cathode ray tube (CRT), etc.

[0150] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In an embodiment of the present invention, the memory may be used to execute program instructions of a data processing method applicable to quasi-real-time services;

[0151] The processor is used to call the program instructions stored in the memory and execute the data processing method suitable for quasi-real-time service according to the obtained program.

[0152] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a data processing method applicable to quasi-real-time services.

[0153] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0157] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0158] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data processing method suitable for quasi-real-time service, characterized in that: include: Acquire target source data as first data from a data source through a first channel at a first time granularity, and store the first data in a distributed database; The distributed database is used to provide quasi-real-time services, and the first channel is used to obtain data in real time; Acquire the target source data as second data from the data source through a second channel at a second time granularity; the second time granularity is greater than the first time granularity, and the second channel is used to acquire data in batches; Comparing the first data of the first time period with the second data of the first time period to determine whether the first data of the first time period and the second data of the first time period meet a preset consistency standard; If they match, continue to compare the first data of the second period with the second data of the second period; Before comparing the first data of the first time period with the second data of the first time period, the method further includes: Determine a data type of the first data in the first time period; wherein the data type is used to indicate the importance of the data; Determining, according to the data type of the first data in the first period, a comparison rule for the first data in the first period; wherein the level of the comparison rule is positively correlated with the importance of the data; The comparing the first data of the first time period with the second data of the first time period includes: According to the comparison rule, the first data of the first time period is compared with the second data of the first time period.

2. The method according to claim 1, characterized in that Also includes: If not, taking the data in the first data of the first time period that is inconsistent with the second data of the first time period as the third data; The third data is reacquired from the data source through the first channel, and the distributed database is updated according to the third data, and the step of comparing the first data of the first time period with the second data of the first time period is returned to be executed.

3. The method according to claim 1, characterized in that Determining a data type of the first data in the first time period includes: If the first data of the first time period includes data of multiple data types, the data type of the data with the highest importance is determined as the data type of the first data of the first time period.

4. The method according to claim 1, characterized in that The comparison rules include at least one of the following: comparison of the number of data items, comparison of key fields, and comparison of all fields.

5. The method according to any one of claims 1 to 4, characterized in that: Acquiring target source data as first data from a data source through a first channel at a first time granularity includes: According to the first time granularity, obtaining the target source data from the data source through a distributed publishing message subscription tool; Calculate the target source data by using a real-time stream data processing tool to obtain the first data; Acquiring the target source data as second data from the data source through a second channel at a second time granularity includes: According to the second time granularity, obtaining the target source data from the data source through a data transfer tool; The target source data is calculated to obtain the second data.

6. The method according to claim 5, characterized in that Reacquiring the third data from the data source through the first channel includes: The third data in the data source is sent to the distributed publishing and message subscription tool through a notification tool.

7. A data processing device suitable for quasi-real-time service, characterized in that: include: A first data acquisition unit is used to acquire target source data as first data from a data source through a first channel at a first time granularity, and store the first data in a distributed database; the distributed database is used to provide quasi-real-time services, and the first channel is used to acquire data in real time; A second data acquisition unit is used to acquire the target source data as second data from the data source through a second channel at a second time granularity; the second time granularity is greater than the first time granularity, and the second channel is used to acquire data in batches; a comparing unit, used to compare the first data of the first time period with the second data of the first time period, and determine whether the first data of the first time period and the second data of the first time period meet a preset consistency standard; The comparison unit is further used to continue to compare the first data of the second time period with the second data of the second time period if they match; a determination unit, configured to determine a data type of the first data of the first time period before comparing the first data of the first time period with the second data of the first time period; wherein the data type is used to indicate the importance of the data; and determine a comparison rule for the first data of the first time period according to the data type of the first data of the first time period; wherein the level of the comparison rule is positively correlated with the importance of the data; The comparison unit is specifically configured to compare the first data of the first time period with the second data of the first time period according to the comparison rule.

8. A computing device, characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 6 according to the obtained program.

9. A computer-readable storage medium, characterized in that: The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

Images