Offline correction method, system, device and medium for real-time calculation results

CN114547039BActive Publication Date: 2024-11-05CHINA PING AN LIFE INSURANCE CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210177307.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-25
Publication Date
2024-11-05
Estimated Expiration
2042-02-25

Smart Images

  • Figure CN114547039B_ABST
    Figure CN114547039B_ABST
Patent Text Reader

Abstract

The present invention relates to big data technology, and provides an offline correction method, system, device and medium for real-time calculation results. The method includes first calculating the real-time calculation results based on the real-time processing layer to obtain the data to be processed and storing them in the HBase database server. If it is determined that the date interval between the current system time and the acquisition time of the real-time calculation results is equal to the preset date threshold, and it is determined that the current system time meets the preset calculation start condition corresponding to the batch processing layer, the data to be processed is obtained, and the batch processing layer performs corresponding calculations on the data to be processed to obtain the offline calculation results, obtain the target number of columns of the real-time calculation results in the HBase database server, and store the offline calculation results in the corresponding columns of the HBase database server according to the target number of columns. The correction of abnormal historical real-time data due to network fluctuations and other reasons is achieved without affecting the writing of current real-time data, thereby improving the correctness of real-time data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to an offline correction method, system, computer equipment and storage medium for real-time calculation results. Background Art

[0002] With the continuous iteration and improvement of big data, the timeliness of data is becoming more and more important for the refined operation of enterprises. It is extremely useful to be able to effectively mine valuable information in real time from the massive amount of data generated every day. Due to the common delays and disorder problems in real-time data processing scenarios, the accuracy of real-time data is often difficult to guarantee.

[0003] To solve this problem, the traditional Lambda architecture is now a common architecture for real-time processing. The notable feature of the Lambda architecture is that data enters both the batch layer (Batch layer) and the real-time processing layer (Speed ​​layer). For the batch layer, data is first stored when it arrives, and the batch results are calculated in a timely manner. Batch calculations are triggered at certain intervals, and the calculated data range can range from a few hours to a few years. The real-time processing layer is used to calculate real-time results and is a supplement to the batch layer. Data is pushed to downstream data services, and the data service layer completes the merging of offline results and real-time results. The disadvantage of this architecture is that it does not correct abnormal real-time results, and it is necessary to query both batch and real-time calculation results, which increases the complexity and timeliness of data services. Summary of the invention

[0004] The embodiments of the present invention provide a method, system, computer device and storage medium for offline correction of real-time computing results, aiming to solve the problem that in the Lambda architecture used in real-time data processing scenarios in the prior art, if the real-time data has delays, disorder and other problems, the real-time computing results are inaccurate, and the abnormal real-time results are not corrected.

[0005] In a first aspect, an embodiment of the present invention provides an offline correction method for real-time calculation results, which includes:

[0006] The first server, in response to the data calculation instruction, obtains the to-be-processed data corresponding to the data calculation instruction;

[0007] The real-time processing layer in the first server performs corresponding operations on the data to be processed to obtain real-time calculation results, and the first server sends the data to be processed to the batch processing layer in the second server;

[0008] The HBase database server receives and stores the real-time calculation results; wherein each column of data in the HBase database server stores the calculation results of a preset number of versions, and the preset number of versions is greater than or equal to 2;

[0009] If the second server determines that the date interval between the current system time and the acquisition time of the real-time calculation result is equal to the preset date threshold, and determines that the current system time meets the preset calculation start condition corresponding to the batch processing layer, the data to be processed is acquired;

[0010] Performing corresponding operations on the data to be processed by the batch processing layer of the second server to obtain an offline calculation result; and

[0011] The HBase database server obtains the target number of columns of the real-time calculation result, and stores the offline calculation result in the corresponding number of columns in the HBase database server according to the target number of columns.

[0012] In a second aspect, an embodiment of the present invention provides an offline correction system for real-time calculation results, which includes a first server, a second server and an HBase database server;

[0013] The first server is used to obtain the to-be-processed data corresponding to the data calculation instruction in response to the data calculation instruction;

[0014] The first server is further configured to perform corresponding operations on the data to be processed through a real-time processing layer in the first server to obtain real-time calculation results, and the first server sends the data to be processed to a batch processing layer in the second server;

[0015] The HBase database server is used to receive and store the real-time calculation results; wherein each column of data in the HBase database server stores the calculation results of a preset number of versions, and the preset number of versions is greater than or equal to 2;

[0016] The second server is configured to obtain the data to be processed if it is determined that the date interval between the current system time and the acquisition time of the real-time calculation result is equal to a preset date threshold, and if it is determined that the current system time satisfies the preset calculation start condition corresponding to the batch processing layer;

[0017] The second server is further configured to perform corresponding operations on the data to be processed through the batch processing layer to obtain offline calculation results; and

[0018] The HBase database server is further used to obtain the target number of columns of the real-time calculation result, and store the offline calculation result in the corresponding number of columns in the HBase database server according to the target number of columns.

[0019] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the offline correction method of the real-time calculation result described in the first aspect is implemented.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the offline correction method of the real-time calculation results described in the first aspect above.

[0021] The embodiment of the present invention provides an offline correction method, system, computer device and storage medium for real-time calculation results. First, the data to be processed is calculated based on the real-time processing layer to obtain the real-time calculation results and stored in the HBase database server. If it is determined that the date interval between the current system time and the acquisition time of the real-time calculation results is equal to the preset date threshold, and it is determined that the current system time meets the preset calculation start condition corresponding to the batch processing layer, the data to be processed is obtained, and the batch processing layer performs corresponding calculations on the data to be processed to obtain the offline calculation results, obtain the target number of columns of the real-time calculation results in the HBase database server, and store the offline calculation results in the corresponding columns of the HBase database server according to the target number of columns. The historical real-time data that is abnormal due to network fluctuations and other reasons is corrected without affecting the writing of the current real-time data, thereby improving the correctness of the real-time data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of an application scenario of the offline correction method of real-time calculation results provided by an embodiment of the present invention;

[0024] Figure 2 A schematic flow chart of an offline correction method for real-time calculation results provided by an embodiment of the present invention;

[0025] Figure 3 A schematic block diagram of an offline correction system for real-time calculation results provided by an embodiment of the present invention;

[0026] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0029] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the offline correction method of real-time calculation results provided by an embodiment of the present invention; Figure 2 A flow chart of an offline correction method for real-time calculation results provided by an embodiment of the present invention is provided. The offline correction method for real-time calculation results is applied in a server cluster and is executed by application software installed in the server cluster.

[0032] like Figure 2 As shown, the method includes steps S101 to S106.

[0033] S101. In response to a data calculation instruction, a first server obtains data to be processed corresponding to the data calculation instruction.

[0034] In this embodiment, the technical solution is described with the server cluster as the execution subject. The server cluster includes a first server, a second server and an HBase database server, wherein the first server is used for real-time data processing, the second server is used for offline data processing, and the HBase database server is used to store real-time calculation results and offline calculation results of each day. The second server can also be understood as a batch layer (i.e., Batch Layer), in which the data storage engine used is HDFS / Hive. On the second server, the full amount of data can be pre-processed to obtain pre-processing results and stored, and the second server can be divided into an operational data layer ODS (ODS full name is Operational Data Store), a data warehouse detail layer DWD (DWD full name is Data Warehouse Detail) and a data market layer DM (DM full name is DataMarket) at the model level. The first server can also be understood as a real-time processing layer (i.e., Speed ​​Layer), which can perform real-time calculations on incremental data to obtain real-time calculation results. In the first server, real-time data can be stored and processed first through a message middleware (such as a Kafka message queue).

[0035] When the data to be processed is sent from other terminals or other servers to the real-time processing layer of the first server for real-time calculation, the data to be processed is first received and added to the processing queue for processing.

[0036] In one embodiment, after step S101, the method further includes:

[0037] The first server sends the data to be processed to the message middleware and saves it.

[0038] In this embodiment, the message middleware is also deployed in the first server. After the message middleware in the first server, specifically the Kafka message queue, receives and stores the data to be processed, the big data computing engine (such as the Flink computing engine, etc.) in the first server can consume the data to be processed and perform real-time computing to obtain the computing results. It can be seen that storing the data to be processed based on the real-time processing layer can effectively improve the subsequent computing efficiency.

[0039] S102: Perform corresponding operations on the data to be processed by the real-time processing layer in the first server to obtain real-time calculation results, and the first server sends the data to be processed to the batch processing layer of the second server.

[0040] In this embodiment, after the data to be processed is first stored in the message middleware for queuing for processing, if it is time to process the data to be processed, the data to be processed is first consumed from the message middleware by the big data computing engine in the first server, and then the data to be processed is calculated and processed based on the calculation logic in the big data computing engine to obtain real-time calculation results. At this time, the real-time calculation results can be sent to the recipient of the data to be processed to receive the processing results, or they can be further sent to the HBase database server in the server cluster for storage for T+1 correction.

[0041] S103. The HBase database server receives and stores the real-time calculation result; wherein each column of data in the HBase database server stores the calculation result of a preset number of versions, and the preset number of versions is greater than or equal to 2.

[0042] In this embodiment, the real-time calculation result is selected to be stored in the HBase database server because the HBase database in the HBase database server has a multi-version storage mechanism. For example, if the preset number of versions of the HBase database server is set to 3, each column of the HBase database server can store three versions of values. In this way, the real-time calculation result obtained by real-time processing of each data to be processed can be stored in the corresponding column in the data table created for the current date (for example, the current date is recorded as the Tth day) in the HBase database server, and the corresponding column in the data table can store the 3 versions of calculation results corresponding to the data to be processed. The reason why multiple versions are stored for the processing results of the data to be processed is that when the data to be processed is sent to the batch processing layer in the second server as offline data for offline calculation (for example, when the data to be processed is sent to the batch processing layer as offline data and is processed offline on T+1 day), because there is no delay, disorder and other problems in real time offline, more accurate results can be obtained. At this time, the data to be processed can also store the offline calculation results in the corresponding storage column in the data table to compare with the real-time calculation results, so as to determine whether to refer to the offline calculation results for correction.

[0043] When the data to be processed is stored in the batch processing layer (the batch processing layer is deployed in the second server), it is stored in the full data table of the Tth day created correspondingly in the data mart layer of the batch processing layer.

[0044] In one embodiment, step S103 includes:

[0045] The HBase database server obtains a first acquisition timestamp of the real-time calculation result, uses the first acquisition timestamp as a first version number of the real-time calculation result, and stores the first version number together with the real-time calculation result in the HBase database server.

[0046] In this embodiment, in order to distinguish different versions of the calculation results of the same data to be processed in the HBase database server, it is necessary to add a timestamp to distinguish the calculation results of each version. At this time, the first acquisition timestamp of the real-time calculation result is used as the first version number of the real-time calculation result. For example, if the first acquisition timestamp is 9:00 on the Tth day, 9:00 on the Tth day is used as the first version number corresponding to the first acquisition timestamp. When querying the calculation results of each column with multiple versions later, the version with the largest timestamp, that is, the latest version, is used as the query result by default. It can be seen that by adding a timestamp to the calculation result as the calculation result version number, the calculation result can be quickly versioned.

[0047] It can be seen that in steps S101-S103, real-time processing of the data to be processed on the Tth day is implemented, and the obtained real-time calculation results are stored in the corresponding columns of the storage table created for the Tth day in the HBase database server.

[0048] S104: If the second server determines that the date interval between the current system time and the acquisition time of the real-time calculation result is equal to the preset date threshold, and determines that the current system time meets the preset calculation start condition corresponding to the batch layer, the data to be processed is obtained.

[0049] In this embodiment, in order to further perform offline correction of the real-time calculation results of the data to be processed for T+n days (where n is a preset date threshold, and n is generally set to 1), the current system time can be first obtained, and then it is determined whether the time interval between the current system time and the acquisition time of the real-time calculation results (that is, the above-mentioned first acquisition timestamp) is equal to the preset time threshold n. If it is determined that the date interval between the current system time and the acquisition time of the real-time calculation results is equal to the date threshold and it is determined that the current system time meets the preset calculation start condition corresponding to the batch processing layer (for example, the preset calculation start condition corresponding to the batch processing layer is 1:00 every day, and the current system time is also 1:00, then it can be determined that the current system time meets the preset calculation start condition corresponding to the batch processing layer), the data to be processed is obtained from the data mart layer of the batch processing layer.

[0050] In one embodiment, before step S104, the method further includes:

[0051] The second server obtains the first full amount of data on a date corresponding to the first acquisition timestamp in the batch processing layer;

[0052] The second server obtains, in the batch processing layer, second full data of the day before the date corresponding to the first acquisition timestamp;

[0053] The second server obtains difference data between the first full data and the second full data, and stores the difference data in a data mart layer in the batch processing layer.

[0054] In this embodiment, in order to improve the efficiency of acquiring the data to be processed offline in the batch layer every day, the first full data corresponding to the Tth day (that is, the date corresponding to the first acquisition timestamp) of the data mart layer of the batch layer can be acquired first (the data to be processed will definitely be included in the first full data), and then the second full data corresponding to the T-1st day (that is, the day before the date corresponding to the first acquisition timestamp) of the data mart layer of the batch layer is acquired (the data to be processed will definitely not be included in the second full data because it is processed in real time on the Tth day). The difference data is obtained based on the comparison between the first full data and the second full data, and the difference data is stored in the incremental data table corresponding to the Tth day of the data mart layer in the batch layer. In addition, the data to be processed must be included in the incremental data table corresponding to the Tth day of the data mart layer.

[0055] S105. Perform corresponding operations on the data to be processed through the batch processing layer of the second server to obtain offline calculation results.

[0056] In this embodiment, since the batch processing layer includes the operation data layer ODS, the data warehouse detail layer DWD and the data mart layer DM, and the data to be processed exists in the incremental data table corresponding to the Tth day of the data mart layer DM, at this time, the operation data layer ODS in the batch processing layer can obtain all data from the incremental data table corresponding to the Tth day of the data mart layer DM and perform calculations one by one to obtain the calculation results corresponding to each data in the incremental data table. Since the incremental data table corresponding to the Tth day of the data mart layer DM is also calculated by the operation data layer ODS, the offline calculation results corresponding to the data to be processed can be obtained. It can be seen that the historical data stored in the data mart layer can be processed offline based on the batch processing layer, and the offline processing results obtained by this type of offline processing can be further used for data verification.

[0057] In one embodiment, step S105 includes:

[0058] The second server obtains the big data computing logic correspondingly created in the operation data layer of the batch processing layer;

[0059] The second server calculates the data to be processed according to the big data calculation logic to obtain an offline calculation result.

[0060] After step S105, the following steps are further included:

[0061] The second server obtains a second acquisition timestamp of the offline calculation result, and adjusts the second acquisition timestamp according to a preset timestamp adjustment strategy to update the second acquisition timestamp.

[0062] In this embodiment, since the operation data layer ODS of the batch processing layer stores the big data computing logic (for example, the computing logic corresponding to the Flink task created based on the Flink computing engine) for respectively computing the incremental data table corresponding to the Tth day in the data mart layer, the data to be processed is then offline computed by the big data computing logic to obtain the offline computing result corresponding to the data to be processed. In addition, in order to increase the data version number of the offline computing result accordingly, the second acquisition timestamp of the offline computing result can be obtained, and the second acquisition timestamp can be adjusted according to the preset timestamp adjustment strategy to update the second acquisition timestamp, and the updated second acquisition timestamp is used as the version number of the offline computing result, so that the offline computing result can be quickly versioned.

[0063] In one embodiment, the timestamp adjustment strategy is used to reduce the date corresponding to the to-be-processed acquisition timestamp by one and set the time corresponding to the to-be-processed acquisition timestamp to a first preset time.

[0064] In this embodiment, for example, if the current system time is 1:00 on the T+1 day, then the second acquisition timestamp corresponding to the offline calculation result is slightly later than 1:00 on the T+1 day (such as 1:05 on the T+1 day). At this time, the second acquisition timestamp can be based on the timestamp adjustment strategy to first subtract one from the T+1 day to get the T day, and then 1:00 is automatically adjusted to the time corresponding to 1ms before 24:00:00, that is, at this time, the second acquisition timestamp is adjusted to 1ms before 24:00:00 on the T day through the timestamp adjustment strategy, that is, the version number corresponding to the offline calculation result is 1ms before 24:00:00 on the T day. In this way, the offline calculation result obtained by recalculating the previous to-be-processed data in an offline state on a later date can be used as a correction value for the real-time calculation result.

[0065] S106. The HBase database server obtains a target number of columns for the real-time calculation result, and stores the offline calculation result in a corresponding number of columns in the HBase database server according to the target number of columns.

[0066] In this embodiment, for example, the offline calculation result corresponding to the data to be processed is obtained on the T+1 day, and the version number of the offline calculation result is set to 1ms before 24:00:00 on the T day. At this time, the target column number in the HBase database server can be retrieved based on the data to be processed. Since the calculation results of multiple versions of the data to be processed can be stored in the target column number, the offline calculation result and its second acquisition timestamp (1ms before 24:00:00 on the T day) can be stored in the corresponding column number in the HBase database server according to the target column number after the target column number is obtained, thereby realizing that the real-time calculation result and the offline calculation result of the data to be processed are stored in the same column of the same table in the HBase database server, and the final calculation value corresponding to the data to be processed can be determined based on the version number after the real-time calculation result and the offline calculation result, so as to be called by other calculation processes. When the calculation results of multiple versions of each column are queried later, the version with the largest timestamp, that is, the latest version, is fed back by default as the query result.

[0067] The embodiment of the present application can acquire and process relevant data in the server cluster based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] The method realizes the correction of historical real-time data that is abnormal due to network fluctuations and the like, without affecting the writing of current real-time data, thereby improving the correctness of real-time data.

[0070] The embodiment of the present invention further provides an offline correction system for real-time calculation results, which is used to execute any embodiment of the above-mentioned offline correction method for real-time calculation results. Figure 3 , Figure 3 is a schematic block diagram of an offline correction system 100 for real-time calculation results provided by an embodiment of the present invention.

[0071] Among them, Figure 3As shown, the offline correction system 100 for real-time calculation results includes a first server 101 , a second server 102 and an HBase database server 103 .

[0072] The first server 101 is used to obtain the to-be-processed data corresponding to the data calculation instruction in response to the data calculation instruction.

[0073] In this embodiment, the technical solution is described with the server cluster as the execution subject. The server cluster includes a first server, a second server and an HBase database server, wherein the first server is used for real-time data processing, the second server is used for offline data processing, and the HBase database server is used to store the real-time calculation results and offline calculation results of each day. The second server can also be understood as a batch layer (i.e., Batch Layer), in which the data storage engine used is HDFS / Hive. The full amount of data can be pre-processed on the second server to obtain the pre-processing results and store them, and the second server can be divided into an operation data layer ODS, a data warehouse detail layer DWD and a data mart layer DM at the model level. The first server can also be understood as a real-time processing layer (i.e., Speed ​​Layer), which can perform real-time calculations on incremental data to obtain real-time calculation results. In the first server, real-time data can be stored and processed first through a message middleware (such as a Kafka message queue).

[0074] When the data to be processed is sent from other terminals or other servers to the real-time processing layer of the first server for real-time calculation, the data to be processed is first received and added to the processing queue for processing.

[0075] In one embodiment, in an offline correction system 100 that calculates results in real time:

[0076] The first server is further configured to send the data to be processed to the message middleware and save the data.

[0077] In this embodiment, the message middleware is also deployed in the first server. After the message middleware in the first server, specifically the Kafka message queue, receives and stores the data to be processed, the big data computing engine (such as the Flink computing engine, etc.) in the first server can consume the data to be processed and perform real-time computing to obtain the computing results. It can be seen that storing the data to be processed based on the real-time processing layer can effectively improve the subsequent computing efficiency.

[0078] The first server 101 is further configured to perform corresponding operations on the data to be processed through a real-time processing layer in the first server to obtain real-time calculation results, and the first server sends the data to be processed to a batch processing layer in the second server.

[0079] In this embodiment, after the data to be processed is first stored in the message middleware for queuing for processing, if it is time to process the data to be processed, the data to be processed is first consumed from the message middleware by the big data computing engine in the first server, and then the data to be processed is calculated and processed based on the calculation logic in the big data computing engine to obtain real-time calculation results. At this time, the real-time calculation results can be sent to the recipient of the data to be processed to receive the processing results, or they can be further sent to the HBase database server in the server for storage for T+1 correction.

[0080] The HBase database server 103 is used to receive and store the real-time calculation results; wherein each column of data in the HBase database server stores the calculation results of a preset number of versions, and the preset number of versions is greater than or equal to 2.

[0081] In this embodiment, the real-time calculation result is selected to be stored in the HBase database server because the HBase database of the HBase database server has a multi-version storage mechanism. For example, if the preset number of versions of the HBase database server is set to 3, each column of the HBase database server can store three versions of values. In this way, the real-time calculation result obtained by real-time processing of each data to be processed can be stored in the corresponding column in the data table created for the current date (for example, the current date is recorded as the Tth day) in the HBase database server, and the corresponding column in the data table can store the 3 versions of calculation results corresponding to the data to be processed. The reason why multiple versions are stored for the processing results of the data to be processed is that when the data to be processed is sent to the batch processing layer in the second server as offline data for offline calculation (for example, when the data to be processed is sent to the batch processing layer as offline data and is processed offline on T+1 day), because there is no delay, disorder and other problems in real time offline, more accurate results can be obtained. At this time, the data to be processed can also store the offline calculation results in the corresponding storage column in the data table to compare with the real-time calculation results, so as to determine whether to refer to the offline calculation results for correction.

[0082] When the data to be processed is stored in the batch processing layer (the batch processing layer is deployed in the second server), it is stored in the full data table of the Tth day created correspondingly in the data mart layer of the batch processing layer.

[0083] In one embodiment, the HBase database server 103 is further specifically used for:

[0084] A first acquisition timestamp of the real-time calculation result is obtained, and the first acquisition timestamp is used as a first version number of the real-time calculation result and stored together with the real-time calculation result in an HBase database server.

[0085] In this embodiment, in order to distinguish different versions of the calculation results of the same data to be processed in the HBase database server, it is necessary to add a timestamp to distinguish the calculation results of each version. At this time, the first acquisition timestamp of the real-time calculation result is used as the first version number of the real-time calculation result. For example, if the first acquisition timestamp is 9:00 on the Tth day, 9:00 on the Tth day is used as the first version number corresponding to the first acquisition timestamp. When querying the calculation results of each column with multiple versions later, the version with the largest timestamp, that is, the latest version, is used as the query result by default. It can be seen that by adding a timestamp to the calculation result as the calculation result version number, the calculation result can be quickly versioned.

[0086] It can be seen that the above processing realizes the real-time processing of the data to be processed on the Tth day, and stores the obtained real-time calculation results in the corresponding columns of the storage table created for the Tth day in the HBase database server.

[0087] The second server 102 is also used to obtain the data to be processed if it is determined that the date interval between the current system time and the acquisition time of the real-time calculation result is equal to a preset date threshold, and if it is determined that the current system time meets the preset calculation start condition corresponding to the batch layer.

[0088] In this embodiment, in order to further perform offline correction of the real-time calculation results of the data to be processed for T+n days (where n is a preset date threshold, and n is generally set to 1), the current system time can be first obtained, and then it is determined whether the time interval between the current system time and the acquisition time of the real-time calculation results (that is, the above-mentioned first acquisition timestamp) is equal to the preset time threshold n. If it is determined that the date interval between the current system time and the acquisition time of the real-time calculation results is equal to the date threshold and it is determined that the current system time meets the preset calculation start condition corresponding to the batch processing layer (for example, the preset calculation start condition corresponding to the batch processing layer is 1:00 every day, and the current system time is also 1:00, then it can be determined that the current system time meets the preset calculation start condition corresponding to the batch processing layer), the data to be processed is obtained from the data mart layer of the batch processing layer.

[0089] In one embodiment, in an offline correction system 100 that calculates results in real time:

[0090] The second server 102 is further configured to obtain the first full amount of data on a date corresponding to the first acquisition timestamp in the batch processing layer;

[0091] The second server 102 is further configured to obtain second full data from the batch layer one day before the date corresponding to the first acquisition timestamp;

[0092] The second server 102 is further configured to obtain difference data between the first full data and the second full data, and store the difference data in a data mart layer in the batch processing layer.

[0093] In this embodiment, in order to improve the efficiency of acquiring the data to be processed offline in the batch layer every day, the first full data corresponding to the Tth day (that is, the date corresponding to the first acquisition timestamp) of the data mart layer of the batch layer can be acquired first (the data to be processed will definitely be included in the first full data), and then the second full data corresponding to the T-1st day (that is, the day before the date corresponding to the first acquisition timestamp) of the data mart layer of the batch layer is acquired (the data to be processed will definitely not be included in the second full data because it is processed in real time on the Tth day). The difference data is obtained based on the comparison between the first full data and the second full data, and the difference data is stored in the incremental data table corresponding to the Tth day of the data mart layer in the batch layer. In addition, the data to be processed must be included in the incremental data table corresponding to the Tth day of the data mart layer.

[0094] The second server 102 is further configured to perform corresponding operations on the data to be processed through the batch processing layer to obtain offline calculation results.

[0095] In this embodiment, since the batch processing layer includes the operation data layer ODS, the data warehouse detail layer DWD and the data mart layer DM, and the data to be processed exists in the incremental data table corresponding to the Tth day of the data mart layer DM, at this time, the operation data layer ODS in the batch processing layer can obtain all data from the incremental data table corresponding to the Tth day of the data mart layer DM and perform calculations one by one to obtain the calculation results corresponding to each data in the incremental data table. Since the incremental data table corresponding to the Tth day of the data mart layer DM is also calculated by the operation data layer ODS, the offline calculation results corresponding to the data to be processed can be obtained. It can be seen that the historical data stored in the data mart layer can be processed offline based on the batch processing layer, and the offline processing results obtained by this type of offline processing can be further used for data verification.

[0096] In one embodiment, the second server 102 is further specifically configured to:

[0097] Obtain the big data computing logic correspondingly created in the operation data layer of the batch processing layer;

[0098] The data to be processed is calculated according to the big data calculation logic to obtain an offline calculation result.

[0099] In the offline correction system 100 for real-time calculation results, the second server 102 is used for:

[0100] A second acquisition timestamp of the offline calculation result is obtained, and the second acquisition timestamp is adjusted according to a preset timestamp adjustment strategy to update the second acquisition timestamp.

[0101] In this embodiment, since the operation data layer ODS of the batch processing layer stores the big data computing logic (for example, the computing logic corresponding to the Flink task created based on the Flink computing engine) for respectively computing the incremental data table corresponding to the Tth day in the data mart layer, the data to be processed is then offline computed by the big data computing logic to obtain the offline computing result corresponding to the data to be processed. In addition, in order to increase the data version number of the offline computing result accordingly, the second acquisition timestamp of the offline computing result can be obtained, and the second acquisition timestamp can be adjusted according to the preset timestamp adjustment strategy to update the second acquisition timestamp, and the updated second acquisition timestamp is used as the version number of the offline computing result, so that the offline computing result can be quickly versioned.

[0102] In one embodiment, the timestamp adjustment strategy is used to reduce the date corresponding to the to-be-processed acquisition timestamp by one and set the time corresponding to the to-be-processed acquisition timestamp to a first preset time.

[0103] In this embodiment, for example, if the current system time is 1:00 on the T+1 day, then the second acquisition timestamp corresponding to the offline calculation result is slightly later than 1:00 on the T+1 day (such as 1:05 on the T+1 day). At this time, the second acquisition timestamp can be based on the timestamp adjustment strategy to first subtract one from the T+1 day to get the T day, and then 1:00 is automatically adjusted to the time corresponding to 1ms before 24:00:00, that is, at this time, the second acquisition timestamp is adjusted to 1ms before 24:00:00 on the T day through the timestamp adjustment strategy, that is, the version number corresponding to the offline calculation result is 1ms before 24:00:00 on the T day. In this way, the offline calculation result obtained by recalculating the previous to-be-processed data in an offline state on a later date can be used as a correction value for the real-time calculation result.

[0104] The HBase database server 103 is further configured to obtain a target number of columns for the real-time calculation result, and store the offline calculation result in a corresponding number of columns in the HBase database server according to the target number of columns.

[0105] In this embodiment, for example, the offline calculation result corresponding to the data to be processed is obtained on the T+1 day, and the version number of the offline calculation result is set to 1ms before 24:00:00 on the T day. At this time, the target column number in the HBase database server can be retrieved based on the data to be processed. Since the calculation results of multiple versions of the data to be processed can be stored in the target column number, the offline calculation result and its second acquisition timestamp (1ms before 24:00:00 on the T day) can be stored in the corresponding column number in the HBase database server according to the target column number after the target column number is obtained, thereby realizing that the real-time calculation result and the offline calculation result of the data to be processed are stored in the same column of the same table in the HBase database server, and the final calculation value corresponding to the data to be processed can be determined based on the version number after the real-time calculation result and the offline calculation result, so as to be called by other calculation processes. When the calculation results of multiple versions of each column are queried later, the version with the largest timestamp, that is, the latest version, is fed back by default as the query result.

[0106] The system realizes the correction of historical real-time data that is abnormal due to network fluctuations and other reasons without affecting the writing of current real-time data, thereby improving the accuracy of real-time data.

[0107] The above-mentioned offline correction system of real-time calculation results can be implemented in the form of a computer program, which can be used in Figure 4 Runs on the computer device shown.

[0108] See also Figure 4 , Figure 4 5 is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device 500 is a server or a server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0109] See also Figure 4 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a device bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .

[0110] The storage medium 503 can store an operating device 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute an offline correction method for real-time calculation results.

[0111] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0112] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can perform an offline correction method for real-time calculation results.

[0113] The network interface 505 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] The processor 502 is used to run a computer program 5032 stored in the memory to implement the offline correction method of the real-time calculation result disclosed in the embodiment of the present invention.

[0115] Those skilled in the art will understand that Figure 4 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Figure 4 The embodiments shown are consistent and will not be described again here.

[0116] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0117] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the offline correction method of the real-time calculation result disclosed in the embodiment of the present invention is implemented.

[0118] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0119] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. Units with the same function may also be combined into one unit. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0121] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0122] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a background server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), disk or CD-ROM and other media that can store program codes.

[0123] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for offline correction of real-time calculation results, characterized in that: include: The first server, in response to the data calculation instruction, obtains the to-be-processed data corresponding to the data calculation instruction; The real-time processing layer in the first server performs corresponding operations on the data to be processed to obtain real-time calculation results, and the first server sends the data to be processed to the batch processing layer in the second server; The HBase database server receives and stores the real-time calculation results; wherein each column of data in the HBase database server stores the calculation results of a preset number of versions, and the preset number of versions is greater than or equal to 2; The HBase database server receives and stores the real-time calculation result, including: The HBase database server obtains a first acquisition timestamp of the real-time calculation result, uses the first acquisition timestamp as a first version number of the real-time calculation result, and stores the first version number together with the real-time calculation result in the HBase database server; If the second server determines that the date interval between the current system time and the acquisition time of the real-time calculation result is equal to the preset date threshold, and determines that the current system time meets the preset calculation start condition corresponding to the batch processing layer, the data to be processed is acquired; If the second server determines that the date interval between the current system time and the acquisition time of the real-time calculation result is equal to the preset date threshold, and determines that the current system time meets the preset calculation start condition corresponding to the batch processing layer, before acquiring the data to be processed, it also includes: The second server obtains the first full amount of data on a date corresponding to the first acquisition timestamp in the batch processing layer; The second server obtains, in the batch processing layer, second full data of the day before the date corresponding to the first acquisition timestamp; The second server obtains difference data between the first full data and the second full data, and stores the difference data in a data mart layer in the batch processing layer; Performing corresponding operations on the data to be processed by the batch processing layer of the second server to obtain an offline calculation result; and The HBase database server obtains the target number of columns of the real-time calculation result, and stores the offline calculation result in the corresponding number of columns in the HBase database server according to the target number of columns; After performing corresponding operations on the to-be-processed data through the batch processing layer of the second server to obtain offline calculation results, the method further includes: The second server obtains a second acquisition timestamp of the offline calculation result, and adjusts the second acquisition timestamp according to a preset timestamp adjustment strategy to update the second acquisition timestamp; The timestamp adjustment strategy is used to reduce the date corresponding to the to-be-processed acquisition timestamp by one and set the time corresponding to the to-be-processed acquisition timestamp to the first preset time.

2. The off-line correction method of real-time calculation results according to claim 1, characterized in that: The performing corresponding operations on the to-be-processed data by the batch processing layer of the second server to obtain an offline calculation result includes: The second server obtains the big data computing logic correspondingly created in the operation data layer of the batch processing layer; The second server calculates the data to be processed according to the big data calculation logic to obtain an offline calculation result.

3. The offline correction method of real-time calculation results according to claim 1, characterized in that: After the first server, in response to the data calculation instruction, obtains the to-be-processed data corresponding to the data calculation instruction, the method further includes: The first server sends the data to be processed to the message middleware and saves it.

4. An offline correction system for real-time calculation results, characterized in that: Includes a first server, a second server and an HBase database server; The first server is used to obtain the to-be-processed data corresponding to the data calculation instruction in response to the data calculation instruction; The first server is further configured to perform corresponding operations on the data to be processed through a real-time processing layer in the first server to obtain real-time calculation results, and the first server sends the data to be processed to a batch processing layer in the second server; The HBase database server is used to receive and store the real-time calculation results; wherein each column of data in the HBase database server stores the calculation results of a preset number of versions, and the preset number of versions is greater than or equal to 2; The HBase database server is further used to obtain a first acquisition timestamp of the real-time calculation result, use the first acquisition timestamp as a first version number of the real-time calculation result, and store it together with the real-time calculation result in the HBase database server; The second server is configured to obtain the data to be processed if it is determined that the date interval between the current system time and the acquisition time of the real-time calculation result is equal to a preset date threshold, and if it is determined that the current system time satisfies the preset calculation start condition corresponding to the batch processing layer; If the second server determines that the date interval between the current system time and the time when the real-time calculation result is obtained is equal to the preset date threshold, and determines that the current system time meets the preset calculation start condition corresponding to the batch layer, before obtaining the data to be processed, the second server is further used to: Acquire the first full amount of data on a date corresponding to the first acquisition timestamp in the batch processing layer; Acquire the second full amount of data from the batch layer one day before the date corresponding to the first acquisition timestamp; Obtain difference data between the first full data and the second full data, and store the difference data in a data mart layer in a batch processing layer; The second server is further configured to perform corresponding operations on the data to be processed through the batch processing layer to obtain offline calculation results; After performing corresponding operations on the to-be-processed data through the batch processing layer of the second server to obtain an offline calculation result, the second server is further used to: Obtaining a second acquisition timestamp of the offline calculation result, and adjusting the second acquisition timestamp according to a preset timestamp adjustment strategy to update the second acquisition timestamp; The timestamp adjustment strategy is used to reduce the date corresponding to the pending acquisition timestamp by one and set the time corresponding to the pending acquisition timestamp to the first preset time; The HBase database server is further used to obtain the target number of columns of the real-time calculation result, and store the offline calculation result in the corresponding number of columns in the HBase database server according to the target number of columns.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the off-line correction method of the real-time calculation result according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to perform the offline correction method of real-time calculation results according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Data backflow method, device, equipment and system

    CN110008269A

  • Data processing method and system and non-instantaneous computer readable storage medium

    CN113760983A

  • Data processing method based on data medium table and related equipment thereof

    CN113836235A