Data processing method and device for real-time computing framework

Abstract

The invention discloses a data processing method and device for a real-time computing framework, and the method comprises the steps: obtaining the time when a result set is written into a database in a log, reading data in the time in the log, and constraining the data through a unique index mechanism which is used for guaranteeing that values of any column or multiple columns in a table of the database are unique, according to the constrained data and the result set, a target index is obtained, the target index is the maximum index under the same index, and the target index is written into a database in a coverage writing mode so as to ensure the smoothness and consistency of the data of the real-time computing framework in the data processing process.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a data processing method and device for a real-time computing framework. Background Art

[0002] In real-time computing of real-time systems, a common problem is that the data source data is repeated during data replay, resulting in duplicate result sets. In addition, when the real-time program is restarted, it will automatically process the data that was not processed in the previous batch, which will also cause duplicate result sets.

[0003] Data replay is the process in which the real-time system needs to re-receive data from a data source that was received at a certain point in time due to logic modification or incorrect original processing methods, and recalculate and rewrite the result set. When the real-time program is stopped, a batch of data may not have been fully processed. When the real-time program is restarted, the previous batch of data will be automatically processed again, and this process will cause all or part of the result set of the batch to be repeated.

[0004] Therefore, how to solve the duplication problem of the result set and ensure the smoothness and consistency of the result set is an urgent problem to be solved in this application. Summary of the invention

[0005] In view of this, the present application discloses a data processing method and device for a real-time computing framework, which aims to solve the problem of duplication of result sets and retain the correct indicators defined in the actual business under the same index, so as to ensure the smoothness and consistency of the result set even under data replay and restart.

[0006] In order to achieve the above purpose, the disclosed technical solution is as follows:

[0007] In a first aspect, the present application discloses a data processing method for a real-time computing framework, the method comprising:

[0008] Get the time in the log when the result set is written to the database;

[0009] Read the data within the time in the log;

[0010] The data is constrained by a unique index mechanism; the unique index mechanism is used to ensure that the value of any column or multiple columns in a table of a database is unique;

[0011] According to the constrained data and result set, a target index is obtained; the target index is a maximum value index under the same index;

[0012] The target indicator is written into the database by overwriting to ensure the smoothness and consistency of data during the data processing of the real-time computing framework.

[0013] Preferably, the time when the result set is written into the log of the database includes:

[0014] Get the in-log start time and in-log end time of writing the result set to the database;

[0015] The in-log time at which the result set is written into the database is determined according to the in-log start time and the in-log start time end time.

[0016] Preferably, the reading of the data within the time period of the log includes:

[0017] Get the current system time and data reporting time;

[0018] Determine a periodic reading method according to the current system time, the time in the log and the data reporting time;

[0019] According to the periodic reading mode, the data within the time in the log is read.

[0020] Preferably, constraining the data by a unique index mechanism includes:

[0021] The data is constrained through a unique index mechanism to ensure that the value of any column or multiple columns in the data table is unique.

[0022] Preferably, obtaining the target indicator based on the constrained data and result set includes:

[0023] Right join the constrained data with the result set;

[0024] Under the right join condition, get the maximum value index under the same index;

[0025] The maximum value indicator is determined as the target indicator.

[0026] Preferably, the target indicator is written into the database by overwriting to ensure the smoothness and consistency of data in the data processing process of the real-time computing framework, including:

[0027] In the process of writing the target indicator into the database, when it is detected that the target indicator conflicts with the data in the database, the conflicting row in the database is deleted; wherein the conflict includes a primary key conflict or a unique index conflict;

[0028] The target index is written into the database after deleting conflicts to avoid data duplication, so as to meet the smoothness and consistency of data in the processing process of the real-time computing framework.

[0029] The second aspect of the present application discloses a data processing device for a real-time computing framework, the device comprising:

[0030] A first acquisition unit is used to acquire the time in the log when the result set is written into the database;

[0031] A reading unit, used for reading data within the time period of the log;

[0032] A constraint unit, used to constrain the data through a unique index mechanism; the unique index mechanism is used to ensure that the value of any column or multiple columns in a table of the database is unique;

[0033] A second acquisition unit is used to obtain a target indicator according to the constrained data and the result set; the target indicator is a maximum value indicator under the same index;

[0034] The writing unit is used to write the target indicator into the database by overwriting, so as to ensure the smoothness and consistency of data in the data processing process of the real-time computing framework.

[0035] Preferably, the first acquisition unit includes:

[0036] The first acquisition module is used to obtain the log start time and the log end time of writing the result set into the database;

[0037] The first determination module is used to determine the in-log time when the result set is written into the database according to the in-log start time and the in-log start time end time.

[0038] Preferably, the reading unit comprises:

[0039] The second acquisition module is used to obtain the current system time and data reporting time;

[0040] A second determination module is used to determine a periodic reading method according to the current system time, the time in the log and the data reporting time;

[0041] The reading module is used to read the data within the time period of the log according to the periodic reading method.

[0042] Preferably, the constraint unit is specifically used for:

[0043] The data is constrained through a unique index mechanism to ensure that the value of any column or multiple columns in the data table is unique.

[0044] Through the above technology, it can be known that the present application discloses a data processing method and device for a real-time computing framework, obtains the time in the log when the result set is written into the database, reads the data within the time in the log, constrains the data through a unique index mechanism, and the unique index mechanism is used to ensure that the value of any column or multiple columns in the table of the database is unique. According to the constrained data and the result set, a target indicator is obtained, and the target indicator is the maximum value indicator under the same index. The target indicator is written into the database by an overwrite write method to ensure the smoothness and consistency of the data in the data processing process of the real-time computing framework. Through the above scheme, in the scenario where data consistency needs to be guaranteed, since the unique index mechanism is used to ensure that the value of any column or multiple columns in the table of the database is unique, that is, there can be no duplicate values, the constrained target indicator is obtained through the unique index mechanism, and the target indicator is written into the database by an overwrite write method. The overwrite write method can delete the conflicting row first when a primary key or unique index conflict is detected, and then insert a new row. The new row will replace the existing conflicting row, and when all dimensions written to the database later are the same as the dimensions in the database, the record will be automatically overwritten, thereby solving the duplication problem of the result set. And because the target indicator is the maximum value indicator under the same index, that is, the correct indicator defined in the actual business is retained under the same index, the smoothness and consistency of the result set can be ensured even under data replay and restart. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 A flowchart of a data processing method for a real-time computing framework disclosed in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a process for reading data within a time period in a log disclosed in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a process for obtaining a target indicator disclosed in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the structure of a data processing device for a real-time computing framework disclosed in an embodiment of the present application;

[0050] Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0053] As can be seen from the background technology, data replay is a process in which the real-time system needs to re-receive the data from the data source received at a previous point in time due to logic modification, incorrect original processing method, etc., and recalculate and rewrite the result set. When the real-time program is stopped, a batch of data may not be fully processed. When the real-time program is restarted again, the previous batch of data will be automatically processed again, and this process will cause all or part of the result set of the batch to be repeated. Therefore, how to solve the problem of duplication of result sets and ensure the smoothness and consistency of result sets is a problem that needs to be solved urgently in this application.

[0054] In order to solve the above problems, the present application discloses a data processing method and device for a real-time computing framework. In a scenario where data consistency needs to be ensured, since the unique index mechanism is used to ensure that the value of any column or multiple columns in the table of the database is unique, that is, there cannot be duplicate values, the constrained target indicator is obtained through the unique index mechanism, and the target indicator is written to the database through an overwrite write method. The overwrite write method can first delete the conflicting row when a primary key or unique index conflict is detected, and then insert a new row. The new row will replace the existing conflicting row. When all dimensions written to the database subsequently are the same as the dimensions in the database, the record will be automatically overwritten, thereby solving the problem of duplication of the result set. And because the target indicator is the maximum value indicator under the same index, that is, the correct indicator defined by the actual business is retained under the same index, so that the smoothness and consistency of the result set can be ensured under data replay and restart. The specific implementation method is specifically described by the following embodiments.

[0055] Based on the above embodiments Figure 1A data processing method for a real-time computing framework is disclosed, and the data processing method for a real-time computing framework mainly includes the following steps:

[0056] S101: Obtain the log time (event time) when the result set is written into the database.

[0057] The event time refers to the time when the user event actually occurs, which is often earlier than the system time. The event time includes the log start time and log end time.

[0058] In S101, the in-log start time and the in-log start time end time of writing the result set into the database are obtained, and the event time of writing the result set into the database is determined according to the in-log start time and the in-log start time end time.

[0059] S102: Read data within event time.

[0060] For example, if the event time of a batch of calculated result sets written to the database is from 12:00 to 12:05, then the data from 12:00 to 12:05 in the database is read first.

[0061] The specific process of reading the data within the log time is as follows Figure 2 Reading the data in the log mainly includes the following steps:

[0062] S201: Obtain the current system time and data reporting time.

[0063] It should be noted that system time refers to the time recorded by the internal clock of the computer system, which is the basis for time tracking and timestamp generation used by the operating system. System time is based on a standard time, such as Coordinated Universal Time (UTC), and can be set to synchronize with external clocks such as the Global Positioning System (GPS) and Network Time Protocol (NTP) to ensure time accuracy and consistency. In a multi-user or network environment, synchronization of system time is critical to ensure the correct sequencing of data and timely processing of events.

[0064] S202: Determine a periodic reading method according to the current system time, event time and data reporting time.

[0065] The periodic reading mode is to read a batch of data at a fixed time interval, for example, the real-time technical framework runs between 12:00 and 15:00, and a batch of data is read every 5 minutes, or the real-time computing framework runs between 9:00 and 13:00, and a batch of data is read every 1 minute, etc. The periodic reading mode is set according to the actual situation, and this application does not make specific restrictions.

[0066] The periodic reading method can help ensure the freshness, continuity, timeliness and accuracy of data, and is suitable for various occasions such as real-time data processing, log collection, and data synchronization.

[0067] To facilitate understanding of how to determine the periodic reading method based on the current system time, eventtime, and data reporting time, an example is given here:

[0068] For example, in a test scenario, if the current system time is 15:00, when the monitoring reports that the data is online at 12:00, the real-time technology framework runs a batch every 5 minutes between 12:00 and 15:00. Running a batch every 5 minutes between 12:00 and 15:00 is a periodic reading method.

[0069] S203: Read the data within the event time according to the periodic reading method.

[0070] In an embodiment of the present application, the data within the event time can be effectively read according to the periodic reading method. Since the periodic reading method can help ensure the freshness, continuity, timeliness and accuracy of the data, the freshness, continuity, timeliness and accuracy of the data read within the event time can be ensured.

[0071] S103: Constraining data through a unique index mechanism; the unique index mechanism is used to ensure that the value of any column or multiple columns in a table of a database is unique.

[0072] In S103, the data is constrained by a unique index mechanism to ensure that the value of any column or multiple columns in the data table is unique.

[0073] It should be noted that the unique index mechanism is a constraint mechanism that is used to ensure that the values ​​of one or more columns in a database table are unique, that is, there cannot be duplicate values. The unique index mechanism plays an important role in database design, especially in scenarios where data consistency needs to be guaranteed.

[0074] S104: Obtain a target indicator based on the constrained data and result set; the target indicator is a maximum value indicator under the same index.

[0075] According to the constrained data and result set, the target indicators are obtained as follows: Figure 3 Obtaining the target index mainly includes the following steps:

[0076] S301: right join the constrained data with the result set.

[0077] Among them, rightjoin is a type of join in database query, which is mainly used to merge data from two tables. When performing a right join, the result set will contain all records in the right table (i.e. the second table being joined), as well as the records in the left table (i.e. the first table being joined) that match the right table. If there are no records in the left table that match the right table, then those fields in the left table will be displayed as null values ​​(NULL) in the result set.

[0078] The advantages of right join are as follows:

[0079] 1. Ensure that all records in the right table are included: Right join can ensure that the result set contains all records in the right table (the second table being joined), which is very useful for analysis and reporting, especially when you need to view all entries in a table, regardless of whether they have matching items in another table;

[0080] 2. Find missing values: Right join allows users to easily identify right table records that have no corresponding relationship in the left table. This helps to find blank or missing parts in the data, so that the cause can be further investigated or the data can be updated;

[0081] 3. Flexible data merging: In some cases, business logic may require that the data integrity of a certain table be given priority. Right join provides this flexibility, allowing the focus of the join to be adjusted according to the needs of the query;

[0082] 4. Auxiliary analysis relationship: By combining rightjoin with other join (JOIN) types (such as leftjoin, innerjoin, etc.), you can have a more comprehensive understanding of the relationship between different data tables, which is particularly important for complex data analysis and decision support systems;

[0083] 5. Simplify query design: In some query scenarios, using right join can simplify the query design because it allows you to build the query directly from the right table instead of starting from the left table first, which may make the logic clearer;

[0084] 6. Applicable to specific business needs: In some business processes, you may need to pay special attention to all instances of a certain set of entities (such as products, services, or customers), even if they currently have no data associated with other entities. Right join is very useful in this case;

[0085] In summary, right join is a powerful tool in database management and query design. It provides an effective way to process and display data, especially in scenarios where all records of a table need to be retained.

[0086] S302: Under the right join condition, obtain the maximum value index under the same index.

[0087] S303: Determine the maximum value indicator as the target indicator.

[0088] In the embodiment of the present application, right join is performed on the constrained data and the result set, and the target indicator is obtained under the condition of right join. Since right join has the advantages of ensuring that all records of the right table are included, finding missing values, flexible data merging, auxiliary analysis of relationships, and simplifying query design, the result of the target indicator obtained is more accurate.

[0089] The method adopted by the real-time computing framework of this solution is to read the data first and then write the subsequent target data into the database by overwriting (replaceinto). For example, if the eventtime start and end time of a batch of calculated result sets written to the database is 12:00 to 12:05, then the data from 12:00 to 12:05 in the database is read first, and right join is performed with the result set. Only the correct target indicator is retained under the same dimension (the correct target indicator here is related to the actual business logic definition. In most scenarios, a larger indicator is retained. In this case, the maximum value indicator is taken). In this way, no matter how the data is replayed and restarted, the result set is smooth.

[0090] It should be noted that data replay refers to the need to re-receive data from a data source that was received at a previous point in time in the real-time computing framework due to logic modification, incorrect original processing method, etc., and recalculate and rewrite the result set.

[0091] A common scenario for data replay is to make up numbers. For some reason, the real-time program stops abnormally, or there is a problem with the processing logic of the processed data. In this case, it is necessary to reset the offset to be consumed by Kafka to the time period before the failure. The process of resetting and consuming the data in the previous time period is data replay.

[0092] Since the real-time program may not have completely processed a batch of data when it stops, when it is restarted, it will automatically process the previous batch of data again. This process will cause all or part of the data in the batch to be repeated, resulting in a larger result set. Restarting is essentially just a replay of a batch of data, allowing the real-time computing framework to meet consistency semantics and solve the problem of real-time result set duplication. In this way, the input data is repeated, and the result set is accurate and meets the consistency requirements.

[0093] The real-time computing framework must first have the ability to replay data and be able to replay Kafka data at any time (such as data existing in Kafka); the real-time computing framework must support idempotent result output, so that the correctness of the result set can be guaranteed when data is replayed or the program is restarted.

[0094] Although the use of replace into to overwrite solves the problem of data duplication, it also brings new problems. The time of storing the offset of the distributed publish-subscribe messaging system (kafka) before data replay is essentially the system time used, while when consuming the data of kafka for calculation, the event time in the log is essentially used. Due to the difference between system time and event time, the essential difference may not be seen when using the incremental update method of insert into. However, when changing to the overwrite writing method of replace into, it will be found that in the first batch of starting the real-time program, the result set index becomes smaller, and the result set image will appear jagged after restarting. This is because insert into is essentially an additional write, and the first batch after the restart needs to be recalculated, which leads to a larger result set. After changing the writing method to replace into, the reason for the smaller result set is that the time of storing the offset is different from the calculation time. In order to solve the problem of the smaller result set, the real-time computing framework of this solution adopts the method of reading the data first and then writing the target data to the database through replace into.

[0095] Among them, Kafka is the most widely used real-time data source, with an offset indicating the offset of the data.

[0096] The specific code example for obtaining the target indicator is as follows:

[0097] spark.read.jdbc(

[0098] dbInfo,

[0099] s"(selecttime_stamp,qudao,page,col,"+

[0100] s"interface_type, click, expose, click_valid, expose_valid, cast(device as char) platform from ${tablename}"+

[0101] s"where time_stamp between ${time_range._1} and ${time_range._2}) t", dbProperties).createOrReplaceTempView("t_tidb")

[0102] / / Here, time_ranger._1 and time_ranger._2 are the minimum event_time and maximum event_time in the output result set

[0103] spark.sql("select

[0104] a.time_stamp time_stamp, a.qudao qudao, a.page page, a.col col, a.interface_type interface_type, a.platform platform,"+

[0105] "case when a.click < b.click then b.click else a.click end as click,"+

[0106] "case when a.expose < b.expose then b.expose else a.expose end as expose,"+

[0107] "case when a.click_valid < b.click_valid then b.click_valid else a.click_valid end as click_valid,"+

[0108] "case when a.expose_valid < b.expose_valid then b.expose_valid else a.expose_valid end as expose_valid"+

[0109] "frombsetTidb a left outerjoint_tidb b on a.time_stamp=b.time_stamp"+

[0110] "and a.qudao=b.qudao and a.page=b.page and a.interface_type=b.interface_type and a.col=b.col and a.platform=b.platform")

[0111] / / Our business implication is that the larger value is the correct indicator.

[0112] S105: Write the target indicator into the database by overwriting (replace into) to ensure the smoothness and consistency of data (Exactly-Once) during the data processing of the real-time computing framework.

[0113] In S105, when writing the target indicator into the database, if it is detected that there is a conflict between the target indicator and the data in the database, the conflicting rows in the database are deleted, and the target indicator is written into the database after the conflict is deleted to avoid data duplication, so as to meet the smoothness and consistency of the data in the real-time computing framework during the processing.

[0114] The conflicts include primary key conflicts or unique index conflicts.

[0115] When any piece of data flows into a distributed system, if the system processes the data exactly once during the entire processing process, without repeated or missed execution, and the processing result meets the system's expectations, the system is considered to meet consistency.

[0116] It should be noted that due to the idempotence of real-time results, the incremental update method cannot be used, such as insert into to append to the database. However, for standard SQL databases such as MySQL and open source distributed relational database (TiDB), replace into needs to be used to overwrite the table (table name). replace into is similar to insert into, but the behavior of replace into is to delete the conflicting row first and then insert a new row when a primary key or unique index conflict is detected. This means that the new row will replace the existing conflicting row. The result table needs to add a unique index of all dimensions. When all dimensions written to the database for the second time are the same as the first time, the record will be automatically overwritten, so as not to cause data duplication.

[0117] The following is a code example of writing the target indicator into the database through replace into:

[0118] dataStream.foreachPartition(part=>{

[0119] Class.forName("com.mysql.jdbc.Driver")

[0120] DriverManager.getConnection(url,user,password)

[0121] valsql=s"REPLACE into${tablename}(column1,column2,column3,column4,metrics1,metrics2)values(?,?,?,?,?,?)"

[0122] valprep=connection.prepareStatement(sql);

[0123] part.foreach(x=>{

[0124] prep.setInt(1,300) ......

[0126] prep.addBatch()

[0127] })

[0128] prep.executeBatch();

[0129] prep.close();

[0130] connection.close().

[0131] To facilitate understanding of the processing logic of the real-time technology framework, the following example is given:

[0132] For example, in the test scenario, the current system time is 15:00, and the real-time technical framework runs a batch every 5 minutes. When it is found that the new logic is launched for the reported data at 12:00, the original calculation logic is invalidated. In this way, the data results calculated from 12:00 to 15:00 are incorrect, so it is necessary to launch a new calculation logic and replay the data from 12:00 to 15:00. At this time, modify the logic, upload the jar package, stop the original real-time job, and reset the kakfa offset to 11:55. It should be noted that it cannot be reset to 12:00 at this time. Although the problem occurred at 12:00, the difference between the system time and eventtime requires at least one more batch to be advanced. Start the real-time job data and replay 11:55 to the latest data. At this time, because the replace into method is used to write the result set, although the database already has more than 3 hours of data, the data with the same dimension and the same unique index will be overwritten by the new data through the unique index mechanism, and there is a logic of reading first and then writing, which ensures that only the more correct target data in the database and the new result set will be retained.

[0133] This solution mainly focuses on supporting idempotent result output in the real-time computing framework, and is committed to solving the problem of repeated data transmission to the result set caused by data replay and restart, so that the result set is not repeated under the premise of repeated input data.

[0134] In the embodiment of the present application, in the scenario where data consistency needs to be ensured, since the unique index mechanism is used to ensure that the value of any column or multiple columns in the table of the database is unique, that is, there cannot be duplicate values, the constrained target indicator is obtained through the unique index mechanism, and the target indicator is written to the database through an overwrite write method. The overwrite write method can first delete the conflicting row when a primary key or unique index conflict is detected, and then insert a new row. The new row will replace the existing conflicting row. When all dimensions written to the database subsequently are the same as the dimensions in the database, the record will be automatically overwritten, thereby solving the problem of duplication of the result set. And because the target indicator is the maximum value indicator under the same index, that is, the correct indicator defined by the actual business is retained under the same index, so that the smoothness and consistency of the result set can be ensured even under data replay and restart.

[0135] Based on the above embodiments Figure 1 A data processing method for a real-time computing framework is disclosed. The embodiment of the present application also discloses a data processing device for a real-time computing framework, such as Figure 4 As shown, the data processing device for the real-time computing framework includes:

[0136] The first acquisition unit 401 is used to acquire the time in the log when the result set is written into the database;

[0137] A reading unit 402, used to read the data in the log time;

[0138] The constraint unit 403 is used to constrain the data through a unique index mechanism; the unique index mechanism is used to ensure that the value of any column or multiple columns in the table of the database is unique;

[0139] The second acquisition unit 404 is used to obtain a target index according to the constrained data and the result set; the target index is a maximum value index under the same index;

[0140] The writing unit 405 is used to write the target indicator into the database by overwriting, so as to ensure the smoothness and consistency of the data in the data processing process of the real-time computing framework.

[0141] Furthermore, the first acquisition unit 401 includes:

[0142] The first acquisition module is used to obtain the log start time and the log end time of writing the result set into the database;

[0143] The first determination module is used to determine the in-log time when the result set is written into the database according to the in-log start time and the in-log start time end time.

[0144] Furthermore, the reading unit 402 includes:

[0145] The second acquisition module is used to obtain the current system time and data reporting time;

[0146] The second determination module is used to determine the periodic reading method according to the current system time, the time in the log and the data reporting time;

[0147] The reading module is used to read the data within the time of the log according to the periodic reading method.

[0148] Furthermore, the constraint unit 403 is specifically used to constrain the data through a unique index mechanism to ensure that the value of any column or multiple columns in the data table is unique.

[0149] Furthermore, the second acquisition unit 404 includes:

[0150] The right join module is used to right join the constrained data with the result set;

[0151] The third acquisition module is used to obtain the maximum value index under the same index under the condition of right connection;

[0152] The third determination module is used to determine the maximum value indicator as the target indicator.

[0153] Furthermore, the writing unit 405 includes:

[0154] A deletion module is used to delete the conflicting rows in the database when a conflict is detected between the target indicator and the data in the database during the process of writing the target indicator into the database; wherein the conflict includes a primary key conflict or a unique index conflict;

[0155] The writing module is used to write the target indicators into the database after deleting conflicts to avoid data duplication and to meet the smoothness and consistency of data in the processing process of the real-time computing framework.

[0156] In the embodiment of the present application, in the scenario where data consistency needs to be ensured, since the unique index mechanism is used to ensure that the value of any column or multiple columns in the table of the database is unique, that is, there cannot be duplicate values, the constrained target indicator is obtained through the unique index mechanism, and the target indicator is written to the database through an overwrite write method. The overwrite write method can first delete the conflicting row when a primary key or unique index conflict is detected, and then insert a new row. The new row will replace the existing conflicting row. When all dimensions written to the database subsequently are the same as the dimensions in the database, the record will be automatically overwritten, thereby solving the problem of duplication of the result set. And because the target indicator is the maximum value indicator under the same index, that is, the correct indicator defined by the actual business is retained under the same index, so that the smoothness and consistency of the result set can be ensured even under data replay and restart.

[0157] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the data processing method for the real-time computing framework as described above.

[0158] The present application also provides an electronic device, whose structural diagram is as follows: Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein the one or more instructions 502 are stored in the memory 501 and are configured to be executed by one or more processors 503 to execute the one or more instructions 502 to perform the above-mentioned data processing method for the real-time computing framework.

[0159] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0160] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0161] The steps in the methods of the embodiments of the present application can be adjusted in order, combined, and deleted according to actual needs.

[0162] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0163] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0164] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method for a real-time computing framework, characterized in that: The method comprises: Get the time in the log when the result set is written to the database; Read the data within the time in the log; The data is constrained by a unique index mechanism; the unique index mechanism is used to ensure that the value of any column or multiple columns in a table of a database is unique; According to the constrained data and result set, a target index is obtained; the target index is a maximum value index under the same index; The target indicator is written into the database by overwriting to ensure the smoothness and consistency of data during the data processing of the real-time computing framework.

2. The method according to claim 1, characterized in that: The time when the result set is written into the database log includes: Get the in-log start time and in-log end time of writing the result set to the database; The in-log time at which the result set is written into the database is determined according to the in-log start time and the in-log start time end time.

3. The method according to claim 1, characterized in that The reading of data within the time period in the log includes: Get the current system time and data reporting time; Determine a periodic reading method according to the current system time, the time in the log and the data reporting time; According to the periodic reading mode, the data within the time in the log is read.

4. The method according to claim 1, characterized in that: The data is constrained by a unique index mechanism, including: The data is constrained through a unique index mechanism to ensure that the value of any column or multiple columns in the data table is unique.

5. The method according to claim 1, characterized in that The target indicators are obtained based on the constrained data and result set, including: Right join the constrained data with the result set; Under the right join condition, get the maximum value index under the same index; The maximum value indicator is determined as the target indicator.

6. The method according to claim 1, characterized in that The target indicator is written into the database by overwriting to ensure the smoothness and consistency of data in the data processing process of the real-time computing framework, including: In the process of writing the target indicator into the database, when it is detected that the target indicator conflicts with the data in the database, the conflicting row in the database is deleted; wherein the conflict includes a primary key conflict or a unique index conflict; The target index is written into the database after deleting conflicts to avoid data duplication, so as to meet the smoothness and consistency of data in the processing process of the real-time computing framework.

7. A data processing device for a real-time computing framework, characterized in that: The device comprises: A first acquisition unit is used to acquire the time in the log when the result set is written into the database; A reading unit, used for reading data within the time period of the log; A constraint unit, used to constrain the data through a unique index mechanism; the unique index mechanism is used to ensure that the value of any column or multiple columns in a table of the database is unique; A second acquisition unit is used to obtain a target indicator according to the constrained data and the result set; the target indicator is a maximum value indicator under the same index; The writing unit is used to write the target indicator into the database by overwriting, so as to ensure the smoothness and consistency of data in the data processing process of the real-time computing framework.

8. The device according to claim 7, characterized in that The first acquisition unit includes: The first acquisition module is used to obtain the log start time and the log end time of writing the result set into the database; The first determination module is used to determine the in-log time when the result set is written into the database according to the in-log start time and the in-log start time end time.

9. The device according to claim 7, characterized in that The reading unit comprises: The second acquisition module is used to obtain the current system time and data reporting time; A second determination module is used to determine a periodic reading method according to the current system time, the time in the log and the data reporting time; The reading module is used to read the data within the time period of the log according to the periodic reading method.

10. The device according to claim 7, characterized in that The constraint unit is specifically used for: The data is constrained through a unique index mechanism to ensure that the value of any column or multiple columns in the data table is unique.

Images