Data processing method and device based on real-time analysis system and storage medium

Abstract

The embodiment of the invention provides a data processing method and device based on a real-time analysis system and a storage medium, time information of each row set of streaming data is acquired according to a preset time attribute type in response to a target operation instruction of the streaming data, and the preset time attribute type comprises event time or processing time; according to the time information of any row of set and preset data survival time, judging whether all data included in any row of set expires; and skipping the row set with all data expired, reading data included in the row set with not all data expired, and executing the target operation. According to the embodiment of the invention, whether the data included in the row sets are all expired or not is judged by taking the row sets as the granularity, the row sets with all expired data are skipped, and the target operation is not carried out, so that the data volume for executing the target operation can be effectively reduced, the processing efficiency is improved, the resource consumption is reduced, and the performance of the target operation in stream-oriented computation is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular, to a data processing method, device and storage medium based on a real-time analysis system. Background Art

[0002] In the field of stream computing, stream data usually carries time attributes, and the stream data is also configured with a time-to-live (TTL). When a row of data reaches the data lifespan after being written, the row of data will become invalid. Stream data can usually be stored in multiple files, each of which includes one or more rows of stream data. Different files may contain different versions of the same row of data. Due to the data lifespan, there may be invalid data in the file, so a merge operation is required to eliminate the invalid data; in addition, data query operations can also be performed from different files.

[0003] In the prior art, when performing operations such as query or merge, it is necessary to scan the data in each file, and then determine whether each data is invalid based on the data survival time. Invalid data is not returned during query, and invalid data is directly eliminated during merge.

[0004] However, in the prior art, as the amount of data increases, the amount of data for operations such as query or merging is too large, resulting in low processing efficiency and a large waste of resources, affecting the performance of streaming computing. Summary of the invention

[0005] The embodiments of the present disclosure provide a data processing method, device and storage medium based on a real-time analysis system to improve the performance of streaming computing.

[0006] In a first aspect, an embodiment of the present disclosure provides a data processing method based on a real-time analysis system, comprising:

[0007] In response to a target operation instruction on the stream data, acquiring time information of each row set of the stream data according to a preset time attribute type, wherein the preset time attribute type includes event time or processing time;

[0008] According to the time information of any row set and the preset data survival time, determine whether all the data included in any row set is expired;

[0009] Skip the row set whose data is all expired, read the data included in the row set whose data is not all expired, and perform the target operation.

[0010] In a second aspect, an embodiment of the present disclosure provides a data processing device based on a real-time analysis system, including:

[0011] an acquisition unit, configured to acquire time information of each row set of the stream data according to a preset time attribute type in response to a target operation instruction on the stream data, wherein the preset time attribute type includes event time or processing time;

[0012] A judging unit, used to judge whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time;

[0013] The execution unit is used to skip the row set whose data is all expired, read the data included in the row set whose data is not all expired, and perform the target operation.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data processing method based on the real-time analysis system as described in the first aspect and various possible designs of the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the data processing method based on the real-time analysis system as described in the first aspect and various possible designs of the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, it implements the data processing method based on the real-time analysis system as described in the first aspect and various possible designs of the first aspect.

[0019] The data processing method, device and storage medium based on the real-time analysis system provided by the embodiments of the present disclosure obtain the time information of each row set of the stream data according to the preset time attribute type in response to the target operation instruction of the stream data, wherein the preset time attribute type includes event time or processing time; judge whether the data included in any row set is all expired according to the time information of any row set and the preset data survival time; skip the row set whose data is all expired, read the data included in the row set whose data is not all expired and perform the target operation. In the embodiments of the present disclosure, by judging whether the data included in the row set is all expired at the granularity of the row set, and skipping the row set whose data is all expired without performing the target operation, the amount of data for performing the target operation can be effectively reduced, the processing efficiency can be improved, the resource consumption can be reduced, and the performance of the target operation in the stream computing can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of a scenario of a data processing method based on a real-time analysis system provided in an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a data processing method based on a real-time analysis system according to an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of a data structure provided for an embodiment of the present disclosure;

[0024] Figure 4 A schematic flow chart of a data processing method based on a real-time analysis system provided by another embodiment of the present disclosure;

[0025] Figure 5 A structural block diagram of a data processing device based on a real-time analysis system provided by an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] In the field of stream computing, such as real-time analysis systems, stream data usually carries time attributes, and the stream data is also configured with a time-to-live (TTL). When a row of data reaches the data survival time after being written, the row of data will become invalid. For example, a user can set the timestamp of a column in a data table to record data. When a user writes a row of data, the timestamp of the row of data will also be recorded in the data table at the same time. Assuming that the timestamp of a row of data written by the user is 2024-01-01 01:00:00, and the data survival time is 3600 seconds, the row of data will become invalid when the time reaches 2024-01-01 02:00:00.

[0029] In addition, stream data can usually be stored in multiple files, each of which contains one or more rows of stream data. Different files may contain different versions of the same row of data. Due to the data survival time, there may be invalid data in the file, so a merge operation is required to eliminate the invalid data; in addition, data query operations can also be performed from different files.

[0030] In the prior art, when performing operations such as query or merge, it is necessary to scan the data in each file, and then determine whether each data is invalid based on the data survival time. Invalid data is not returned during query, and invalid data is directly eliminated during merge. For example, the data survival time filter is used to determine whether each data is invalid.

[0031] However, in the prior art, as the amount of data increases, the amount of data that needs to be scanned for operations such as queries or merges also increases accordingly. If each data is scanned and then judged whether the data is invalid, the processing efficiency is low and a large amount of resources is wasted, affecting the performance of streaming computing.

[0032] In order to solve the above technical problems, the embodiments of the present disclosure provide a data processing method based on a real-time analysis system, which can determine whether the data is expired at a larger granularity, rather than determining whether each row of data is expired one by one. This can effectively reduce the amount of data for executing the target operation, improve processing efficiency, reduce resource consumption, and improve the performance of the target operation in streaming computing.

[0033] The present disclosure provides a data processing method based on a real-time analysis system. Figure 1 As shown, in response to a target operation instruction on the stream data, the time information of each row set of the stream data is obtained according to a preset time attribute type, wherein the preset time attribute type includes event time or processing time; based on the time information of any row set and the preset data survival time, it is determined whether all the data included in any row set is expired; the row set with all expired data is skipped, the data included in the row set with not all expired data is read and the target operation is performed.

[0034] The data processing method based on the real-time analysis system disclosed in the present invention will be described in detail below in conjunction with specific embodiments.

[0035] refer to Figure 2 , Figure 2 A flow chart of a data processing method based on a real-time analysis system provided in an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device or a server, and the data processing method based on a real-time analysis system includes:

[0036] S201. In response to a target operation instruction on stream data, obtain time information of each row set of the stream data according to a preset time attribute type, wherein the preset time attribute type includes event time or processing time.

[0037] In this embodiment, in the scenario of streaming computing, streaming data is usually stored in a specific data structure, for example, in multiple files, each file including one or more rows of streaming data, where each row of data may include a key, a value, and a timestamp (that is, event time); and different versions of data with the same key may exist between different files (with different timestamps and possibly different values).

[0038] Optionally, in a data structure, a row set (Rowset) can be used to store metadata of the data included in each file and the storage path of the data included in each file (for example, which file is included, and the file address, etc.), where the metadata can include statistical information of the data, and each file can be stored in a distributed file storage system, such as HDFS. Figure 3 As shown, in a data structure, the data table Table of stream data can be divided into multiple slices Tablet, each slice Tablet can be divided into multiple rowsets, each row set Rowset can include metadata of multiple rows of data and storage paths of multiple rows of data. Of course, in another optional embodiment, each row set Rowset can also include multiple rows of real data instead of metadata and data storage paths. Furthermore, the file where the multiple rows of data of each row set Rowset are located can also further include different data blocks Page

[0039] Since the data survival time (TTL) can be configured for the streaming data in advance, any row of data will become invalid when the data survival time is reached after being written, and the subsequent merging (Compaction) operation needs to eliminate the row of data, or the subsequent query (Query) operation will no longer display the row of data if the row of data is queried, or other operations do not require the participation of the row of data. Considering that the prior art usually scans the data before judging whether the data is invalid, which wastes time and resources, in order to optimize the merging, querying and other operations, in this embodiment, after receiving the target operation instruction for the streaming data and triggering the target operation, it can be judged whether all the data included in the row set is expired at the granularity of the row set Rowset, and the judgment process needs to rely on the time information of each row set of the streaming data, so it is necessary to obtain the time information of each row set of the streaming data.

[0040] The time attribute types of data usually include event time and processing time. The event time is the time when the row of data is generated, which can be the timestamp of the time when the row of data is generated; the processing time is the time when the row of data is processed, such as the time when the row of data is processed by the message queue or stored. When judging whether all the data included in the row set is expired based on different time attribute types and taking the row set as the granularity, the time information of each row set of the stream data is different. Therefore, the time information of each row set of the stream data can be obtained according to the preset time attribute type.

[0041] Optionally, if the preset time attribute type is event time, the maximum event time and minimum event time of the data included in each row set of the streaming data can be obtained as the time information of each row set, where the maximum event time and minimum event time of the data can be obtained from the metadata of the row set, where the metadata includes statistical information on the event time of the data included in the row set.

[0042] If the preset time attribute type is processing time, the creation time of each row set of the stream data can be obtained as the time information of each row set. What needs to be satisfied is that the maximum event time of the data included in any row set of the stream data is less than the creation time of the row set, that is, the data included in any row set of the stream data is not data that will take effect in the future. For example, the data written today will take effect tomorrow, and the creation time of all data is no later than the creation time of the row set. The creation of the row set is to create the row set when the data included in the row set is written to the disk in the form of a file.

[0043] Of course, in the case of other time attribute types, any other time information may be used as the time information of each row set of stream data, as long as it can be determined whether all the data included in any row set is expired.

[0044] S202: Determine whether all data included in any row set are expired based on the time information of any row set and the preset data survival time.

[0045] In this embodiment, after obtaining the time information of each row set, a specific judgment strategy can be used to judge whether all the data included in any row set is expired according to the time information of any row set and the preset data survival time. Different judgment strategies correspond to different preset time attribute types.

[0046] Optionally, if the preset time attribute type is event time, the time information of each row set of the stream data may include the maximum event time of the data included in each row set, and of course may also include the minimum event time. Therefore, judgment can be made based on the maximum event time and the data survival time. Specifically, for any row set, if the maximum event time of the data included in any row set is less than the current time minus the data survival time, it means that the latest created row of data in the row set has expired, and further means that the data in the row set that was created earlier than the latest created row of data has also expired. Therefore, it can be determined that all the data included in the row set is expired.

[0047] Optionally, if the preset time attribute type is processing time, and the maximum event time of the data included in any row set of the stream data is less than the creation time of the row set, then the time information of each row set of the stream data is the creation time of each row set, and the judgment can be made based on the creation time of each row set and the data survival time. Specifically, for any row set, if the creation time of any row set is less than the current time minus the data survival time, it means that the event time of all data in the row set is less than the current time minus the data survival time. Therefore, it can be determined that all data included in the row set is expired.

[0048] Of course, in this embodiment, any other feasible judgment strategy can be used to judge whether all the data included in any row set is expired according to the time information of any row set and the preset data survival time. Of course, any other feasible method can be used to judge whether all the data included in any row set is expired, and this is not limited in this embodiment.

[0049] S203: Skip the row set whose data is completely expired, read the data included in the row set whose data is not completely expired, and perform the target operation.

[0050] In this embodiment, in the above steps, it can be screened out which row sets include data that are all expired. On this basis, for the row sets with all expired data, there is no need to read the data or perform the target operation. Therefore, skipping the row sets with all expired data can greatly improve processing efficiency and reduce resource consumption. It is only necessary to read the data included in the row sets with not all expired data and perform the target operation, which improves the performance of executing the target operation, especially the performance of query and merge. That is, when querying, there is no need to scan the data included in the row sets with all expired data. It is sufficient to query from the data included in the row sets with not all expired data. When merging, the data included in the row sets with all expired data can be directly discarded.

[0051] Optionally, when reading the data included in the row set whose data is not completely expired and performing the target operation, the data can be obtained according to the storage path of the data in the row set. For example, the storage path of the data includes the file where the data is located and the file address, and the data can be obtained by reading the file according to the file address. Of course, in this embodiment, data can also be obtained by other methods, which will not be described here.

[0052] The data processing method based on the real-time analysis system provided in this embodiment obtains the time information of each row set of the stream data according to the preset time attribute type in response to the target operation instruction of the stream data, wherein the preset time attribute type includes event time or processing time; determines whether the data included in any row set is all expired according to the time information of any row set and the preset data survival time; skips the row set whose data is all expired, reads the data included in the row set whose data is not all expired and performs the target operation. In this embodiment, by judging whether the data included in the row set is all expired at the granularity of the row set, and skipping the row set whose data is all expired without performing the target operation, the amount of data for performing the target operation can be effectively reduced, the processing efficiency can be improved, the resource consumption can be reduced, and the performance of the target operation in the stream computing can be improved.

[0053] Based on any of the above embodiments, when reading the data included in the row set whose data is not completely expired and performing the target operation in S203, similar optimization can also be performed, such as Figure 4 As shown, it may specifically include:

[0054] S301, reading a target file containing data included in a row set in which all data are not expired;

[0055] S302, obtaining the maximum event time and the minimum event time of the data included in each data segment in the target file;

[0056] S303: If the maximum event time of the data included in any data segment is less than the current time minus the data survival time, it is determined that all the data included in any data segment is expired;

[0057] S304: Skip the data segments whose data are all expired, and perform the target operation on the data included in the data segments whose data are not all expired.

[0058] In this embodiment, the file can also be divided into different data segments (Pages), and statistical information of the data included in different data segments can be obtained and stored in the file, which may include statistical information of the maximum event time and minimum event time of the data included in each data segment.

[0059] Furthermore, after reading the target file where the data included in the row set whose data is not completely expired is located, the maximum event time and the minimum event time of the data included in each data segment in the target file can be obtained from the target file, and then the data included in any data segment can be judged based on the maximum event time of the data included in any data segment and the data survival time. Similar to the above embodiment, if the maximum event time of the data included in any data segment is less than the current time minus the data survival time, it means that the latest created row of data in the data segment has expired, and then it means that the data created earlier than the latest created row of data in the data segment has also expired, so it can be determined that the data included in the row of data segment is all expired. Then, the data segment with all expired data can be skipped and the target operation can be performed on the data included in the data segment with not all expired data. Through the further filtering of the data segment granularity in this embodiment, the amount of data for executing the target operation can be further reduced, the processing efficiency can be improved, the resource consumption can be reduced, and the performance of the target operation in the streaming computing can be improved.

[0060] Based on any of the above embodiments, the target operation instruction may be a query instruction, that is, the stream data needs to be queried. When the query conditions of the query instruction include the target time range, the optimization may also be performed using the method provided in the above embodiments.

[0061] In an optional embodiment, in the case where the preset time attribute type is event time, if the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then the time intersection between the target time range and the current data survival time range is determined, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time with a data survival time interval with the current time interval, for example, if the current time is 12:00 and the data survival time is 2 hours, then the first historical time is 10:00, and the current data survival time range is 10:00 to 12:00;

[0062] If there is no time intersection between the target time range and the current data survival time range, it means that all data in the target time range has expired, and all row sets of the stream data can be skipped, that is, all row sets are not queried, and the query result corresponding to the query instruction is directly determined to be empty;

[0063] If there is a time intersection between the target time range and the current data survival time range, it means that the data in the time intersection has not expired. You only need to query the data in the time intersection, and there is no need to query the data outside the time intersection. As for which row sets of stream data all the data are outside the time intersection, you can compare the maximum event time or the minimum event time of each row set of stream data with the time intersection. Specifically, if the maximum event time of any row set of stream data is less than the start time of the time intersection, or the minimum event time of any row set of stream data is greater than the end time of the time intersection, it means that all the data included in the row set is outside the time intersection, and the row set can be skipped.

[0064] In another optional embodiment, in the case where the preset time attribute type is processing time, if the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then the time intersection between the target time range and the current data survival time range can also be determined, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time with a data survival time interval with the current time interval, for example, if the current time is 12:00 and the data survival time is 2 hours, then the first historical time is 10:00, and the current data survival time range is 10:00-12:00;

[0065] If there is no time intersection between the target time range and the current data survival time range, it means that all data in the target time range has expired, and all row sets of the stream data can be skipped, that is, all row sets are not queried, and the query result corresponding to the query instruction is directly determined to be empty;

[0066] If there is a time intersection between the target time range and the current data survival time range, it means that the data in the time intersection has not expired. You only need to query the data in the time intersection, and there is no need to query the data outside the time intersection. As for which row sets of stream data all the data are outside the time intersection, this can be determined by comparing the relationship between the creation time of each row set of stream data and the time intersection. Specifically, if the creation time of any row set of stream data is less than the start time of the time intersection, it means that all the data included in the row set was created before the start time of the time intersection. There is no need to query all the data included in the row set, and the row set can be skipped.

[0067] The above embodiment is a further optimization for query operations. In addition to being able to skip row sets with all expired data at the row set granularity, it also takes into account the target time range specified by the query condition in the query instruction. For row sets outside the target time range, they can be skipped regardless of whether they are expired, effectively reducing the amount of data for executing query operations, improving processing efficiency, reducing resource consumption, and improving query performance in streaming computing.

[0068] On the basis of any of the above embodiments, the user may set the preset time attribute type, thereby determining the corresponding judgment strategy. Specifically, the preset time attribute type may be set in response to a time attribute type setting instruction.

[0069] Alternatively, the preset time attribute type may be set according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set, that is, according to whether the data included in any row set of the stream data includes data that will take effect in the future. Specifically, if the maximum event time of the data included in any row set of the stream data is less than the creation time of the row set, it means that the data included in any row set of the stream data does not include data that will take effect in the future, and the preset time attribute type may be determined to be processing time; if the maximum event time of the data included in at least one row set of the stream data is not less than the creation time of the row set, it means that the data included in at least one row set of the stream data includes data that will take effect in the future, and the preset time attribute type may be determined to be event time.

[0070] Optionally, the above judgment strategies may be used in combination. For example, a part of the row sets adopts the judgment strategy corresponding to the event time, and another part of the row sets adopts the judgment strategy corresponding to the processing time; or, the judgment strategy corresponding to the processing time is adopted first, and the judgment strategy corresponding to the event time is adopted when no judgment can be made.

[0071] Corresponding to the data processing method based on the real-time analysis system in the above embodiment, Figure 5 This is a structural block diagram of a data processing device based on a real-time analysis system provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 5 The data processing device 500 based on the real-time analysis system includes: an acquisition unit 501, a judgment unit 502, and an execution unit 503.

[0072] The acquisition unit 501 is used to respond to the target operation instruction on the stream data and acquire the time information of each row set of the stream data according to the preset time attribute type, wherein the preset time attribute type includes event time or processing time;

[0073] A judging unit 502, configured to judge whether all data included in any row set are expired according to the time information of any row set and the preset data survival time;

[0074] The execution unit 503 is used to skip the row set whose data is completely expired, read the data included in the row set whose data is not completely expired, and perform the target operation.

[0075] In one or more embodiments of the present disclosure, when acquiring the time information of each row set of the stream data according to a preset time attribute type, the acquiring unit 501 is configured to:

[0076] If the preset time attribute type is event time, obtaining the maximum event time of the data included in each row set of the stream data as a kind of time information of each row set;

[0077] Correspondingly, when acquiring the time information of each row set of the stream data according to the preset time attribute type, the acquiring unit 501 is further configured to:

[0078] Obtaining the minimum event time of the data included in each row set of the stream data as another time information of each row set;

[0079] When judging whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time, the judging unit 502 is used to:

[0080] If the maximum event time of the data included in any row set is less than the current time minus the data survival time, it is determined that all the data included in any row set is expired.

[0081] In one or more embodiments of the present disclosure, the determining unit 502 is further configured to:

[0082] If the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then determine the time intersection between the target time range and the current data survival time range, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time that is separated from the current time interval by the data survival time;

[0083] If there is no time intersection, all row sets of the stream data are skipped, and the query result corresponding to the query instruction is determined to be empty; or

[0084] If there is a time intersection, compare the relationship between the maximum event time or the minimum event time of each row set of the stream data and the time intersection;

[0085] If the maximum event time of any row set of the stream data is less than the start time of the time intersection, or the minimum event time of any row set of the stream data is greater than the end time of the time intersection, then the row set is skipped.

[0086] In one or more embodiments of the present disclosure, when acquiring the time information of each row set of the stream data according to a preset time attribute type, the acquiring unit 501 is configured to:

[0087] If the preset time attribute type is processing time, and the maximum event time of data included in any row set of the stream data is less than the creation time of the row set, then the creation time of each row set of the stream data is obtained as the time information of each row set;

[0088] Accordingly, when judging whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time, the judging unit 502 is used to:

[0089] If the creation time of any row set is less than the current time minus the data survival time, it is determined that all data included in the row set is expired.

[0090] In one or more embodiments of the present disclosure, the determining unit 502 is further configured to:

[0091] If the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then determine the time intersection between the target time range and the current data survival time range, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time that is separated from the current time interval by the data survival time;

[0092] If there is no time intersection, all row sets of the stream data are skipped, and the query result corresponding to the query instruction is determined to be empty; or

[0093] If there is a time intersection, compare the relationship between the creation time of each row set of the stream data and the time intersection;

[0094] If the creation time of any row set of the stream data is less than the start time of the time intersection, the row set is skipped.

[0095] In one or more embodiments of the present disclosure, when reading the data included in the row set whose data is not completely expired and performing the target operation, the execution unit 503 is used to:

[0096] Read the target file where the data included in the row set where all data is not expired is located;

[0097] Obtain the maximum event time and the minimum event time of the data included in each data segment in the target file;

[0098] If the maximum event time of the data included in any data segment is less than the current time minus the data survival time, it is determined that all the data included in any data segment is expired;

[0099] The data segments whose data are all expired are skipped, and the target operation is performed on the data included in the data segments whose data are not all expired.

[0100] In one or more embodiments of the present disclosure, the acquiring unit 501 is further configured to:

[0101] In response to the time attribute type setting instruction, setting the preset time attribute type; or

[0102] The preset time attribute type is set according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set.

[0103] In one or more embodiments of the present disclosure, when the acquisition unit 501 sets the preset time attribute type according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set, it is used to:

[0104] If the maximum event time of the data included in any row set of the stream data is less than the creation time of the row set, the preset time attribute type is determined to be the processing time; or, if the maximum event time of the data included in at least one row set of the stream data is not less than the creation time of the row set, the preset time attribute type is determined to be the event time.

[0105] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0106] refer to Figure 6, which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure, and the electronic device 600 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0107] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0108] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0109] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0110] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0111] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0112] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0113] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0116] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0117] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] In a first aspect, according to one or more embodiments of the present disclosure, a data processing method based on a real-time analysis system is provided, comprising:

[0119] In response to a target operation instruction on the stream data, acquiring time information of each row set of the stream data according to a preset time attribute type, wherein the preset time attribute type includes event time or processing time;

[0120] According to the time information of any row set and the preset data survival time, determine whether all the data included in any row set is expired;

[0121] Skip the row set whose data is all expired, read the data included in the row set whose data is not all expired, and perform the target operation.

[0122] According to one or more embodiments of the present disclosure, the step of acquiring time information of each row set of the stream data according to a preset time attribute type includes:

[0123] If the preset time attribute type is event time, obtaining the maximum event time of the data included in each row set of the stream data as a kind of time information of each row set;

[0124] Accordingly, judging whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time includes:

[0125] If the maximum event time of the data included in any row set is less than the current time minus the data survival time, it is determined that all the data included in any row set is expired.

[0126] According to one or more embodiments of the present disclosure, the acquiring time information of each row set of the stream data according to a preset time attribute type further includes:

[0127] Obtaining the minimum event time of the data included in each row set of the stream data as another time information of each row set;

[0128] The method further comprises:

[0129] If the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then determine the time intersection between the target time range and the current data survival time range, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time that is separated from the current time interval by the data survival time;

[0130] If there is no time intersection, all row sets of the stream data are skipped, and the query result corresponding to the query instruction is determined to be empty; or

[0131] If there is a time intersection, compare the relationship between the maximum event time or the minimum event time of each row set of the stream data and the time intersection;

[0132] If the maximum event time of any row set of the stream data is less than the start time of the time intersection, or the minimum event time of any row set of the stream data is greater than the end time of the time intersection, then the row set is skipped.

[0133] According to one or more embodiments of the present disclosure, the step of acquiring time information of each row set of the stream data according to a preset time attribute type includes:

[0134] If the preset time attribute type is processing time, and the maximum event time of data included in any row set of the stream data is less than the creation time of the row set, then the creation time of each row set of the stream data is obtained as the time information of each row set;

[0135] Accordingly, judging whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time includes:

[0136] If the creation time of any row set is less than the current time minus the data survival time, it is determined that all data included in the row set is expired.

[0137] According to one or more embodiments of the present disclosure, the method further includes:

[0138] If the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then determine the time intersection between the target time range and the current data survival time range, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time that is separated from the current time interval by the data survival time;

[0139] If there is no time intersection, all row sets of the stream data are skipped, and the query result corresponding to the query instruction is determined to be empty; or

[0140] If there is a time intersection, compare the relationship between the creation time of each row set of the stream data and the time intersection;

[0141] If the creation time of any row set of the stream data is less than the start time of the time intersection, the row set is skipped.

[0142] According to one or more embodiments of the present disclosure, reading data included in a row set whose data is not completely expired and performing a target operation includes:

[0143] Read the target file where the data included in the row set where all data is not expired is located;

[0144] Obtain the maximum event time and the minimum event time of the data included in each data segment in the target file;

[0145] If the maximum event time of the data included in any data segment is less than the current time minus the data survival time, it is determined that all the data included in any data segment is expired;

[0146] The data segments whose data are all expired are skipped, and the target operation is performed on the data included in the data segments whose data are not all expired.

[0147] According to one or more embodiments of the present disclosure, the method further includes:

[0148] In response to the time attribute type setting instruction, setting the preset time attribute type; or

[0149] The preset time attribute type is set according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set.

[0150] According to one or more embodiments of the present disclosure, the setting of the preset time attribute type according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set includes:

[0151] If the maximum event time of the data included in any row set of the stream data is less than the creation time of the row set, the preset time attribute type is determined to be the processing time; or, if the maximum event time of the data included in at least one row set of the stream data is not less than the creation time of the row set, the preset time attribute type is determined to be the event time.

[0152] In a second aspect, according to one or more embodiments of the present disclosure, a data processing device based on a real-time analysis system is provided, comprising:

[0153] an acquisition unit, configured to acquire time information of each row set of the stream data according to a preset time attribute type in response to a target operation instruction on the stream data, wherein the preset time attribute type includes event time or processing time;

[0154] A judging unit, used to judge whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time;

[0155] The execution unit is used to skip the row set whose data is all expired, read the data included in the row set whose data is not all expired, and perform the target operation.

[0156] According to one or more embodiments of the present disclosure, when the acquisition unit acquires the time information of each row set of the stream data according to a preset time attribute type, it is configured to:

[0157] If the preset time attribute type is event time, obtaining the maximum event time of the data included in each row set of the stream data as a kind of time information of each row set;

[0158] Accordingly, when the judgment unit judges whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time, it is used to:

[0159] If the maximum event time of the data included in any row set is less than the current time minus the data survival time, it is determined that all the data included in any row set is expired.

[0160] According to one or more embodiments of the present disclosure, when the acquisition unit acquires the time information of each row set of the stream data according to the preset time attribute type, it is further configured to:

[0161] Obtaining the minimum event time of the data included in each row set of the stream data as another time information of each row set;

[0162] The judging unit is also used for:

[0163] If the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then determine the time intersection between the target time range and the current data survival time range, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time that is separated from the current time interval by the data survival time;

[0164] If there is no time intersection, all row sets of the stream data are skipped, and the query result corresponding to the query instruction is determined to be empty; or

[0165] If there is a time intersection, compare the relationship between the maximum event time or the minimum event time of each row set of the stream data and the time intersection;

[0166] If the maximum event time of any row set of the stream data is less than the start time of the time intersection, or the minimum event time of any row set of the stream data is greater than the end time of the time intersection, then the row set is skipped.

[0167] According to one or more embodiments of the present disclosure, when the acquisition unit acquires the time information of each row set of the stream data according to a preset time attribute type, it is configured to:

[0168] If the preset time attribute type is processing time, and the maximum event time of data included in any row set of the stream data is less than the creation time of the row set, then the creation time of each row set of the stream data is obtained as the time information of each row set;

[0169] Accordingly, when the judgment unit judges whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time, it is used to:

[0170] If the creation time of any row set is less than the current time minus the data survival time, it is determined that all data included in the row set is expired.

[0171] According to one or more embodiments of the present disclosure, the judging unit is further configured to:

[0172] If the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then determine the time intersection between the target time range and the current data survival time range, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time that is separated from the current time interval by the data survival time;

[0173] If there is no time intersection, all row sets of the stream data are skipped, and the query result corresponding to the query instruction is determined to be empty; or

[0174] If there is a time intersection, compare the relationship between the creation time of each row set of the stream data and the time intersection;

[0175] If the creation time of any row set of the stream data is less than the start time of the time intersection, the row set is skipped.

[0176] According to one or more embodiments of the present disclosure, when reading data included in a row set whose data is not completely expired and performing a target operation, the execution unit is configured to:

[0177] Read the target file where the data included in the row set where all data is not expired is located;

[0178] Obtain the maximum event time and the minimum event time of the data included in each data segment in the target file;

[0179] If the maximum event time of the data included in any data segment is less than the current time minus the data survival time, it is determined that all the data included in any data segment is expired;

[0180] The data segments whose data are all expired are skipped, and the target operation is performed on the data included in the data segments whose data are not all expired.

[0181] According to one or more embodiments of the present disclosure, the acquisition unit is further configured to:

[0182] In response to the time attribute type setting instruction, setting the preset time attribute type; or

[0183] The preset time attribute type is set according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set.

[0184] According to one or more embodiments of the present disclosure, when the acquisition unit sets the preset time attribute type according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set, it is used to:

[0185] If the maximum event time of the data included in any row set of the stream data is less than the creation time of the row set, the preset time attribute type is determined to be the processing time; or, if the maximum event time of the data included in at least one row set of the stream data is not less than the creation time of the row set, the preset time attribute type is determined to be the event time.

[0186] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0187] The memory stores computer-executable instructions;

[0188] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data processing method based on the real-time analysis system as described in the first aspect and various possible designs of the first aspect.

[0189] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the data processing method based on the real-time analysis system as described in the first aspect and various possible designs of the first aspect is implemented.

[0190] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the data processing method based on the real-time analysis system as described in the first aspect and various possible designs of the first aspect is implemented.

[0191] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0192] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0193] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data processing method based on a real-time analysis system, characterized in that: include: In response to a target operation instruction on the stream data, acquiring time information of each row set of the stream data according to a preset time attribute type, wherein the preset time attribute type includes event time or processing time; According to the time information of any row set and the preset data survival time, determine whether all the data included in any row set is expired; Skip the row set whose data is all expired, read the data included in the row set whose data is not all expired, and perform the target operation.

2. The method according to claim 1, characterized in that The acquiring the time information of each row set of the stream data according to the preset time attribute type includes: If the preset time attribute type is event time, obtaining the maximum event time of the data included in each row set of the stream data as a kind of time information of each row set; Accordingly, judging whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time includes: If the maximum event time of the data included in any row set is less than the current time minus the data survival time, it is determined that all the data included in any row set is expired.

3. The method according to claim 2, characterized in that The step of acquiring the time information of each row set of the stream data according to the preset time attribute type further includes: Obtaining the minimum event time of the data included in each row set of the stream data as another time information of each row set; The method further comprises: If the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then determine the time intersection between the target time range and the current data survival time range, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time that is separated from the current time interval by the data survival time; If there is no time intersection, all row sets of the stream data are skipped, and the query result corresponding to the query instruction is determined to be empty; or If there is a time intersection, compare the relationship between the maximum event time or the minimum event time of each row set of the stream data and the time intersection; If the maximum event time of any row set of the stream data is less than the start time of the time intersection, or the minimum event time of any row set of the stream data is greater than the end time of the time intersection, then the row set is skipped.

4. The method according to claim 1, characterized in that The acquiring the time information of each row set of the stream data according to the preset time attribute type includes: If the preset time attribute type is processing time, and the maximum event time of data included in any row set of the stream data is less than the creation time of the row set, then the creation time of each row set of the stream data is obtained as the time information of each row set; Accordingly, judging whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time includes: If the creation time of any row set is less than the current time minus the data survival time, it is determined that all data included in the row set is expired.

5. The method according to claim 4, characterized in that The method further comprises: If the target operation instruction is a query instruction, and the query condition of the query instruction includes a target time range, then determine the time intersection between the target time range and the current data survival time range, wherein the current data survival time range is a time range from a first historical time to a current time, and the first historical time is a historical time that is separated from the current time interval by the data survival time; If there is no time intersection, all row sets of the stream data are skipped, and the query result corresponding to the query instruction is determined to be empty; or If there is a time intersection, compare the relationship between the creation time of each row set of the stream data and the time intersection; If the creation time of any row set of the stream data is less than the start time of the time intersection, the row set is skipped.

6. The method according to any one of claims 1 to 5, characterized in that: The step of reading the data included in the row set whose data are not all expired and performing the target operation includes: Read the target file where the data included in the row set where all data is not expired is located; Obtain the maximum event time and the minimum event time of the data included in each data segment in the target file; If the maximum event time of the data included in any data segment is less than the current time minus the data survival time, it is determined that all the data included in any data segment is expired; The data segments whose data are all expired are skipped, and the target operation is performed on the data included in the data segments whose data are not all expired.

7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to the time attribute type setting instruction, setting the preset time attribute type; or The preset time attribute type is set according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set.

8. The method according to claim 7, characterized in that The setting of the preset time attribute type according to the relationship between the maximum event time of the data included in any row set of the stream data and the creation time of the row set includes: If the maximum event time of the data included in any row set of the stream data is less than the creation time of the row set, the preset time attribute type is determined to be the processing time; or, if the maximum event time of the data included in at least one row set of the stream data is not less than the creation time of the row set, the preset time attribute type is determined to be the event time.

9. A data processing device based on a real-time analysis system, characterized in that: include: an acquisition unit, configured to acquire time information of each row set of the stream data according to a preset time attribute type in response to a target operation instruction on the stream data, wherein the preset time attribute type includes event time or processing time; A judging unit, used to judge whether all the data included in any row set are expired according to the time information of any row set and the preset data survival time; The execution unit is used to skip the row set whose data is all expired, read the data included in the row set whose data is not all expired, and perform the target operation.

10. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 8 is implemented.

Images