Data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a time version and a first data version corresponding to a periodic task when the periodic task is triggered, and progressively increasing a second data version of newly written data based on the first data version when the data is newly written, the time version is used for representing a timestamp when the periodic task is triggered, and the first data version is used for representing the highest version of the written data when the periodic task is triggered; determining a snapshot data set corresponding to the periodic task based on the time version, and determining an incremental data set corresponding to the periodic task in parallel according to data of a second data version if it is determined that the second data version higher than the first data version exists in the process of determining the snapshot data set; and determining a task result corresponding to the periodic task according to the snapshot data set and the incremental data set. According to the method and the device, the calculation efficiency in a high-concurrency conflict scene is improved, and the reliability of cross-cycle time sequence data aggregation calculation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, cross-period time-series data aggregation calculation faces complex concurrency challenges. When the statistical time window ends, the system needs to perform a large amount of aggregation calculation, at this time, new data streams are still continuously written, causing inconsistency of the calculation benchmark data. In order to solve the above problems, a pessimistic lock mechanism or an optimistic lock mechanism is usually used to solve concurrency conflicts. However, the pessimistic lock mechanism needs to lock data, but if there is new written data, it will cause mutual blockage of read and write operations, which seriously limits the concurrency throughput; the optimistic lock mechanism does not need to lock data, but when new data is written, it needs to recalculate the full amount of data, and in a high concurrency conflict scenario, a large amount of invalid calculation overhead will be generated. Obviously, the above two mechanisms for solving concurrency conflicts seriously affect the reliability of cross-period time-series data aggregation calculation. SUMMARY

[0003] The present application provides a data processing method, device, electronic equipment and storage medium to solve the low reliability problem of cross-period time-series data aggregation calculation in the prior art.

[0004] In a first aspect, the present application provides a data processing method, comprising: When a periodic task is triggered, determining a time version corresponding to the periodic task and a first data version, and when new data is written, incrementing a second data version of the new data based on the first data version, the time version being used to represent a time stamp when the periodic task is triggered, and the first data version being used to represent the highest version of the data written when the periodic task is triggered; Determining a snapshot data set corresponding to the periodic task based on the time version, and in the process of determining the snapshot data set, if it is determined that there is a second data version higher than the first data version, determining an incremental data set corresponding to the periodic task in parallel according to the data of the second data version; Determining a task result corresponding to the periodic task according to the snapshot data set and the incremental data set.

[0005] In an optional implementation, the determining an incremental data set corresponding to the periodic task in parallel according to the data of the second data version comprises: Determining a target aggregation function required for executing the periodic task; When the target aggregation function meets a first preset condition, the incremental data set corresponding to the periodic task is determined in parallel according to the data of the second data version, and the first preset condition includes that the target aggregation function is a combinable and commutative function.

[0006] In an optional implementation, the determining the task result corresponding to the periodic task according to the snapshot data set and the incremental data set includes: When the target aggregation function meets the first preset condition, the snapshot data set and the incremental data are aggregated by using the target aggregation function to obtain the task result corresponding to the periodic task. When the target aggregation function does not meet the first preset condition, the snapshot data set is segmented to obtain a plurality of snapshot data segments. A target feature corresponding to each snapshot data segment is determined, and the target feature is used to describe the distribution feature of all snapshot data in the snapshot data segment. The task result corresponding to the periodic task is determined according to all obtained target features.

[0007] In an optional implementation, the periodic task is used to count usage data of a target device. The determining the incremental data set corresponding to the periodic task in parallel according to the data of the second data version when the target aggregation function meets the first preset condition includes: When the target aggregation function meets the first preset condition, it is determined whether the target device needs to report the usage data of the target device within a first time period. When it is determined that the target device needs to report the usage data of the target device within the first time period, the incremental data set corresponding to the periodic task is determined in parallel according to the data of the second data version, and the trigger time of the periodic task is located within the first time period.

[0008] In an optional implementation, the determining that the target device needs to report the usage data of the target device within the first time period includes: A historical usage data set reported by the target device is obtained, and each historical usage data in the historical usage data set corresponds to a historical reporting time. The historical usage data set and the historical reporting time corresponding to each historical usage data in the historical usage data set are input into a preset prediction model, so that a preset prediction model outputs a first probability, and the first probability is used to represent the probability that the target device reports the usage data of the target device within the first time period. determining that the target device needs to report usage data of the target device in the first time period when the first probability is greater than a first probability threshold.

[0009] In an optional implementation, the method further includes: After obtaining the time version and the first data version, if the written data is stored in a data block and the data block corresponds to a first association relationship, determining, based on the time version, each data block to which the snapshot data set corresponding to the periodic task belongs, the first association relationship storing a correspondence between a plurality of sets of first aggregation information and a preset aggregation function, the first aggregation information being obtained by performing statistics on all data in the data block through the preset aggregation function, and the preset aggregation function satisfying the first preset condition; The aggregating, by using the target aggregation function, the snapshot data set and the incremental data set to obtain a task result corresponding to the periodic task includes: According to the target aggregation function, determining, from the first association relationship corresponding to each data block to which the snapshot data set belongs, first target aggregation information corresponding to the target aggregation function; Aggregating, by using the target aggregation function, all the first target aggregation information obtained and the incremental data set to obtain a task result corresponding to the periodic task.

[0010] In an optional implementation, the determining, based on the time version, the snapshot data set corresponding to the periodic task includes: When the written data is stored in a data block, determining, according to the time version, each data block to which the snapshot data set corresponding to the periodic task belongs and a physical address of each data block; Generating each index key according to the time version and each data block to which the snapshot data set belongs; Associating each index key and a respective physical address through a logical pointer to obtain a snapshot index corresponding to the periodic task and creating a mapping relationship between each index key and a respective logical pointer; Determining, according to the time version and each data block to which the snapshot data set belongs, each logical pointer from the mapping relationship; Determining, according to each logical pointer obtained, each physical address from the snapshot index; Determining, according to each physical address obtained, the snapshot data set corresponding to the periodic task.

[0011] In an optional implementation, the snapshot index corresponds to a reference count, the reference count being used to represent a number of tasks using the snapshot index. After obtaining the snapshot index corresponding to the periodic task, the method further includes: performing an increase operation on the reference count; After obtaining the task result corresponding to the periodic task, the method further includes: performing a decrease operation on the reference count; cleaning up the snapshot index when the reference count is determined to be zero.

[0012] In a second aspect, the present application provides a data processing apparatus, including: a version determining module, configured to determine a time version and a first data version corresponding to a periodic task when the periodic task is triggered, and to increment a second data version of newly written data based on the first data version when the newly written data is written, the time version being used to represent a time stamp when the periodic task is triggered, and the first data version being used to represent a highest version of data written when the periodic task is triggered; a data determining module, configured to determine a snapshot data set corresponding to the periodic task based on the time version, and to determine an incremental data set corresponding to the periodic task in parallel according to data of the second data version if it is determined that there is the second data version higher than the first data version during the determination of the snapshot data set; a result determining module, configured to determine a task result corresponding to the periodic task according to the snapshot data set and the incremental data set.

[0013] In a third aspect, the present application provides an electronic device, including a processor and a memory, the processor being configured to execute a data processing program stored in the memory to implement the data processing method as described above.

[0014] In a fourth aspect, the present application provides a storage medium, the storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the data processing method as described above.

[0015] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages. The data processing method provided by the embodiments of the present application comprises: determining a time version and a first data version corresponding to a periodic task when the periodic task is triggered, and incrementing a second data version of newly written data based on the first data version when the newly written data is written. The time version is used to represent a time stamp when the periodic task is triggered, and the first data version is used to represent the highest version of the written data when the periodic task is triggered. A snapshot data set corresponding to the periodic task is determined based on the time version, and in the process of determining the snapshot data set, if it is determined that there is a second data version higher than the first data version, the incremental data set corresponding to the periodic task is determined in parallel according to the data of the second data version. The task result corresponding to the periodic task is determined according to the snapshot data set and the incremental data set. In the above manner, the present application assigns a time version and a first data version to the periodic task when the periodic task is triggered, to allow the writing of new data based on the first data version, without locking the new data, thereby improving the concurrent throughput. Moreover, the snapshot data set corresponding to the periodic task is determined based on the time version, and in the process of determining the snapshot data set, only when there is a second data version higher than the first data version, the determination of the incremental data set corresponding to the periodic task is triggered, so that the task result corresponding to the periodic task is obtained according to the snapshot data set and the incremental data set, without full retry, thereby improving the computing efficiency in the high-concurrency conflict scenario and improving the reliability of the cross-period time-sequenced data aggregation calculation. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0018] One or more embodiments are exemplarily illustrated by the pictures in the corresponding drawings, and these exemplary illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, unless otherwise specified. The drawings in the drawings do not constitute a proportional limitation.

[0019] Figure 1 A flowchart of a data processing method provided by the embodiments of the present application is shown in the figure. Figure 2 A flowchart of another data processing method provided by the embodiments of the present application is shown in the figure. Figure 3A flowchart of another data processing method provided by an embodiment of the present application is shown in FIG. 6. Figure 4 A structure diagram of a data processing device provided by an embodiment of the present application is shown in FIG. 7. Figure 5 A structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 8. In the above figures: 10, version determining module; 20, data determining module; 30, result determining module. 500, electronic device; 501, processor; 502, memory; 5021, operating system; 5022, application program; 503, user interface; 504, network interface; 505, bus system. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of embodiments of the present application clearer, the technical solutions in embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in embodiments of the present application. Obviously, the described embodiments are only a part of embodiments of the present application, rather than all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0021] The following disclosure provides many different embodiments, or examples, for implementing different structures of the present application. For the purpose of simplicity, the components and arrangements of the examples, which are depicted in the following disclosure are shown as being in electrical communication or mechanical communication with one another. It, however, is to be understood that many of the components and arrangements can be implemented in various ways without resorting to such

[0022] Reference Figure 1 , Figure 1 A flowchart of a data processing method provided by an embodiment of the present application is shown in FIG. 6. The data processing method provided by an embodiment of the present application includes the following steps. S101: determining a time version corresponding to a periodic task and a first data version when the periodic task is triggered, and determining a second data version of newly written data based on the first data version when the newly written data is written.

[0023] In the present embodiment, the time version is used to represent a time stamp when the periodic task is triggered, and the first data version is used to represent the highest version of the data written when the periodic task is triggered. The periodic task can be triggered at a fixed time, for example, the periodic task can be to count the power of device A every 1 hour, and the periodic task can be set according to actual needs, which is not limited in the present embodiment.

[0024] In the period task triggering, the time version is determined to define the time boundary of the period task calculation. In order to determine the time version corresponding to the period task, the time stamp at the time of triggering the period task is obtained, and the time stamp is used as the time version corresponding to the period task. For example, the triggering time of the period task is xx year xx month xx day xx: xx, and the triggering time is used as the time version corresponding to the period task. In order to ensure data consistency and improve concurrent throughput, each data version of the current written data is obtained to determine the highest data version of the written data, and the highest data version is determined as the first data version corresponding to the period task, so that in the period task execution process, the new data is allowed to be written based on the first data version, without the need for data locking, ensuring data consistency and improving concurrent throughput. After determining the first data version, if new data is written, the first data version is incremented, so as to assign a second data version to the newly written data. For example, when the first data version is 100, if new data is written at this time, the second data version of the newly written data is assigned to 101. The embodiment can distinguish the written data before the period task triggering and the newly written data in the period task execution process by maintaining the binary tuple [time version, first data version], so as to facilitate the determination of the snapshot data set and the incremental data set.

[0025] It should be noted that the period task is a subtask in a period total task. When the period total task is triggered, a task distribution event is triggered. At this time, the load balancing algorithm is used to evaluate each computing node to determine the applicable computing nodes, and the period total task is divided into each period task, so as to distribute each period task to different computing nodes to execute S101 step, and then complete the execution of the period task. When a certain computing node fails, the load balancing algorithm can be used to switch to a matched computing node to continue to complete the execution of the period task by using the switched computing node, so as to ensure that the period task is not interrupted and the data is not lost.

[0026] S102: determining the snapshot data set corresponding to the period task based on the time version, and in the process of determining the snapshot data set, if it is determined that there is a second data version higher than the first data version, the incremental data set corresponding to the period task is determined in parallel according to the data of the second data version.

[0027] In the embodiment, since the time version is the timestamp when the periodic task is triggered, all the written data located before the time version and within the period included in the periodic task is determined from the database according to the period included in the periodic task, and all the determined written data is determined as the snapshot data set corresponding to the periodic task. It should be noted that after all the written data is determined from the database according to the time version and the period included in the periodic task, all the written data is filtered based on the first data version to ensure that the data version of all the filtered written data is less than or equal to the first data version, and all the filtered written data is determined as the snapshot data set corresponding to the periodic task.

[0028] At the same time when the snapshot data set is determined in the above manner, if it is determined that there is no second data version higher than the first data version, indicating that there is no new data written at the time when the periodic task is triggered, the determined snapshot data set is directly aggregated to obtain the task result corresponding to the periodic task; if it is determined that there is a second data version higher than the first data version, indicating that there is new data written at the time when the periodic task is triggered, at this time, the incremental data set corresponding to the periodic task is determined in parallel according to the data of the second data version.

[0029] In the embodiment, when the data of the second data version is obtained, the data reporting time of each second data version is determined, all the data with a reporting time less than the time version is filtered from the second data version, and all the filtered data is determined as the incremental data set corresponding to the periodic task. Since in the high-concurrency scenario, the data reporting time is often inconsistent with the write time, and only the data with a reporting time less than or equal to the time version is the incremental data of the periodic task, and the data with a reporting time greater than the time version does not belong to the incremental data of the periodic task, therefore, all the data with a reporting time less than the time version is filtered to obtain the incremental data set corresponding to the periodic task. In the embodiment, the determination of the incremental data set corresponding to the periodic task is triggered only when there is a second data version higher than the first data version in the process of determining the snapshot data set, without the need for full retry, thereby improving the computing efficiency in the high-concurrency conflict scenario S103: determining the task result corresponding to the periodic task according to the snapshot data set and the incremental data set.

[0030] In the embodiment, after the snapshot data set and the incremental data set are determined, the snapshot data set and the incremental data set are aggregated to obtain an aggregated result, and the aggregated result is determined as the task result corresponding to the periodic task. After the task result corresponding to the periodic task is obtained, the task result is written to the storage system through an asynchronous message queue to obtain the final task result corresponding to the total periodic task according to the task results written by each computing node, and the flow control is performed through a back pressure mechanism in the process of writing the task result to improve the concurrency and the overall throughput.

[0031] The data processing method provided in the embodiment assigns a time version and a first data version to the periodic task when the periodic task is triggered, allows writing of new data based on the first data version without locking the new data, improves concurrent throughput, determines a snapshot data set corresponding to the periodic task based on the time version, and only triggers determination of an incremental data set corresponding to the periodic task when there is a second data version higher than the first data version in the process of determining the snapshot data set, so as to obtain a task result corresponding to the periodic task according to the snapshot data set and the incremental data set without full retry, thereby improving the computing efficiency in a high-concurrency conflict scenario and improving the reliability of cross-period time-sequenced data aggregation calculation.

[0032] Reference Figure 2 , Figure 2 A flowchart of a data processing method provided in the embodiment of the application. The data processing method provided in the embodiment of the application comprises the following steps: S201: determining a time version and a first data version corresponding to the periodic task when the periodic task is triggered, and incrementing a second data version of newly written data based on the first data version when the data is newly written.

[0033] In the embodiment, the time version is used to represent a time stamp when the periodic task is triggered, and the first data version is used to represent the highest version of the data written when the periodic task is triggered. The S201 step is consistent with the S101 step described above, and reference can be made to the description of the S101 step above, which will not be repeated here in the embodiment.

[0034] S202: determining a snapshot data set corresponding to the periodic task based on the time version, and determining a target aggregation function required for executing the periodic task if it is determined that there is a second data version higher than the first data version in the process of determining the snapshot data set.

[0035] S203: determining an incremental data set corresponding to the periodic task in parallel according to the data of the second data version when the target aggregation function meets a first preset condition.

[0036] For the S202 step and the S203 step, the target aggregation function is used to determine the task result corresponding to the periodic task based on the snapshot dataset and the incremental dataset corresponding to the periodic task. The first preset condition includes that the target aggregation function is a combinable and commutative function. For example, the target aggregation function can be a summation function, a maximum value function, a minimum value function, etc. The combinable can be understood as that the calculation order does not affect the final calculation result, and the commutative can be understood as that the data order does not affect the final calculation result. The target aggregation function in the above description satisfies the combinable and commutative. When the target aggregation function is a median function or a percentile function, the calculation order and the data order will affect the final calculation result, so the target aggregation function in the above description does not satisfy the first preset condition.

[0037] In the process of determining the snapshot set, if it is determined that there is a second data version higher than the first data version, it is indicated that new data is written at the time of triggering the periodic task. Since the target aggregation function satisfies the first preset condition, the calculation order and the data order do not affect the final calculation result, so the newly written data is determined to obtain the incremental dataset corresponding to the periodic task, and then the snapshot dataset and the incremental dataset are aggregated by using the target aggregation function to obtain the final task result corresponding to the periodic task. When the target aggregation function does not satisfy the first preset condition, the calculation order and the data order affect the final calculation result, in order to improve the calculation efficiency and the accuracy of the calculation, the newly written data is not determined, and the snapshot dataset is aggregated to obtain the final task result corresponding to the periodic task. Through the above method, when the incremental dataset corresponding to the periodic task is determined, unnecessary calculation is abandoned, the calculation efficiency is improved, and the accuracy of the calculation is improved.

[0038] In the embodiment, the determination of the snapshot dataset corresponding to the periodic task based on the time version in the S202 step includes: When the written data is stored in the data block, the snapshot dataset corresponding to the periodic task is determined according to the time version and the physical address of each data block to which the snapshot dataset belongs; According to the time version and the snapshot dataset, each index key is generated; Each index key and the corresponding physical address are associated through a logical pointer to obtain a snapshot index corresponding to the periodic task, and a mapping relationship between each index key and each logical pointer is created; According to the time version and each data block, the logical pointer corresponding to each data block is determined from the mapping relationship; According to the time version and each data block to which the snapshot dataset belongs, each logical pointer is determined from the mapping relationship; According to the obtained each logical pointer, each physical address is determined from the snapshot index; According to the obtained respective physical addresses, the snapshot data set corresponding to the periodic task is determined.

[0039] In the embodiment, when the written data is stored in the data blocks, all the data blocks located before the time stamp corresponding to the time version and within the period included by the periodic task are determined, and all the data blocks with data versions less than or equal to the first data version are screened out from all the data blocks, which constitute the data blocks to which the snapshot data set corresponding to the periodic task belongs, and the physical addresses of the respective obtained data blocks are determined (the physical addresses can be data acquisition paths). For each data block obtained, a unique index key is generated for the data block according to the data block and the time version, the index key can be in the form of a combination of the time version and the ID of the data block, and a logical pointer pointing to the physical address corresponding to each data block is created, so as to associate each index key with the respective physical address (i.e. the physical address corresponding to the data block included in the index key) through the logical pointer by means of the B+ tree algorithm, so as to obtain the snapshot index corresponding to the periodic task. Meanwhile, a mapping relationship between the index keys and the respective logical pointers is created by means of the red-black tree algorithm, so as to complete the positioning of the snapshot data set corresponding to the periodic task by means of the index keys.

[0040] After the snapshot index and the mapping relationship are obtained, according to the time version and the respective data blocks to which the snapshot data set belongs, a plurality of index keys are obtained, so as to determine the logical pointers corresponding to the respective index keys from the mapping relationship. The physical addresses pointed to by the respective logical pointers are obtained by querying in the snapshot index through the respective logical pointers, so as to access the data in the respective corresponding data blocks through the respective physical addresses, and obtain the snapshot data set corresponding to the periodic task. In the above manner, in the embodiment, when the snapshot data set corresponding to the periodic task is determined, the index is used to quickly locate the respective data blocks, so as to obtain the required data from the respective data blocks, thereby avoiding scanning all the data and improving the calculation efficiency.

[0041] It should be noted that, when the written data is stored in the data blocks, when the time version corresponding to the periodic task and the first data version are determined, if there is newly written data, the target state and the data version are assigned to the data block in which the newly written data is located, the target state can be to be merged, so as to determine the data block in which the incremental data is stored according to the target state and the data version, so as to determine the incremental data set based on the above-mentioned use of the second version of the data. In the above manner, the traversal time of the data blocks is reduced, and the calculation efficiency is improved.

[0042] S204: When the target aggregation function meets the first preset condition, the snapshot data set and the incremental data set are aggregated by means of the target aggregation function, so as to obtain the task result corresponding to the periodic task.

[0043] S205: When the target aggregation function does not satisfy the first preset condition, the snapshot data set is segmented to obtain a plurality of snapshot data segments.

[0044] S206: Determine the target feature corresponding to each snapshot data segment.

[0045] S207: According to all the target features obtained, determine the task result corresponding to the periodic task.

[0046] For the above S204 and S207 steps, the target feature is used to describe the distribution characteristics of all snapshot data in the snapshot data segment. When the target aggregation function satisfies the first preset condition, the target aggregation function is a function that can be combined and exchanged, at which time the snapshot data set and the incremental data set are aggregated directly using the target aggregation function to obtain the task result corresponding to the periodic task. For example, when the target aggregation function is a summation function, if the total power corresponding to the snapshot data set is 100 degrees, and the total power corresponding to the incremental data set is 20 degrees, then the target aggregation function is used to aggregate the snapshot data set and the incremental data set, and the task result corresponding to the periodic task is obtained as 120 degrees.

[0047] When the target aggregation function does not satisfy the first preset condition, the target aggregation function is not a function that can be combined and exchanged, and since the incremental data set corresponding to the periodic task is not determined, the task result corresponding to the periodic task can be determined directly based on the snapshot data set. When the task result corresponding to the periodic task is determined based on the snapshot data set, in order to improve the calculation efficiency, the snapshot data set can be segmented to obtain a plurality of snapshot data segments. For each snapshot data segment, a target feature is determined that can describe the distribution of the snapshot data segment, such as maximum value, minimum value, total sum, variance, etc. By summarizing and analyzing the target features of each snapshot data segment obtained above, the distribution shape and range of all snapshot data in the snapshot data set are understood, and the final task result corresponding to the periodic task is obtained using interpolation or other estimation algorithms, without the need to load all snapshot data in the snapshot data set into the memory for calculation, thereby improving the calculation efficiency. Through the above method, when the target aggregation function is a function that can be combined and exchanged, the snapshot data set and the incremental data set are aggregated directly using the target aggregation function, which improves the calculation accuracy without affecting the calculation efficiency, and when the target aggregation function is not a function that can be combined and exchanged, the task result corresponding to the periodic task is approximately determined using the segmentation method, which improves the calculation efficiency while ensuring the calculation accuracy.

[0048] In the embodiment, the snapshot data set is stored in each data block, each data block corresponds to a first association relationship, the first association relationship stores a correspondence relationship between a plurality of sets of first aggregation information and a preset aggregation function, the first aggregation information is obtained by aggregating all snapshot data in the data block through the preset aggregation function, and the preset aggregation function satisfies the first preset condition. The form of the preset aggregation function can refer to the target aggregation function described above, and will not be described here in the embodiment. In the above, for example, when the preset aggregation function is a summation function, the first aggregation information corresponding to the preset aggregation function is the sum of all snapshot data in the data block; when the preset aggregation function is a maximum value function, the first aggregation information corresponding to the preset aggregation function is the maximum value of all snapshot data in the data block.

[0049] The data processing method provided in the embodiment further includes the following steps: After obtaining the time version and the first data version, if the written data is stored in the data block and the data block corresponds to the first association relationship, each data block to which the snapshot data set corresponding to the periodic task belongs is determined based on the time version, the first association relationship stores a correspondence relationship between a plurality of sets of first aggregation information and a preset aggregation function, the first aggregation information is obtained by statistically processing all data in the data block through the preset aggregation function, and the preset aggregation function satisfies the first preset condition.

[0050] The step S204 described above includes the following steps: According to the target aggregation function, the first target aggregation information corresponding to the target aggregation function is determined from the first association relationship corresponding to each data block to which the snapshot data set belongs; All the first target aggregation information obtained and the incremental data set are aggregated by using the target aggregation function, to obtain the task result corresponding to the periodic task.

[0051] Specifically, when data is written, the data is stored in the data blocks in a time sequence relationship, and after the storage of a data block is completed, all data in the data block is aggregated by using each preset aggregation function to obtain corresponding first aggregation information. Thus, after the time version and the first data version are obtained, if the data is stored in the data blocks and each data block corresponds to the first association relationship, the data blocks (i.e., the data blocks to which the snapshot data set corresponding to the periodic task belongs) before the time version and within the period can be directly determined according to the period included in the periodic task and the time version. After the data blocks to which the snapshot data set corresponding to the periodic task belongs are determined, the first association relationship corresponding to each data block is queried by using the target aggregation function, so as to obtain each first target aggregation information corresponding to the target aggregation function. The obtained all first target aggregation information and the incremental data set are aggregated by using the target aggregation function, so as to obtain the task result corresponding to the periodic task. Through the above method, the calculation amount of the periodic task is greatly reduced, and the efficiency of processing the snapshot data set is improved.

[0052] More specifically, if the written data is stored in the data blocks and the data blocks correspond to the first association relationship, the snapshot index and the mapping relationship can be created by the above method, so as to be positioned to each physical address through the snapshot index and the mapping relationship, so as to obtain the first association relationship corresponding to each data block by accessing each physical address, and then the determination of the task result corresponding to the periodic task is implemented based on each first association relationship.

[0053] It should be noted that after the first association relationship corresponding to each data block is obtained, a plurality of second association relationships can be obtained based on the plurality of first association relationships. The second association relationship stores a plurality of second aggregation information and a corresponding relationship between the preset aggregation function. The second aggregation information corresponding to the preset aggregation function is obtained by aggregating at least two first aggregation information corresponding to the preset aggregation function by using the preset aggregation function. When the snapshot data set and the incremental data set are used to obtain the task result corresponding to the periodic task, if there are a plurality of second association relationships, the second target aggregation information corresponding to the target aggregation function can be determined from each second association relationship according to the target aggregation function. The obtained all second target statistical information and the incremental data set are aggregated by using the target aggregation information, so as to obtain the task result corresponding to the periodic task. Through the above, after the first association relationship corresponding to each data block is obtained, a plurality of second association relationships can be obtained based on each first association relationship by using the downsampling method, so as to further reduce the calculation amount of the periodic task and improve the efficiency of processing the snapshot data set.

[0054] In the embodiment, the snapshot index corresponds to a reference count, and the reference count is used to represent the number of tasks using the snapshot index.

[0055] After obtaining the snapshot index corresponding to the periodic task, the data processing method provided in this embodiment further includes: An increase operation is performed on the reference count.

[0056] After obtaining the task result corresponding to the periodic task, the data processing method provided in this embodiment further includes: A decrease operation is performed on the reference count. When the reference count is determined to be zero, the snapshot index is cleaned up.

[0057] The reference count is initially 0, and when the snapshot index is created, an increase operation (for example, incrementing by 1) is performed on the reference count. If the task still needs to use the snapshot index, each time the reference count is used by a task, an increase operation is performed on the reference count. Each time the reference count is no longer used by a task, a decrease operation (for example, decrementing by 1) is performed on the reference count. After the determination of the task result corresponding to the periodic task is completed, a decrease operation is performed on the reference count. If the reference count is zero at this time, it indicates that no task uses the snapshot index. In order to release resources, the snapshot index is cleaned up. In the above manner, the reference count is set to monitor the life cycle of the created snapshot index, so that the snapshot index is cleaned up when no task uses the snapshot index, thereby releasing resources.

[0058] The data processing method provided in this embodiment assigns a time version and a first data version to the periodic task when the periodic task is triggered, to allow the writing of new data based on the first data version without locking the new data, thereby improving the concurrent throughput. Moreover, the snapshot data set corresponding to the periodic task is determined based on the time version, and in the process of determining the snapshot data set, the determination of the incremental data set corresponding to the periodic task is triggered only when there is a second data version higher than the first data version, so that the task result corresponding to the periodic task is obtained according to the snapshot data set and the incremental data set, without full retry, thereby improving the computing efficiency in a high-concurrency conflict scenario and improving the reliability of cross-period time-sequenced data aggregation calculation.

[0059] Reference Figure 3 , Figure 3 This is a flowchart of another data processing method provided in this embodiment. The data processing method provided in this embodiment includes the following steps: S301: When the periodic task is triggered, the time version and the first data version corresponding to the periodic task are determined, and when new data is written, the second data version of the new written data is incremented based on the first data version.

[0060] S302: Determine the snapshot dataset corresponding to the periodic task based on the time version, and in the process of determining the snapshot dataset, if it is determined that there is a second data version higher than the first data version, determine the target aggregation function required to execute the periodic task.

[0061] For the above S301 step and S302 step, S301 step is consistent with the above S201 step, S302 step is consistent with the above S202 step, and specific reference can be made to the above S201 step and S202 step. In this embodiment, no further description is made.

[0062] S303: When the target aggregation function meets the first preset condition, determine whether the target device needs to report the use data of the target device within the first time period.

[0063] S304: When it is determined that the target device needs to report the use data of the target device within the first time period, determine the incremental dataset corresponding to the periodic task in parallel according to the data of the second data version.

[0064] For the above S303 step and S304 step, the periodic task is used to count the use data of the target device, and the target device can be each air conditioning device, etc. The use data can be power, etc. The specific form of the target device and the use data can be selected as needed, and no further description is made in this embodiment. The trigger time of the periodic task is within the first time period, and the first time period is determined according to the floating upper limit and the floating lower limit of the trigger time of the periodic task. If the floating upper limit is 2s, the floating lower limit is 2s, and the trigger time of the periodic task is t, the first time period is [t-2s, t+2s].

[0065] Wherein, when the target aggregation function meets the first preset condition, it indicates that the incremental dataset corresponding to the periodic task needs to be determined at this time. At this time, in order to reduce unnecessary calculation and improve calculation efficiency, it is determined whether the target device needs to report the use data of the target device within the first time period. If it is determined that the target device needs to report the use data of the target device within the first time period, at this time, the newly written data will exist in the use data reported by the target device within the first time period. Therefore, the incremental dataset corresponding to the periodic task is determined in parallel according to the data of the second version. If it is determined that the target device does not need to report the use data of the target device within the first time period, at this time, the newly written data will not exist in the use data reported by the target device within the first time period. Therefore, the incremental dataset corresponding to the periodic task is determined in parallel according to the data of the second version, which increases unnecessary calculation. Through the above method, in this embodiment, when the target aggregation function meets the first preset condition, only when it is determined that the target device needs to report the use data of the target device within the first time period, the determination process of the incremental dataset corresponding to the periodic task is triggered, unnecessary calculation is reduced, and calculation efficiency is improved.

[0066] The determination of the target device in the S304 step needs to report the use data of the target device within the first time period, including: Obtain the historical use data set reported by the target device; Input the historical use data set and the historical reporting time corresponding to each historical use data in the historical use data set into the preset prediction model, so that the preset prediction model outputs the first probability; When the first probability is greater than the first probability threshold, it is determined that the target device needs to report the use data of the target device within the first time period.

[0067] In this embodiment, each historical use data in the historical use data set corresponds to a historical reporting time. The first probability is used to represent the probability of the target device reporting the use data of the target device within the first time period, and the first probability threshold is used to represent the lower limit value of the target probability corresponding to the target device reporting the use data of the target device within the first time period. The first probability threshold can be set according to actual needs, and the first probability threshold is not limited in this embodiment. The preset prediction model can be a Markov chain model, and of course, the preset prediction model can be selected according to actual needs, and this embodiment is not limited.

[0068] Wherein, when the target aggregation function meets the first preset condition, the historical use data reported by the target device at the historical reporting time can be obtained, the first time period, each historical reporting time and the historical use data corresponding to the historical reporting time are input into the preset prediction model, so that the preset prediction model learns the rule of the target device reporting use data according to each historical reporting time and each historical use data corresponding to the historical reporting time., so that the preset prediction model outputs the first probability of the target device reporting the use data of the target device within the first time period according to the first time period. When the first probability is greater than the first probability threshold, it is represented that the target device needs to report the use data of the target device within the first time period. At this time, the incremental data set corresponding to the periodic task needs to be determined according to the data of the second data version; while the first probability is less than or equal to the first probability threshold, it is represented that the target device does not need to report the use data of the target device within the first time period. Therefore, at this time, the new written data does not exist the use data reported by the target device, so at this time, it is not necessary to determine the incremental data set corresponding to the periodic task according to the data of the second data version, further reducing unnecessary calculation. Through the above method, the reporting rule of the target device is learned in this embodiment to determine whether the target device reports the use data of the target device when the periodic task is triggered, so that when the target device reports the use data of the target device when the periodic task is triggered, the incremental data set corresponding to the periodic task is determined according to the data of the second data version, further reducing the necessary calculation and improving the calculation efficiency.

[0069] In the embodiment, the preset prediction model also outputs a plurality of second probabilities, the second probabilities being used to represent probabilities of the use data range to which the use data reported by the target device in the first time period belongs. When the first probability is greater than the first probability threshold, if there is a second probability greater than a second probability threshold in the plurality of second probabilities, all data within the use data corresponding to the second probability is filtered out from the data of the second data version, and the incremental data set corresponding to the periodic task is determined according to all the data within the use data corresponding to the second probability.

[0070] The second probability threshold is used to represent a lower limit value of the second probability of the use data reported in the first time period belonging to the use data range, and the second probability threshold can be set according to actual needs, and the specific value of the second probability threshold is not limited in the embodiment. In the embodiment, the use data range to which the use data reported by the target device belongs is determined, so that when it is necessary to determine the task result corresponding to the periodic task according to the data of the second data version, the data of the second data version can be filtered according to the use data range, the efficiency of determining the incremental data set is improved, and the calculation efficiency of the subsequent task result is further improved.

[0071] S305: determining the task result corresponding to the periodic task according to the snapshot data set and the incremental data set.

[0072] In the embodiment, the S305 step is consistent with the S101 step described above, and the specific description can be referred to the S101 step described above, which will not be repeated here in the embodiment.

[0073] The data processing method provided in the embodiment assigns a time version and a first data version to the periodic task when the periodic task is triggered, so that the writing of new data is allowed based on the first data version, and the new data does not need to be locked, thereby improving the concurrent throughput. Moreover, the snapshot data set corresponding to the periodic task is determined based on the time version, and only when there is a second data version higher than the first data version, the determination of the incremental data set corresponding to the periodic task is triggered, so that the task result corresponding to the periodic task is obtained according to the snapshot data set and the incremental data set, without full retry, thereby improving the calculation efficiency in the high-concurrency conflict scenario and improving the reliability of the cross-period time-series data aggregation calculation.

[0074] Reference Figure 4 , Figure 4A structural schematic diagram of a data processing apparatus provided in an embodiment of the present application. The data processing apparatus provided in an embodiment of the present application comprises a version determining module 10, a data determining module 20 and a result determining module 30. The version determining module 10 is configured to determine a time version corresponding to a periodic task and a first data version when the periodic task is triggered, and to increment a second data version of newly written data based on the first data version when the newly written data is written. The time version is used to represent a time stamp when the periodic task is triggered, and the first data version is used to represent a highest version of data written when the periodic task is triggered. The data determining module 20 is configured to determine a snapshot data set corresponding to the periodic task based on the time version, and to determine an incremental data set corresponding to the periodic task in parallel according to data of the second data version if it is determined that the second data version higher than the first data version exists during the determination of the snapshot data set. The result determining module 30 is configured to determine a task result corresponding to the periodic task according to the snapshot data set and the incremental data set.

[0075] In the embodiment, the data determining module 20 is further configured to: determine a target aggregation function required for executing the periodic task; determine an incremental data set corresponding to the periodic task in parallel according to data of the second data version when the target aggregation function meets a first preset condition, wherein the first preset condition comprises that the target aggregation function is a combinable and commutative function.

[0076] In the embodiment, the result determining module 30 is further configured to: aggregate the snapshot data set and the incremental data using the target aggregation function to obtain a task result corresponding to the periodic task when the target aggregation function meets the first preset condition; segment the snapshot data set to obtain a plurality of snapshot data segments when the target aggregation function does not meet the first preset condition; determine a target feature corresponding to each snapshot data segment, wherein the target feature is used to describe a distribution feature of all snapshot data in the snapshot data segment; determine a task result corresponding to the periodic task according to all the target features obtained.

[0077] In the embodiment, the periodic task is used to count usage data of a target device, and the data determining module 20 is further configured to: determine whether the target device needs to report the usage data of the target device in a first time period when the target aggregation function meets a first preset condition; determining, in parallel, a set of incremental data corresponding to the periodic task according to the second version of data, and a trigger time of the periodic task is located in the first time period, when it is determined that the target device needs to report the usage data of the target device in the first time period.

[0078] In this embodiment, the data determination module 20 is further configured to: obtain a set of historical usage data reported by the target device, each historical usage data in the set of historical usage data corresponding to a historical reporting time; input the set of historical usage data and the historical reporting time corresponding to each historical usage data in the set of historical usage data into a preset prediction model, so that the preset prediction model outputs a first probability, the first probability being used to represent a probability that the target device reports the usage data of the target device in the first time period; determine that the target device needs to report the usage data of the target device in the first time period when the first probability is greater than a first probability threshold.

[0079] In this embodiment, the data determination module 20 is further configured to: After obtaining the time version and the first version of data, if the written data is stored in a data block and the data block corresponds to a first association relationship, determining, based on the time version, each data block to which the set of snapshot data corresponding to the periodic task belongs, the first association relationship storing a correspondence between a plurality of sets of first aggregation information and a preset aggregation function, the first aggregation information being obtained by performing statistics on all data in the data block through the preset aggregation function, and the preset aggregation function satisfying the first preset condition.

[0080] In this embodiment, the result determination module 30 is further configured to: determining, according to the target aggregation function, a first target aggregation information corresponding to the target aggregation function from the first association relationship corresponding to each data block to which the set of snapshot data belongs; performing aggregation on all the first target aggregation information and the set of incremental data obtained by using the target aggregation function, to obtain a task result corresponding to the periodic task.

[0081] In this embodiment, the data determination module 20 is further configured to: when the written data is stored in a data block, determining, according to the time version, each data block to which the set of snapshot data corresponding to the periodic task belongs and a physical address of each data block; generating each index key according to the time version and each data block to which the set of snapshot data belongs; associate each of the index keys with the respective corresponding physical address through a logical pointer to obtain a snapshot index corresponding to the periodic task, and create a mapping relationship between each of the index keys and each of the logical pointers; determine each of the logical pointers from the mapping relationship according to each of the data blocks to which the time version and the snapshot dataset belong; determine each of the physical addresses from the snapshot index according to each of the obtained logical pointers; determine the snapshot dataset corresponding to the periodic task according to each of the obtained physical addresses.

[0082] In the embodiment, the snapshot index corresponds to a reference count, and the reference count is used to represent the number of tasks using the snapshot index.

[0083] The data processing apparatus provided in the embodiment further includes a management module, and the management module is configured to: perform an increase operation on the reference count after obtaining the snapshot index corresponding to the periodic task; perform a decrease operation on the reference count after obtaining the task result corresponding to the periodic task; clean up the snapshot index when the reference count is determined to be zero.

[0084] The data processing apparatus provided in the embodiment assigns a time version and a first data version to the periodic task when the periodic task is triggered, allows writing of new data based on the first data version, and does not need to lock the new data, thereby improving the concurrent throughput. In addition, the time version is used to determine the snapshot dataset corresponding to the periodic task, and in the process of determining the snapshot dataset, only when there is a second data version higher than the first data version, the determination of the incremental dataset corresponding to the periodic task is triggered, so that the task result corresponding to the periodic task is obtained according to the snapshot dataset and the incremental dataset, without full retry, thereby improving the computing efficiency in the high-concurrency conflict scenario and improving the reliability of the cross-period time-series data aggregation calculation.

[0085] Figure 5 a structural schematic diagram of an electronic device provided for an embodiment of the present application, Figure 5 The electronic device 500 shown includes at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. Each component in the electronic device 500 is coupled together through a bus system 505. It can be understood that the bus system 505 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, only the data bus is shown in the figure. Figure 5Various buses are labeled as bus 505 in the figure, for clarity.

[0086] The user interface 503 can include, for example, a display, a keyboard or a pointing device (e.g., a mouse, a trackball, a touchpad, or a touchscreen).

[0087] It can be understood that the memory 502 in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, without being limited to, these and any other suitable types of memory.

[0088] In some embodiments, the memory 502 stores the following elements, executable units or data structures, or a subset of them, or an extended set of them: an operating system 5021 and an application program 5022.

[0089] The operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program 5022 includes various application programs, such as a Media Player, a Browser, etc., for implementing various application services. The program for implementing the method of the embodiments of the present application can be included in the application program 5022.

[0090] In the embodiments of the present application, the processor 501 is configured to execute the method steps of the method embodiments by invoking the programs or instructions stored in the memory 502, specifically, the programs or instructions stored in the application program 5022.

[0091] The method disclosed in the embodiments of the present application can be applied to the processor 501 or implemented by the processor 501. The processor 501 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by hardware integrated logic circuit or software form of instructions in the processor 501. The processor 501 described above can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software units in the code processor for execution. The software unit can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and combines the hardware to complete the steps of the above method.

[0092] It can be understood that the embodiments described herein can be realized by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units for executing the functions described in the present application or a combination thereof.

[0093] For software implementation, the techniques described herein can be implemented by means of software programs. The software programs can be stored in memory and executed by processors. The memory can be implemented within the processors or externally to the processors.

[0094] The electronic device provided by the embodiment can be an electronic device as shown in Figure 5 The electronic device provided by the embodiment can be an electronic device as shown in Figures 1-3 The electronic device provided by the embodiment can be an electronic device as shown in Figures 1-3 The electronic device provided by the embodiment can be an electronic device as shown in Figures 1-3 The electronic device provided by the embodiment can be an electronic device as shown in

[0095] The embodiment of the present application further provides a storage medium (computer readable storage medium). The storage medium stores one or more programs. The storage medium can include a volatile memory such as a random access memory, and the memory can also include a non-volatile memory such as a read-only memory, a flash memory, a hard disk or a solid state disk, and the memory can also include a combination of the above kinds of memories.

[0096] When the one or more programs stored in the storage medium can be executed by the one or more processors, the data processing method executed at the data processing device side described above can be implemented.

[0097] The skilled person should further appreciate that the units and algorithm steps of various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The various examples have been described in general terms above with reference to a functional description of the various units and steps. Whether the described functionality is implemented in hardware or software depends on the particular application and design constraints of the technical solution. Skilled persons can use various methods to implement the described functions for each particular application, but such implementation should not be considered beyond the scope of the present application.

[0098] The steps of the methods or algorithms described in connection with the embodiments disclosed herein can be implemented in hardware, software executed by a processor, or a combination of both. The software modules can be stored in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0099] The above detailed description of the specific embodiments of the present application has been given to understand the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: determining a time version and a first data version corresponding to the periodic task when the periodic task is triggered, and determining a second data version of newly written data based on the first data version when the data is newly written, wherein the time version is used to represent a time stamp when the periodic task is triggered, and the first data version is used to represent the highest version of the data written when the periodic task is triggered; determining a snapshot data set corresponding to the periodic task based on the time version, and determining an incremental data set corresponding to the periodic task in parallel according to the data of the second data version if it is determined that there is the second data version higher than the first data version during the determination of the snapshot data set; determining a task result corresponding to the periodic task according to the snapshot data set and the incremental data set.

2. The method of claim 1, wherein, The method of determining the incremental data set corresponding to the periodic task in parallel according to the data of the second data version comprises: determining a target aggregation function required for executing the periodic task; when the target aggregation function meets a first preset condition, determining the incremental data set corresponding to the periodic task in parallel according to the data of the second data version, wherein the first preset condition comprises that the target aggregation function is a combinable and commutative function.

3. The method of claim 2, wherein, The method of determining the task result corresponding to the periodic task according to the snapshot data set and the incremental data set comprises: when the target aggregation function meets the first preset condition, aggregating the snapshot data set and the incremental data by using the target aggregation function to obtain the task result corresponding to the periodic task; when the target aggregation function does not meet the first preset condition, segmenting the snapshot data set to obtain a plurality of snapshot data segments; determining a target feature corresponding to each snapshot data segment, wherein the target feature is used to describe the distribution characteristics of all snapshot data in the snapshot data segment; determining the task result corresponding to the periodic task according to all the target features obtained.

4. The method of claim 2, wherein, The periodic task is used to count the usage data of a target device. The method of determining the incremental data set corresponding to the periodic task in parallel according to the data of the second data version when the target aggregation function meets the first preset condition comprises: determining whether the target device needs to report the usage data of the target device within a first time period when the target aggregation function meets the first preset condition; determining the incremental data set corresponding to the periodic task in parallel according to the data of the second data version when it is determined that the target device needs to report the usage data of the target device within the first time period, wherein the trigger time of the periodic task is located within the first time period.

5. The method of claim 4, wherein, The method of determining that the target device needs to report the usage data of the target device within the first time period comprises: obtaining a historical usage data set reported by the target device, wherein each historical usage data in the historical usage data set corresponds to a historical reporting time. inputting the historical usage data set and the historical reporting time corresponding to each of the historical usage data in the historical usage data set into a preset prediction model, so that the preset prediction model outputs a first probability, the first probability being used to represent a probability that the target device reports usage data of the target device in the first time period; when the first probability is greater than a first probability threshold, determining that the target device needs to report usage data of the target device in the first time period.

6. The method of claim 3, wherein, The method further comprises: After obtaining the time version and the first data version, if the written data is stored in a data block and the data block corresponds to a first association relationship, determining, based on the time version, each data block to which the snapshot data set corresponding to the periodic task belongs, the first association relationship storing a correspondence relationship between a plurality of first aggregation information and a preset aggregation function, the first aggregation information being obtained by performing statistics on all data in the data block through the preset aggregation function, and the preset aggregation function satisfying the first preset condition; The aggregation of the snapshot data set and the incremental data set by using the target aggregation function to obtain the task result corresponding to the periodic task comprises: According to the target aggregation function, determining first target aggregation information corresponding to the target aggregation function from the first association relationship corresponding to each data block to which the snapshot data set belongs; aggregating all the first target aggregation information obtained and the incremental data set by using the target aggregation function to obtain the task result corresponding to the periodic task.

7. The method of claim 1, wherein, The determination of the snapshot data set corresponding to the periodic task based on the time version comprises: When the written data is stored in a data block, determining, according to the time version, each data block to which the snapshot data set corresponding to the periodic task belongs and a physical address thereof; generating each index key according to the time version and each data block to which the snapshot data set belongs; associating each index key with a respective physical address through a logical pointer to obtain a snapshot index corresponding to the periodic task and creating a mapping relationship between each index key and each logical pointer; determining each logical pointer from the mapping relationship according to the time version and each data block to which the snapshot data set belongs; determining each physical address from the snapshot index according to each logical pointer obtained; determining the snapshot data set corresponding to the periodic task according to each physical address obtained.

8. The method of claim 7, wherein, The snapshot index corresponds to a reference count, and the reference count is used to represent a number of tasks using the snapshot index; After obtaining the snapshot index corresponding to the periodic task, the method further comprises: performing an increase operation on the reference count; After obtaining the task result corresponding to the periodic task, the method further comprises: performing a decrease operation on the reference count; when the reference count is determined to be zero, cleaning up the snapshot index.

9. A data processing apparatus, characterized by, comprises: A version determining module is configured to determine a time version corresponding to the periodic task and a first data version when the periodic task is triggered, and to increment a second data version of newly written data based on the first data version when the newly written data is written, wherein the time version is used to represent a time stamp when the periodic task is triggered, and the first data version is used to represent a highest version of written data when the periodic task is triggered. A data determining module is configured to determine a snapshot data set corresponding to the periodic task based on the time version, and to determine an incremental data set corresponding to the periodic task in parallel according to data of the second data version if it is determined that there is the second data version higher than the first data version during the determination of the snapshot data set. A result determining module is configured to determine a task result corresponding to the periodic task according to the snapshot data set and the incremental data set.

10. An electronic device, comprising: The data processing device comprises: A processor and a memory, wherein the processor is configured to execute a data processing program stored in the memory, so as to implement the data processing method in any one of claims 1-8.

11. A storage medium, characterized by The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method in any one of claims 1-8.

Images