Data processing method and device

Abstract

The invention provides a data processing method and device and a storage medium. Based on an analysis result of the feature calculation task, a first operation statement is generated, and the first operation statement is used for carrying out statistics on the to-be-processed data set; based on a statistical result obtained by executing the first operation statement, a second operation statement is generated, the second operation statement is used for marking data in the data set, and the mark of the data is used for identifying a block where the data is located; on the basis of a marking result obtained by executing the second operation statement, a third operation statement is generated, the third operation statement is used for supplementing dependent data lacking in the block, and the dependent data are data which need to depend on during feature calculation of the data in the block but do not exist in the block; and the data set is expanding by executing the third operation statement, and the data in the expanded data set is divided into a plurality of blocks. Therefore, while the problem of data skew possibly existing when the feature calculation task is executed is solved, the correctness of the calculation logic of the data in the partition can be ensured.

Description

technical field [0001] The present invention generally relates to the field of data processing, and more specifically, relates to a data processing method, device and storage medium. Background technique [0002] In the field of data processing, it is usually necessary to perform feature calculations on data to extract features that can represent specific meanings from the data. Taking the feature calculation task based on the ordered window as an example, for each piece of data, the feature calculation can be performed based on the data within the range of the ordered window corresponding to the piece of data. [0003] When performing feature calculation tasks, especially feature calculation tasks based on ordered windows, it is necessary to avoid data skew as much as possible. Data skew means that a large amount of data is allocated to a computing node to perform calculations, making the calculation speed of these data much lower than the average calculation speed, result...

AI Technical Summary

Problems solved by technology

Data skew means that a large amount of data is allocated to a computing node to perform calculations, making the calculation speed of these data much lower than the average calculation speed, resulting in the entire calculation process being too slow

[0004] Due to the different reasons for the data skew problem, the existing data skew optimization scheme is more cumbersome to implement, and the optimization scheme occupies more resources (such as time resources), which will make the optimization scheme lose the meaning of existence

Images