Data processing method and device, equipment, storage medium and program product
By determining the processing strategy based on the outlier type and data quantity, the problem of low processing efficiency of abnormal data in the prior art is solved, and more efficient abnormal data processing is achieved.
Patent Information
- Application Number
- CN202411823002.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-03-28
AI Technical Summary
When the prior art processes abnormal data in complex data scenarios, statistical analysis is difficult to quickly cover all data abnormal situations, resulting in low efficiency in abnormal data processing.
When the outlier type is of multiple types, the outlier value processing strategy adopted is determined based on the outlier type, the data volume of different outlier types, and the preset quantity threshold, so as to improve the efficiency of exception data processing.
It improves the efficiency of exception data processing in complex data scenarios, and can process multiple types of exception data more quickly and accurately.
Smart Images
Figure CN119719086A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art
[0002] With the advent of the big data era, the amount of data processed by enterprises and organizations is increasing, and ensuring data quality has become a key issue. Especially in the ETL processing tasks of data warehouses, it is particularly important to perform quality inspection and exception handling on the target table data. These data may come from different sources and undergo multiple processing and conversions, so there may be various quality issues, such as missing data, erroneous data, duplicate data, etc.
[0003] The current abnormal data processing methods mainly rely on simple statistical analysis for detection, and then use pre-configured processing strategies for the different types of detected abnormal value data. For complex data quality issues, statistical analysis is difficult to quickly cover all data anomalies, and the efficiency of abnormal data processing is low. Summary of the invention
[0004] The embodiments of the present application provide a data processing method, apparatus, device, storage medium and program product, which are beneficial to improving the efficiency of abnormal data processing in complex data scenarios by determining the adopted outlier processing strategy according to the outlier type, the data volume of different outlier types and a preset quantity threshold when there are multiple outlier types.
[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:
[0006] The preprocessed first target table data is quality checked according to the first data detection model; abnormal data is detected in the first target table data, and the outlier value type of the abnormal data in the first target table data is determined; it is detected that the outlier value types of multiple abnormal data in the first target table data are different, and the outlier value processing strategies corresponding to the data of different outlier value types in the first target table data are different, and the target outlier value processing strategy is determined according to the outlier value type, the first quantity threshold and the second quantity threshold; the abnormal data in the first target table data is processed according to the target outlier value processing strategy.
[0007] In a second aspect, an embodiment of the present application provides a data processing device, including:
[0008] A detection module, used for performing quality detection on the preprocessed first target table data according to the first data detection model;
[0009] a determination module, configured to detect that abnormal data exists in the first target table data, determine the abnormal value type of the abnormal data in the first target table data; and to detect that the abnormal value types of multiple abnormal data in the first target table data are different, and that the abnormal value processing strategies corresponding to the data of different abnormal value types in the first target table data are different, and determine the target abnormal value processing strategy according to the abnormal value type, the first quantity threshold and the second quantity threshold;
[0010] A processing module is used to process the abnormal data in the first target table data according to the target abnormal value processing strategy.
[0011] In a third aspect, an embodiment of the present application provides a data processing device, the device comprising:
[0012] A memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program is configured to implement part or all of the steps described in any method in the first aspect.
[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a data processing program is stored. When the data processing program is executed by a processor, some or all of the steps described in any method in the first aspect are implemented.
[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product can be a software installation package.
[0015] By implementing the embodiment of the present application, the server first performs quality inspection on the preprocessed first target table data according to the first data detection model; then detects that there are abnormal data in the first target table data, and determines the abnormal value type of the abnormal data in the first target table data; then detects that the abnormal value types of multiple abnormal data in the first target table data are different, and the abnormal value processing strategies corresponding to the data of different abnormal value types in the first target table data are different, and determines the target abnormal value processing strategy according to the abnormal value type, the first quantity threshold and the second quantity threshold; finally, the abnormal data in the first target table data is processed according to the target abnormal value processing strategy. By determining the abnormal value processing strategy according to the abnormal value type, the data amount of different abnormal value types and the preset quantity threshold when the abnormal value type is different, it is beneficial to improve the efficiency of abnormal data processing in complex data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.
[0017] Figure 1 It is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;
[0018] Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application;
[0019] Figure 3 is a schematic diagram of a first prompt information provided in an embodiment of the present application;
[0020] Figure 4 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;
[0021] Figure 5 It is a structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0023] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0024] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0025] With the advent of the big data era, the amount of data processed by enterprises and organizations is increasing, and ensuring data quality has become a key issue. Especially in the ETL processing tasks of data warehouses, it is particularly important to perform quality inspection and exception handling on the target table data. These data may come from different sources and undergo multiple processing and conversions, so there may be various quality issues, such as missing data, erroneous data, duplicate data, etc.
[0026] At present, there are some data processing technologies for data quality detection and abnormal data processing, such as processing methods based on rule engines and processing methods based on statistical analysis. However, these methods have the following problems:
[0027] When dealing with data quality issues in complex scenarios, the rule engine-based processing method often fails to cover all situations, and the abnormal data processing strategy cannot be changed in time according to different situations.
[0028] When dealing with data quality issues in complex scenarios, statistical analysis may not be able to accurately capture the correlation between data and the exception handling strategies that should be adopted for different abnormal data.
[0029] It can be seen that the current abnormal data processing methods mainly rely on simple rule engines or statistical analysis for detection, and then adopt pre-configured processing strategies for the detected abnormal data. For data quality problems in complex scenarios, the existing technology is difficult to quickly cover all data anomalies when performing abnormal processing, and the efficiency of abnormal data processing is low.
[0030] In view of the above problems, the embodiments of the present application provide a data processing method, device, equipment, storage medium and program product, wherein the server first performs quality detection on the preprocessed first target table data according to the first data detection model; then detects that the first target table data has abnormal data, and determines the abnormal value type of the abnormal data in the first target table data; then detects that the abnormal value types of multiple abnormal data in the first target table data are different, and the abnormal value processing strategies corresponding to the data of different abnormal value types in the first target table data are different, and determines the target abnormal value processing strategy according to the abnormal value type, the first quantity threshold and the second quantity threshold; finally, the abnormal data in the first target table data is processed according to the target abnormal value processing strategy. By determining the abnormal value processing strategy according to the abnormal value type, the data amount of different abnormal value types and the preset quantity threshold when the abnormal value type is different, it is beneficial to improve the abnormal data processing efficiency in complex data scenarios.
[0031] The data processing method, apparatus, device, storage medium and program product provided in the embodiments of the present application can be applied to Figure 1 In the data processing system shown, see Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application. The data processing system 100 includes a terminal 101 and a server 102. The terminal 101 can communicate with the server 102 through a network. The terminal 101 refers to a device used by a user, such as a smart phone, a computer, etc. In this solution, the terminal 101 is provided with a user interface, which is mainly responsible for interacting with the data warehouse to query, analyze and visualize the data in the data warehouse. The user can interact with the system through the terminal 101; the server 102 refers to a remote computer used to process a large number of computing tasks and store data. In this solution, the server 102 is used to store data in the data warehouse and run the ETL (Extract, Transform, Load) process, which is responsible for extracting, cleaning, converting and loading data from the source system into the data warehouse. The data quality detection model or algorithm deployed on the server 102 can be used to analyze data and identify abnormal values. The server 102 can send abnormal prompt information to the terminal 101.
[0032] Based on this, the present application provides a data processing method, apparatus, device, storage medium and program product, and the present application is described in detail below in conjunction with the accompanying drawings.
[0033] See also Figure 2 , Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:
[0034] S201: Perform quality detection on the preprocessed first target table data according to a first data detection model.
[0035] Among them, the data of the first target table can be obtained from the data warehouse. The server of the data processing system can extract the required data from the data warehouse regularly through batch job scheduling tools such as Cron or Airflow. The server can also write query statements through Structured Query Language (SQL) and then extract the required data from the data warehouse. The data extracted by the server from the data warehouse can be exported to file formats, such as CSV or Excel, and then the files can be further analyzed through other tools.
[0036] Among them, after the server extracts the first target table data from the data warehouse, the preprocessing operations on the first target table data include data cleaning, feature selection and transformation, data normalization / standardization, etc. Specifically, the server first checks whether there are missing values in the first target table data, and chooses to delete the rows including missing values or fill the missing values by interpolation according to the missing value situation, and then determines whether there is duplicate data by comparing the matching between different fields, and removes it; if the first target table data includes multiple numerical type features and the scales of different numerical type features are different, the minimum-maximum normalization (Min-Max Scaling) or Z-score standardization (Z-score Standardization) can be used to process the data, wherein the minimum-maximum normalization linearly maps the data to a specified range, and the Z-score standardization converts the data into a normal distribution with a mean of 0 and a variance of 1.
[0037] In a possible implementation, before performing quality detection on the preprocessed first target table data according to the first data detection model, the method further includes:
[0038] Preprocess the third target table data in the data warehouse to obtain fourth target table data; obtain multiple feature data in the fourth target table data through feature extraction; and train the initial data detection model according to the multiple feature data to obtain the first data detection model.
[0039] Among them, preprocessing includes data cleaning, feature selection and transformation, data normalization / standardization, etc. Specifically, the server first checks whether there are missing values in the third target table data, and chooses to delete the rows including missing values or fill the missing values by interpolation according to the missing value situation, and then determines whether there is duplicate data by comparing the matching between different fields, and removes it; if the first target table data includes multiple numerical type features and the scales of different numerical type features are different, the minimum-maximum normalization (Min-Max Scaling) or Z-score standardization (Z-scoreStandardization) can be used to process the data, among which the minimum-maximum normalization linearly maps the data to a specified range, and the Z-score standardization converts the data into a normal distribution with a mean of 0 and a variance of 1.
[0040] Among them, feature extraction extracts the most representative and informative features from the original data for model training and prediction. The feature extraction method can be statistical features, frequency domain features or time domain features. Preferably, this solution adopts feature extraction based on statistical analysis.
[0041] Specifically, the model training steps are as follows: First, the original data set is divided into a training set, a validation set, and a test set. The training set is used to update the model's parameters, the validation set is used for hyperparameter tuning and model selection, and the test set is used to finally evaluate the model performance. Then, according to the problem type and the selected algorithm, a blank neural network model is initialized and the initial parameters are set. The training samples are input into the model for forward propagation calculation and a prediction result is generated. According to the difference between the prediction result and the true label, the loss function is calculated. Common loss functions include mean square error (MSE), cross entropy, etc. Then, the gradient of the loss function to each parameter is calculated through the back propagation algorithm, and the model parameters are updated using gradient descent or other optimization algorithms to minimize the loss function. In the periodic iteration, the learning rate can be set to a large value for initial training, and then the learning rate can be gradually reduced to make the convergence more stable. The impact of different hyperparameter combinations on model performance is evaluated through the validation set, and the best performing hyperparameter combination is selected as the final configuration. Then, regularization techniques (such as L1 regularization and L2 regularization) are applied to control model complexity and prevent overfitting. By adding regularization terms to the loss function, a balance is found between weighing fitting ability and generalization ability. Finally, the first data processing model that has been trained is evaluated using the test set data, and the corresponding indicators are obtained to complete the model training.
[0042] It can be seen that in this example, the data detection model is obtained by using multiple extracted feature information and combining it with the preprocessed input data training, which is beneficial to improving the accuracy and generalization performance of the data detection model to adapt to the data detection needs of different scenarios.
[0043] Among them, the first data detection model is a trained neural network model, the input of the first data detection model can be the preprocessed first target table data or the feature data of the preprocessed first target table data after feature extraction, and the output of the first data detection model is the quality detection result of the preprocessed first target table data. Specifically, the quality detection result may include outlier detection results, missing value conditions, and duplicate data conditions, etc.
[0044] Among them, in addition to detecting outliers and outlier types in the first target table data through the first data detection model, outliers can also be detected through statistical analysis, rule engines, and clustering algorithms, and then the outlier types and outlier processing strategies of the outliers are further determined based on the outliers. For example, a density clustering algorithm (DBSCAN) is used to detect outliers in a two-dimensional data set. Each data in the two-dimensional data set consists of two feature variables X and Y. The data is clustered by selecting a suitable domain radius and a minimum number of samples, and then the noise or isolated points in the data are determined to be outliers based on the clustering results. Then, the outlier type of the outlier data can be further analyzed for the outlier data.
[0045] S202: Detecting that abnormal data exists in the first target table data, and determining an abnormal value type of the abnormal data in the first target table data.
[0046] Among them, according to the different characteristics of outliers, the outlier types of outliers are usually divided into the following categories: single-point outliers, that is, a single data point that deviates significantly from the rest of the data set; upper and lower limit outliers, that is, data points that exceed the predetermined upper and lower limits; pattern outliers, that is, data points that are different from the normal data pattern; periodic outliers, that is, abnormal data points that appear at a specific time or frequency; cluster outliers, that is, a group of abnormal data points concentrated in a specific area of the data space.
[0047] In a possible implementation, when it is detected that the outlier value types of the plurality of outlier data in the first target table data are different, and the outlier value processing strategies corresponding to the data of different outlier value types in the first target table data are different, before determining the target outlier value processing strategy according to the outlier value type, the first quantity threshold, and the second quantity threshold, the method further includes:
[0048] It is detected that the outlier value types of multiple outlier data in the first target table data are the same, or the outlier value processing strategies corresponding to data of different outlier value types in the first target table data are the same, and the outlier value processing strategy corresponding to data of any outlier value type in the first target table data is determined to be the target outlier value processing strategy.
[0049] Among them, for outliers of different outlier types, the user can set the server's outlier processing strategies for different outlier types through the client. The outlier processing strategies corresponding to the outliers of each outlier type can be one or more. If the outlier types in the first target table data are different, but the outlier processing strategies configured for different outlier types exist with the same outlier processing strategy, then the outlier processing strategy is selected as the target outlier processing strategy.
[0050] Specifically, for single-point outliers, the outlier processing strategy can be to delete the outlier if it is caused by data collection or input errors, or mark it as an outlier for further analysis or processing, or replace the outlier with the median, mean or mode value of the overall data; for upper and lower limit outliers, the outlier processing strategy can be to trim the outlier to within the upper and lower limits, set the data points exceeding the upper limit as the upper limit value, set the data points below the lower limit as the lower limit value, or directly truncate and remove the data points exceeding the upper and lower limits; for mode outliers, the outlier processing strategy can be to adjust the weights, specifically to reduce the weights of these data points in the model to reduce their impact on the overall model, Or identify such outliers and separate them for independent analysis; for periodic outliers, the outlier processing strategy can be to apply the sliding window method to calculate local statistics (such as mean, standard deviation), and detect and process outliers based on these local statistics, such as directly converting outliers to means, or using Fourier transform or wavelet transform to identify and filter out these periodic abnormal signals in the frequency domain; for cluster outliers, the outlier processing strategy can be to resample these cluster outliers to reduce their impact on the overall data set, or use clustering algorithms such as K-means and DBSCAN to identify these abnormal clusters and process them separately, such as deleting such cluster outliers.
[0051] Exemplarily, after quality inspection of the preprocessed first target table data according to the first data inspection model, it is determined that there is only one type of abnormal data in the preprocessed first target table data, and the type of abnormal data is a single-point abnormal value. At this time, the server will use the pre-configured abnormal value processing strategy for single-point abnormal values as the target abnormal value processing strategy, that is, further determine that the abnormal cause of the single-point abnormal value is caused by data collection errors, and delete all single-point abnormal values. Alternatively, after quality inspection of the preprocessed first target table data according to the first data inspection model, it is determined that there are multiple types of abnormal data in the preprocessed first target table data, and the types of abnormal values include single-point abnormal values and cluster abnormal values. At this time, the server will traverse the pre-configured abnormal value processing strategy for single-point abnormal values and the abnormal value processing strategy for cluster abnormal values. Both of the abnormal value processing strategies have the direct deletion of abnormal values, so deleting abnormal values is used as the target abnormal value processing strategy.
[0052] It can be seen that in this example, multiple outlier processing strategies are configured for different outlier types. When the same outlier processing strategy is detected for different outlier types, applying the same outlier processing strategy to different types of outlier processing can improve the efficiency of the system in outlier processing.
[0053] S203, detecting that the outlier value types of multiple outlier data in the first target table data are different, and the outlier value processing strategies corresponding to the data of different outlier value types in the first target table data are different, and determining the target outlier value processing strategy according to the outlier value type, the first quantity threshold and the second quantity threshold.
[0054] Among them, for outliers of different outlier types, the user can set the server's outlier processing strategies for different outlier types through the client, and the outlier processing strategies corresponding to outliers of each outlier type can be one or more.
[0055] Specifically, for single-point outliers, the outlier processing strategy can be to delete the outlier if it is caused by data collection or input errors, or mark it as an outlier for further analysis or processing, or replace the outlier with the median, mean or mode value of the overall data; for upper and lower limit outliers, the outlier processing strategy can be to trim the outlier to within the upper and lower limits, set the data points exceeding the upper limit as the upper limit value, set the data points below the lower limit as the lower limit value, or directly truncate and remove the data points exceeding the upper and lower limits; for mode outliers, the outlier processing strategy can be to adjust the weights, specifically to reduce the weights of these data points in the model to reduce their impact on the overall model, Or identify such outliers and separate them for independent analysis; for periodic outliers, the outlier processing strategy can be to apply the sliding window method to calculate local statistics (such as mean, standard deviation), and detect and process outliers based on these local statistics, such as directly converting outliers to means, or using Fourier transform or wavelet transform to identify and filter out these periodic abnormal signals in the frequency domain; for cluster outliers, the outlier processing strategy can be to resample these cluster outliers to reduce their impact on the overall data set, or use clustering algorithms such as K-means and DBSCAN to identify these abnormal clusters and process them separately, such as deleting such cluster outliers.
[0056] In a possible implementation, determining a target outlier processing strategy according to the outlier type, the first quantity threshold, and the second quantity threshold includes:
[0057] When it is detected that the number of the outlier value types does not exceed the first quantity threshold and the amount of outlier data in the first target table data does not exceed the second quantity threshold, corresponding outlier processing strategies are adopted for the outlier data of different outlier value types; when it is detected that the number of the outlier value types exceeds the first quantity threshold, or the amount of outlier data in the first target table data exceeds the second quantity threshold, all outlier data are standardized or normalized.
[0058] The first quantity threshold is used to limit the number of abnormal value types, and the second quantity threshold is used to limit the amount of abnormal data in the first target table data. The first quantity threshold and the second quantity threshold can be changed and set on the server.
[0059] Exemplarily, target table data is obtained from a data warehouse, and the target table data includes the purchase behavior of users on website A. In order to detect and process abnormal situations in user purchase behaviors, the first quantity threshold is set to 2, and the second quantity threshold is set to 50. When an abnormal value is detected in the target table data, the abnormal value types include single-point abnormal values and upper and lower limit abnormal values. The amount of data with single-point abnormal values is 10, and the amount of data with upper and lower limit abnormal values is 25. At this time, the type of abnormal value in the target table data is 2, which does not exceed the first quantity threshold, and the amount of abnormal data in the target table data is 35, which is less than the second quantity threshold. Therefore, corresponding abnormal value processing strategies are adopted for abnormal data of different abnormal value types; or, the target table data is obtained from a data warehouse, and the target table data is The data includes the purchase behavior of users on website B. In order to detect and process the anomalies in the user's purchase behavior, the first quantity threshold is set to 2 and the second quantity threshold is set to 50. When it is detected that there are outliers in the target table data, the outlier types include single-point outliers, upper and lower limit outliers, and cluster outliers. The amount of data with single-point outliers is 10, the amount of data with upper and lower limit outliers is 25, and the amount of data with cluster outliers is 20. At this time, the outlier type in the target table data is 3, which is greater than the first quantity threshold, and the amount of abnormal data in the target table data is 55, which is greater than the second quantity threshold. Therefore, the same outlier processing strategy is adopted for abnormal data of different outlier types, that is, all outliers are normalized or standardized.
[0060] Among them, normalization processing includes minimum-maximum normalization (Min-Max Scaling) and decimal scaling normalization (Decimal Scaling), etc. Minimum-maximum normalization is used to linearly map data to a specified range, and decimal scaling normalization is used to divide data by a fixed factor to achieve scaling; standardization processing includes Z-score standardization (Z-scoreStandardization) and range standardization, etc.
[0061] In a possible implementation, each outlier type corresponds to one or more outlier processing strategies. After determining the outlier type of multiple outlier data in the first target table data, the multiple outlier data are divided into multiple outlier data sets. The multiple outlier data in each outlier data set have the same first outlier processing strategy. The multiple outlier data in each outlier data set are processed using the first outlier processing strategy.
[0062] Among them, when setting corresponding multiple outlier processing strategies for each outlier type, priorities can be set for multiple outlier processing strategies. For example, for single-point outliers, the multiple outlier processing strategies set are sorted by priority as follows: delete outliers, mark outliers, and replace outliers with medians. At this time, if a certain abnormal data that needs to be divided is a single-point outlier, and the three outlier processing strategies corresponding to the single-point outlier all have corresponding abnormal data sets, then the abnormal data set to which the abnormal data to be divided is classified is selected according to the priority of the abnormal processing strategy. For example, the processing strategy of abnormal data set 1 is to delete outliers, the processing strategy of abnormal data set 2 is to mark outliers, and the processing strategy of abnormal data set 3 is to replace outliers with medians. At this time, since the priority processing strategy of single-point outliers is to delete outliers, the abnormal data is classified into abnormal data set 1 at once.
[0063] When the amount of abnormal data does not exceed the third quantity threshold, the abnormal data is not divided, and each abnormal data is directly processed according to the preferred processing strategy corresponding to the abnormal value type of each abnormal data. For example, if the third quantity threshold is 10 and the amount of abnormal data currently detected is 3, then the abnormal data is directly processed according to the abnormal type of each abnormal data using the corresponding abnormal value processing strategy.
[0064] For example, there are 5 abnormal data in the first target table data, namely abnormal data 1, abnormal data 2, abnormal data 3, abnormal data 4, and abnormal data 5. The abnormal value types of abnormal data 1 and abnormal data 2 are the same, and the abnormal number types of abnormal data 3, abnormal number 4, and abnormal data 5 are different. At this time, each abnormal value type corresponds to two abnormal value processing strategies, the abnormal value processing strategies of abnormal data 1 and abnormal data 2 are strategy 1 or strategy 2, the abnormal value processing strategy of abnormal data 3 is strategy 3 or strategy 4, the abnormal value processing strategy of abnormal data 4 is strategy 4 or strategy 5, and the abnormal value processing strategy of abnormal data 5 is strategy 6 or strategy 7. According to whether multiple abnormal data have the same abnormal value processing strategy, the 5 abnormal data are divided into 3 data sets, namely {abnormal data 1, abnormal data 2} (corresponding to strategy 1 or strategy 2), {abnormal data 3, abnormal data 4} (corresponding to strategy 4), and {abnormal data 5} (corresponding to strategy 6 or strategy 7).
[0065] At the same time, the original multiple outliers need to execute the outlier processing strategy in sequence in the same processor. After the outliers are divided into multiple outlier sets, the multiple outlier sets can be further processed in parallel by the processors of multiple computers in the distributed system. For example, if it takes 1 second to process one outlier, then it takes a total of 100 seconds to process 100 outliers. After the distributed system is used to divide these outliers into multiple sets, different sets can be processed simultaneously on different computing nodes (such as different servers or processor cores). If the 100 outliers are divided into 10 sets, each set has 10 outliers, and assuming that the distributed system has 10 computing nodes, then each computing node processes one set at the same time, and it only takes 10 seconds to process all the outliers.
[0066] It can be seen that in this example, by first determining the outlier value type of the abnormal data, and then dividing the abnormal data according to the same first outlier value processing strategy to form an abnormal data set, a unified strategy can be used to process multiple data in the abnormal data set with the same processing strategy. At the same time, after the data is divided into multiple data sets, a distributed system is used for processing, so that the original task is allocated to other idle computing resources, which is conducive to more efficient abnormal data processing and improved computer efficiency.
[0067] In a possible implementation, a first ratio is obtained according to the number of abnormal data and the number of abnormal value types in the first target table data, and an abnormal value processing strategy for the abnormal data is determined according to the first ratio and a ratio threshold.
[0068] Among them, the first ratio is obtained by calculating the quotient of the number of outlier value types in the first target table data and the number of abnormal data. The first ratio can reflect the richness of the outlier value types of unit abnormal data in the first target table, and can also characterize the complexity of the abnormal data in the first target table. The first ratio is high, indicating that the number of abnormal data corresponding to each outlier value type in the first target table data is small, that is, more outlier value types need to be considered when processing abnormal data. For example, the first ratio is 1, indicating that each outlier value type corresponds to only one abnormal data, and the first ratio is 0.5, indicating that each outlier value type corresponds to two abnormal data.
[0069] By setting a ratio threshold and comparing the ratio threshold with the first ratio, the number of abnormal data corresponding to each abnormal value type in the first target table can be determined, and then a suitable abnormal value processing strategy can be selected.
[0070] Specifically, when the first ratio is greater than the ratio threshold, the type of outliers in the unit abnormal data is more abundant, the abnormal data is more complex, and the number of abnormal data corresponding to each outlier type is relatively small. At this time, the outlier processing strategy can be refined processing, and each outlier type needs to be analyzed and processed separately. For example, a detailed rule base can be established for different outlier types, and different corrective measures can be taken according to the characteristics of the outlier type.
[0071] When the first ratio is equal to the ratio threshold, the number of abnormal data corresponding to each abnormal value type is relatively moderate. The abnormal value processing strategy can be a comprehensive processing, that is, a combination of automated processing and manual inspection is used to batch process abnormal data according to preset rules, and at the same time, manual sampling inspection is arranged for the processed results.
[0072] When the first ratio is less than the ratio threshold, the type of outliers in the unit outlier data is less abundant, and the number of outlier data corresponding to each outlier type is large. At this time, the outlier processing strategy can be batch processing, and the outlier types with a large proportion are uniformly corrected. For example, if most of the outlier data is caused by data unit conversion errors, the outliers are directly processed by batch modifying the data units. For outlier types with a small proportion, a general processing method can be adopted according to their characteristics, for example, a small number of outlier outliers can be directly deleted.
[0073] It can be seen that in this example, by setting the first ratio and the ratio threshold, the richness of the outlier value types of the unit abnormal data in the target table can be comprehensively considered, so as to determine whether the data processing strategy adopts refined processing or batch processing. When the data in the target table continues to increase, the processing strategy can be dynamically adjusted, which is conducive to effectively processing abnormal data in various data scenarios.
[0074] S204: Process the abnormal data in the first target table data according to the target abnormal value processing strategy.
[0075] It can be seen that in this example, by determining the adopted outlier processing strategy according to the outlier type, the data volume of different outlier types and the preset quantity threshold when there are multiple outlier types, it is beneficial to improve the efficiency of abnormal data processing in complex data scenarios.
[0076] In a possible implementation, the method further includes:
[0077] After detecting that an abnormality exists in the first target table data, a first prompt message is sent, and the first prompt message includes the abnormal value type of the first target table data, and / or the data amount of abnormal data corresponding to different abnormal value types; after detecting that the abnormal data in the first target table data is processed according to the target abnormal value processing strategy, a second prompt message is sent, and the second prompt message is used to prompt that the abnormal data processing is completed; after detecting that an abnormality exists in the first target table data, and the abnormal data in the first target table data is not processed according to the target abnormal value processing strategy after a first time threshold, a third prompt message is sent, and the third prompt message is used to prompt that the data processing server is abnormal.
[0078] Among them, the first prompt information, the second prompt information and the third prompt information can be notified via email, text message or pop-up window in the application.
[0079] Among them, see Figure 3 , Figure 3 is a schematic diagram of a first prompt information provided in an embodiment of the present application, such as Figure 3 As shown, the first prompt information can be a histogram, which includes outlier type information and data volume information corresponding to the outlier type. The user can intuitively understand the abnormal situation through the histogram. At the same time, the user can enter the configuration interface of the corresponding outlier type by clicking the text or image corresponding to the outlier type in the histogram. In this configuration interface, the user can select the outlier processing strategy for the outlier type. For example, the user can enter the configuration interface of outlier type 1 by clicking image A or text B in the histogram, and thus select the outlier processing strategy for outlier type 1.
[0080] It can be seen that by sending various prompt information, real-time feedback of the data processing system can be provided to users, helping users to quickly detect and resolve abnormal situations and enhance users' perception of the data processing system.
[0081] See also Figure 4 , Figure 4 is a structural diagram of a data processing device provided in an embodiment of the present application, such as Figure 4 As shown, the data processing device 400 includes:
[0082] A detection module 401 is used to perform quality detection on the preprocessed first target table data according to a first data detection model;
[0083] Determination module 402, for detecting that there are abnormal data in the first target table data, determining the abnormal value type of the abnormal data in the first target table data; and for detecting that the abnormal value types of multiple abnormal data in the first target table data are different, and the abnormal value processing strategies corresponding to the data of different abnormal value types in the first target table data are different, and determining the target abnormal value processing strategy according to the abnormal value type, the first quantity threshold and the second quantity threshold;
[0084] The processing module 403 is used to process the abnormal data in the first target table data according to the target abnormal value processing strategy.
[0085] In a possible implementation, when it is detected that the outlier value types of multiple outlier data in the first target table data are different, and the outlier value processing strategies corresponding to the data of different outlier value types in the first target table data are different, before determining the target outlier value processing strategy according to the outlier value type, the first quantity threshold and the second quantity threshold, the determination module 402 is also used to: detect that the outlier value types of multiple outlier data in the first target table data are the same, or the outlier value processing strategies corresponding to the data of different outlier value types in the first target table data are the same, and determine that the outlier value processing strategy corresponding to the data of any outlier value type in the first target table data is the target outlier value processing strategy.
[0086] In a possible implementation, in determining the target outlier processing strategy according to the outlier type, the first quantity threshold and the second quantity threshold, the determination module 402 is specifically used to: when it is detected that the number of the outlier type does not exceed the first quantity threshold and the data amount of the outlier data in the first target table data does not exceed the second quantity threshold, adopt corresponding outlier processing strategies for the outlier data of different outlier types;
[0087] When it is detected that the number of the abnormal value types exceeds a first quantity threshold, or the data amount of abnormal data in the first target table data exceeds a second quantity threshold, all abnormal data are standardized or normalized.
[0088] In a possible implementation, before performing quality inspection on the preprocessed first target table data according to the first data detection model, the detection module 401 is also used to: preprocess the second target table data in the data warehouse to obtain third target table data; obtain multiple feature data in the third target table data through feature extraction; and train the initial data detection model according to the multiple feature data to obtain the first data detection model.
[0089] In a possible implementation, the preprocessing includes removing duplicate data, processing missing values, and format conversion, and the feature extraction is feature extraction based on statistical analysis.
[0090] In a possible implementation, the processing module 403 is also used to: after detecting that an abnormality exists in the first target table data, send a first prompt message, the first prompt message including the abnormal value type of the first target table data, and / or the data amount of abnormal data corresponding to different abnormal value types; after detecting that the abnormal data in the first target table data is processed according to the target abnormal value processing strategy, send a second prompt message, the second prompt message is used to prompt that the abnormal data processing is completed; after detecting that an abnormality exists in the first target table data, and the abnormal data in the first target table data is not processed according to the target abnormal value processing strategy after a first time threshold, send a third prompt message, the third prompt message is used to prompt that the data processing server is abnormal.
[0091] It is worth noting that the specific functional implementation of the data processing device 400 is shown in the above Figure 2 The description of the data processing method shown in the figure, for example, the detection module 401 is used to implement the relevant content of executing S201, the determination module 402 is used to implement the relevant content of executing S202-S203, and the processing module 403 is used to implement the relevant content of executing S204. Each unit or module in the data processing device 400 can be separately or completely combined into one or more other units or modules to constitute, or one (some) of the units or modules can also be further divided into multiple smaller units or modules in function to constitute, which can achieve the same operation without affecting the realization of the technical effect of the embodiment of the present invention. The above-mentioned units or modules are divided according to logical functions. In practical applications, the function of one unit (or module) is realized by multiple units (or modules), or the function of multiple units (or modules) is realized by one unit (or module).
[0092] According to the description of the above method embodiment and related device embodiment, please refer to Figure 5 , an embodiment of the present invention also provides a structural schematic diagram of a data processing device 500. Figure 5 The data processing device 500 shown includes a processor 501 , a memory 502 , a communication interface 503 and a bus 504 . The processor 501 , the memory 502 , and the communication interface 503 are connected to each other through the bus 504 .
[0093] Optionally, the memory 502 is a ROM, a static storage device, a dynamic storage device or a RAM.
[0094] The memory 502 can store executable program codes. When the executable program codes stored in the memory 502 are executed by the processor 501, the processor 501 and the communication interface 503 are used to execute Figure 2The various steps of the data processing method of the illustrated embodiment.
[0095] The processor 501 adopts a general-purpose CPU, a microprocessor, an application-specific integrated circuit ASIC, a GPU or one or more integrated circuits to execute relevant programs to perform the data processing method of the method embodiment of the present application.
[0096] The processor 501 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the data processing method of the present application can be completed by the hardware integrated logic circuit or software instructions in the processor 501. Optionally, the processor 501 is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The processor can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor is a microprocessor or the processor is any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. The optional software module is located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502, and combines its hardware to complete the functions required to be executed by the modules included in the data processing device 400 of the embodiment of the present application, or executes the data processing method of the method embodiment of the present application.
[0097] The communication interface 503 uses a transceiver or other related device such as, but not limited to, a transceiver.
[0098] Bus 504 may include a path for transmitting information between various components of data processing device 500 (eg, memory 502 , processor 501 , communication interface 503 ).
[0099] It should be noted that although Figure 5 The data processing device 500 shown only shows a memory, a processor, and a communication interface. However, in the specific implementation process, those skilled in the art should understand that the data processing device 500 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the data processing device 500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the data processing device 500 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 5 All devices shown in .
[0100] An embodiment of the present application provides a computer-readable storage medium, in which a computer program for electronic data exchange is stored. The computer program includes execution instructions, and the execution instructions are used to execute part or all of the steps of any one of the data processing methods recorded in the above-mentioned data processing method embodiments. The above-mentioned computer includes an electronic terminal device.
[0101] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program can be operated to enable a computer to perform part or all of the steps of any data processing method recorded in the above method embodiments. The computer program product can be a software installation package.
[0102] It should be noted that, for the sake of simplicity, the embodiments of any of the aforementioned data processing methods are described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application.
[0103] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of a data processing method, device, equipment, storage medium and program product of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, based on the idea of a data processing method, device, equipment, storage medium and program product of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, hardware products, and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 The memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0106] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude multiple situations. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.
[0107] A person skilled in the art can understand that all or part of the steps in the various methods of the method embodiments of any of the above-mentioned data processing methods can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a disk or an optical disk, etc.
[0108] It can be understood that any product that is controlled or configured to execute the processing method of the flowchart described in an embodiment of a data processing method of the present application, such as the device and computer program product of the above flowchart, falls within the scope of the related products described in the present application.
[0109] Obviously, those skilled in the art can make various changes and modifications to the data processing method, device, equipment, storage medium and program product provided by the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A data processing method, characterized in that: The method comprises: Performing quality detection on the preprocessed first target table data according to the first data detection model; Detecting that abnormal data exists in the first target table data, and determining an abnormal value type of the abnormal data in the first target table data; It is detected that the outlier value types of the plurality of outlier data in the first target table data are different, and the outlier value processing strategies corresponding to the data of different outlier value types in the first target table data are different, and a target outlier value processing strategy is determined according to the outlier value type, the first quantity threshold, and the second quantity threshold; The abnormal data in the first target table data is processed according to the target abnormal value processing strategy.
2. The method according to claim 1, characterized in that After detecting that the outlier value types of the plurality of outlier data in the first target table data are different, and the outlier value processing strategies corresponding to the data of different outlier value types in the first target table data are different, before determining the target outlier value processing strategy according to the outlier value type, the first quantity threshold, and the second quantity threshold, the method further includes: It is detected that the outlier value types of multiple outlier data in the first target table data are the same, or the outlier value processing strategies corresponding to data of different outlier value types in the first target table data are the same, and the outlier value processing strategy corresponding to data of any outlier value type in the first target table data is determined to be the target outlier value processing strategy.
3. The method according to claim 2, characterized in that The determining of a target outlier processing strategy according to the outlier type, the first quantity threshold, and the second quantity threshold includes: When it is detected that the number of the abnormal value types does not exceed the first number threshold and the data volume of the abnormal data in the first target table data does not exceed the second number threshold, adopting corresponding abnormal value processing strategies for abnormal data of different abnormal value types; When it is detected that the number of the abnormal value types exceeds a first quantity threshold, or the data amount of abnormal data in the first target table data exceeds a second quantity threshold, all abnormal data are standardized or normalized.
4. The method according to claim 1, characterized in that Before performing quality detection on the preprocessed first target table data according to the first data detection model, the method further includes: Preprocessing the second target table data in the data warehouse to obtain the third target table data; Acquire multiple feature data in the third target table data by feature extraction; The initial data detection model is trained according to the multiple feature data to obtain a first data detection model.
5. The method according to claim 4, characterized in that The preprocessing includes removing duplicate data, processing missing values and format conversion, and the feature extraction is feature extraction based on statistical analysis.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: After detecting that an abnormality exists in the first target table data, a first prompt message is sent, and the first prompt message includes the abnormal value type of the first target table data, and / or the data amount of abnormal data corresponding to different abnormal value types; after detecting that the abnormal data in the first target table data is processed according to the target abnormal value processing strategy, a second prompt message is sent, and the second prompt message is used to prompt that the abnormal data processing is completed; after detecting that an abnormality exists in the first target table data, and the abnormal data in the first target table data is not processed according to the target abnormal value processing strategy after a first time threshold, a third prompt message is sent, and the third prompt message is used to prompt that the data processing server is abnormal.
7. A data processing device, characterized in that: The device comprises: A detection module, used for performing quality detection on the preprocessed first target table data according to the first data detection model; a determination module, configured to detect that abnormal data exists in the first target table data, determine the abnormal value type of the abnormal data in the first target table data; and to detect that the abnormal value types of multiple abnormal data in the first target table data are different, and that the abnormal value processing strategies corresponding to the data of different abnormal value types in the first target table data are different, and determine the target abnormal value processing strategy according to the abnormal value type, the first quantity threshold and the second quantity threshold; A processing module is used to process the abnormal data in the first target table data according to the target abnormal value processing strategy.
8. A data processing device, characterized in that: The device comprises: A memory, a processor, and an executable program code stored in the memory and executable on the processor, wherein the processor executes the steps of the data processing method according to any one of claims 1 to 6 when executing the executable program code.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable program code, wherein the executable program code includes execution instructions, and the execution instructions are used to execute the steps of the data processing method according to any one of claims 1 to 6.
10. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is used to make a computer execute the steps of the data processing method according to any one of claims 1 to 6.