Data processing method and system

Abstract

The invention relates to a data processing method and system.The method comprises the steps that first configuration information and second configuration information are obtained, the first configuration information comprises a source data source and a data processing strategy, and the second configuration information comprises a target data source, a storage format, a partitioning strategy and a cleaning strategy; the data processing strategy supports a self-defined script, a self-defined variable and a self-defined function, real-time data is collected according to a source data source, the real-time data is processed in a mode of quoting the self-defined variable and the self-defined function through a placeholder in the self-defined script, and the real-time data is partitioned and cleaned; and storing the final data to the target data source according to the storage format. Therefore, according to the method and the system, diversified user-defined configuration of a user is supported, diversified business requirements are met, the user-defined variables and the user-defined functions are quoted in the user-defined script in a placeholder mode, the data processing efficiency is improved, partitioning and cleaning are combined, and the data value is better protected.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and system. Background Art

[0002] Traditional data management often uses a single architecture to store all data. Data processing is usually performed in batches and cannot meet real-time data processing needs. When layering data, it is limited to the data surface, usually in the form of data tables. Subsequent data analysis is also based on data tables, which limits the depth of data analysis. In addition, due to the increasing diversity and complexity of data, traditional data management often finds it difficult to meet the personalized needs of different business departments for data processing and application. Summary of the invention

[0003] The technical problem to be solved by the present invention is: the present invention provides a data processing method and system to ensure the flexibility of data processing to adapt to diverse business needs, while protecting the value of data and improving the efficiency of data processing.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] In a first aspect, the present invention provides a method for data processing, comprising:

[0006] Obtaining first configuration information and second configuration information, wherein the first configuration information includes a source data source and a data processing strategy, and the second configuration information includes a target data source, a storage format, a partitioning strategy, and a cleaning strategy, and the data processing strategy supports custom scripts, custom variables, and custom functions;

[0007] Collecting corresponding real-time data according to the source data source, combining the custom script with the custom variables and the custom functions, and processing the real-time data by referencing the custom variables and the custom functions by placeholders in the custom script to obtain processed data;

[0008] The processed data is partitioned according to the partitioning strategy to obtain partitioned data, the partitioned data is cleaned according to the cleaning strategy to obtain cleaned data, and the cleaned data is stored in the target data source according to the storage format.

[0009] The beneficial effects of the present invention are: supporting users to perform customized configuration, and the customized configuration includes source data source, data processing strategy, target data source, storage format, partition strategy and cleaning strategy, ensuring the breadth and depth of user customized configuration, combining data processing with personalization, improving data processing flexibility, and better protecting data value. The data processing strategy supports customized scripts, customized variables and customized functions, that is, adapting to diversified business needs, and when processing real-time data, the customized script is combined with the customized variables and customized functions, and the customized variables and customized functions are referenced in the customized script by placeholders, which improves the automation and processing efficiency of data processing, while ensuring the flexibility and maintainability of data processing, and the processed data is not only partitioned but also cleaned, that is, partitioning is combined with cleaning, which better protects the data value and broadens the depth of subsequent data analysis.

[0010] Optionally, the data processing strategy includes a throughput strategy, a dirty data strategy, a read concurrency strategy, and a query strategy. The real-time data is processed by referencing the custom variables and the custom functions by placeholders in the custom script to obtain the processed data, including:

[0011] In the custom script of the throughput strategy, the transmission rate of the real-time data is processed by referencing the corresponding custom throughput threshold variable and the custom throughput function by placeholders to obtain preliminary processing data;

[0012] In the custom script of the dirty data strategy, dirty data in the preliminary processed data is marked by referencing the corresponding custom dirty data threshold variable and the custom dirty data function by placeholders to obtain marked preliminary processed data;

[0013] In the custom script of the read concurrency strategy, the marked preliminary processing data is read by referencing the corresponding custom read concurrency threshold variable and the custom read concurrency function through the placeholder to obtain the read data;

[0014] In the custom script of the query strategy, the read data is queried by referring to the corresponding custom query variables and custom query functions through placeholders to obtain processed data.

[0015] According to the above description, data processing strategies include throughput strategies, dirty data strategies, read concurrency strategies, query strategies and other strategies, which improve the flexibility of data processing. The throughput strategy limits the transmission rate of real-time data to prevent excessive burden on the system. The dirty data strategy is used to mark dirty data, which helps data cleaning and quality control. The read concurrency strategy speeds up data reading and improves processing efficiency. The query strategy is used to meet personalized query needs and optimize user experience.

[0016] Optionally, partitioning the processed data according to the partitioning strategy to obtain partitioned data includes:

[0017] Obtaining a partition category from the partition strategy, performing a primary partition on the processed data according to the partition category to obtain preliminary partition data, obtaining a partition parameter, a partition size and a partition key from the partition category, performing a secondary partition on the preliminary partition data according to the partition parameter, the partition size and the partition key to obtain partitioned data;

[0018] The partition categories include timestamp, geographic location and user ID.

[0019] According to the above description, the processed data is partitioned by multi-level partitioning, and partitions at different levels have diverse indicators, which improves the breadth and depth of partitions, better protects the value of data, and optimizes data management and data analysis.

[0020] Optionally, the cleaning strategy includes a missing value processing strategy, a data conversion strategy and a filtering strategy, and the partitioned data is cleaned according to the cleaning strategy to obtain the cleaned data including:

[0021] Fill the missing values ​​in the partitioned data using the mean, median or mode according to the missing value processing strategy to obtain filled data;

[0022] Performing standardization processing on the filled data according to the data conversion strategy to obtain standardized data, wherein the standardization processing includes standardizing the date, standardizing the timestamp format and standardizing the currency symbol;

[0023] A temporary buffer is created, and the standardized data is deduplicated in the temporary buffer according to the filtering strategy to obtain the deduplicated standardized data and release the temporary buffer at the same time, and the deduplicated standardized data is segmented and stemmed by Apache Lucene to obtain cleaned data.

[0024] According to the above description, the cleaning strategy includes missing value processing strategy, data conversion strategy and filtering strategy. The diversified cleaning strategies improve the comprehensiveness and completeness of data cleaning. The data conversion strategy is used to standardize the data and optimize data management. When the data is processed by the filtering strategy, the data is deduplicated by creating a temporary buffer to ensure the stable operation of the system. Apache Lucene is used for word segmentation and stem extraction, providing powerful text processing capabilities. Compared with basic text processing functions, it can better explore the value of data.

[0025] Optionally, the cleaning strategy includes an exception handling strategy, and obtaining standardized data includes:

[0026] An alarm is issued for abnormal values ​​in the standardized data according to an abnormality handling strategy.

[0027] According to the above description, the abnormal value alarm is issued through the abnormal handling strategy, which enhances the security and quality control of the data and improves the reliability of the data.

[0028] In a second aspect, the present invention provides a data processing system, comprising:

[0029] An acquisition module, used to acquire first configuration information and second configuration information, wherein the first configuration information includes a source data source and a data processing strategy, and the second configuration information includes a target data source, a storage format, a partition strategy, and a cleaning strategy, and the data processing strategy supports custom scripts, custom variables, and custom functions;

[0030] a processing module, configured to collect corresponding real-time data according to the source data source, combine the custom script with the custom variables and the custom functions, and process the real-time data by referencing the custom variables and the custom functions by placeholders in the custom script to obtain processed data;

[0031] The target module is used to partition the processed data according to the partitioning strategy to obtain partitioned data, clean the partitioned data according to the cleaning strategy to obtain cleaned data, and store the cleaned data to the target data source according to the storage format.

[0032] The beneficial effects of the present invention are: supporting users to perform customized configuration, and the customized configuration includes source data source, data processing strategy, target data source, storage format, partition strategy and cleaning strategy, ensuring the breadth and depth of user customized configuration, combining data processing with personalization, improving data processing flexibility, and better protecting data value. The data processing strategy supports customized scripts, customized variables and customized functions, that is, adapting to diversified business needs, and when processing real-time data, the customized script is combined with the customized variables and customized functions, and the customized variables and customized functions are referenced in the customized script by placeholders, which improves the automation and processing efficiency of data processing, while ensuring the flexibility and maintainability of data processing, and the processed data is not only partitioned but also cleaned, that is, partitioning is combined with cleaning, which better protects the data value and broadens the depth of subsequent data analysis.

[0033] Optionally, the data processing strategy includes a throughput strategy, a dirty data strategy, a read concurrency strategy and a query strategy, and the processing module is specifically:

[0034] In the custom script of the throughput strategy, the transmission rate of the real-time data is processed by referencing the corresponding custom throughput threshold variable and the custom throughput function by placeholders to obtain preliminary processing data;

[0035] In the custom script of the dirty data strategy, dirty data in the preliminary processed data is marked by referencing the corresponding custom dirty data threshold variable and the custom dirty data function by placeholders to obtain marked preliminary processed data;

[0036] In the custom script of the read concurrency strategy, the marked preliminary processing data is read by referencing the corresponding custom read concurrency threshold variable and the custom read concurrency function through the placeholder to obtain the read data;

[0037] In the custom script of the query strategy, the read data is queried by referring to the corresponding custom query variables and custom query functions through placeholders to obtain processed data.

[0038] According to the above description, data processing strategies include throughput strategies, dirty data strategies, read concurrency strategies, query strategies and other strategies, which improve the flexibility of data processing. The throughput strategy limits the transmission rate of real-time data to prevent excessive burden on the system. The dirty data strategy is used to mark dirty data, which helps data cleaning and quality control. The read concurrency strategy speeds up data reading and improves processing efficiency. The query strategy is used to meet personalized query needs and optimize user experience.

[0039] Optionally, the target module includes:

[0040] A partitioning module, used to obtain a partitioning category from the partitioning strategy, perform a primary partitioning on the processed data according to the partitioning category to obtain preliminary partitioning data, obtain a partitioning parameter, a partitioning size and a partitioning key from the partitioning category, perform a secondary partitioning on the preliminary partitioning data according to the partitioning parameter, the partitioning size and the partitioning key to obtain partitioned data;

[0041] The partition categories include timestamp, geographic location and user ID.

[0042] According to the above description, the processed data is partitioned by multi-level partitioning, and partitions at different levels have diverse indicators, which improves the breadth and depth of partitions, better protects the value of data, and optimizes data management and data analysis.

[0043] Optionally, the cleaning strategy includes a missing value processing strategy, a data conversion strategy and a filtering strategy, and the target module includes:

[0044] A cleaning module, used for filling missing values ​​in the partitioned data using a mean, a median, or a mode according to a missing value processing strategy to obtain filled data;

[0045] Performing standardization processing on the filled data according to the data conversion strategy to obtain standardized data, wherein the standardization processing includes standardizing the date, standardizing the timestamp format and standardizing the currency symbol;

[0046] A temporary buffer is created, and the standardized data is deduplicated in the temporary buffer according to the filtering strategy to obtain the deduplicated standardized data and release the temporary buffer at the same time, and the deduplicated standardized data is segmented and stemmed by Apache Lucene to obtain cleaned data.

[0047] According to the above description, the cleaning strategy includes missing value processing strategy, data conversion strategy and filtering strategy. The diversified cleaning strategies improve the comprehensiveness and completeness of data cleaning. The data conversion strategy is used to standardize the data and optimize data management. When the data is processed by the filtering strategy, the data is deduplicated by creating a temporary buffer to ensure the stable operation of the system. Apache Lucene is used for word segmentation and stem extraction, providing powerful text processing capabilities. Compared with basic text processing functions, it can better explore the value of data.

[0048] Optionally, the cleaning module is specifically:

[0049] An alarm is issued for abnormal values ​​in the standardized data according to an abnormality handling strategy.

[0050] According to the above description, the abnormal value alarm is issued through the abnormal handling strategy, which enhances the security and quality control of the data and improves the reliability of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of a data processing method provided by this embodiment;

[0052] Figure 2 A schematic diagram of the overall flow of a data processing method provided in this embodiment;

[0053] Figure 3 A schematic diagram of the structure of a data processing system provided in this embodiment.

[0054] [Description of Reference Numerals]

[0055] 1. A data processing system;

[0056] 2. Get the module;

[0057] 3. Processing module;

[0058] 4. Target module; 41 partition module; 42 cleaning module. DETAILED DESCRIPTION

[0059] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0060] Embodiment 1

[0061] Please refer to Figure 1 to Figure 2 The present invention provides a method for data processing, comprising the steps of:

[0062] S1. Obtain first configuration information and second configuration information, wherein the first configuration information includes a source data source and a data processing strategy, and the second configuration information includes a target data source, a storage format, a partitioning strategy, and a cleaning strategy, and the data processing strategy supports custom scripts, custom variables, and custom functions;

[0063] In this embodiment, users are supported to customize the input and output of data, such as Figure 2As shown, the first configuration information and the second configuration information are obtained. The first configuration information includes a source data source and a data processing strategy. The source data source is the starting point of data collection and supports a variety of source data sources. The data processing strategy supports users to configure in the form of custom scripts, custom functions, and custom variables. The second configuration information includes a target data source, a storage format, a partitioning strategy, and a cleaning strategy. The target data source is the storage location of the final data. The specific storage target table in the target data source can be further configured. The storage format includes but is not limited to Parquet format, ORC format, JSON format, and CSV format. The partitioning strategy and the cleaning strategy also support configuration in the form of custom scripts, custom variables, and custom functions.

[0064] S2. Collect corresponding real-time data according to the source data source, combine the custom script with the custom variables and the custom functions, and process the real-time data by referencing the custom variables and the custom functions through placeholders in the custom script to obtain processed data;

[0065] In this embodiment, if Figure 2 As shown, the corresponding real-time data is collected according to the source data source in the first configuration information of step S1, and the custom script is combined with the custom variables and the custom functions. The real-time data is processed by referencing the custom variables and the custom functions through placeholders in the custom script, such as using: placeholder ${#custom variable}${##custom function} in the custom script to realize the processing of real-time data, thereby obtaining processed data.

[0066] In a specific embodiment, the specific scenario is: real-time conversion and calculation of real-time sales records, custom variable 1: $currencyConversionRates, stores the latest currency conversion rate, custom variable 2: $taxRates, adapts to tax rate changes in different countries, custom variable 3: $promotionalDiscounts, reflects the current promotional discount, custom function 1: convertCurrency(amount,currency), used to convert the sales amount into the company's standard reporting currency, custom function 2: applyTax(amount,country), used to automatically calculate the tax amount based on the country of purchase, custom function 3: calculateDiscountedPrice(price,discount), used to calculate the discounted price, therefore, for each sales record, custom function 1 is called to convert the sales amount into the company's standard reporting currency, then custom function 2 is used to calculate the tax amount, and finally custom function 3 is used to calculate any discounted value, the parameters of these custom functions are obtained and provided by the corresponding custom variables 1, 2 and 3 mentioned above.

[0067] At this time, the data processing strategy in step S2 includes a throughput strategy, a dirty data strategy, a read concurrency strategy and a query strategy. The real-time data is processed by referencing the custom variables and the custom functions by placeholders in the custom script, and the processed data obtained includes:

[0068] S21, in the custom script of the throughput strategy, the transmission rate of the real-time data is processed by referencing the corresponding custom throughput threshold variable and the custom throughput function by a placeholder to obtain preliminary processed data;

[0069] In this embodiment, if Figure 2 As shown, in the custom script of the throughput policy, the transmission rate of real-time data is processed by referencing the corresponding custom throughput threshold variable and the custom throughput function through the placeholder, that is, the transmission rate of real-time data is limited to prevent excessive burden on the system.

[0070] S22, marking the dirty data in the preliminary processed data by referencing the corresponding custom dirty data threshold variable and the custom dirty data function by placeholders in the custom script of the dirty data strategy, to obtain marked preliminary processed data;

[0071] In this embodiment, if Figure 2As shown, in the custom script of the dirty data strategy, the dirty data in the preliminary processed data is marked by referencing the corresponding custom dirty data threshold variable and the custom dirty data function through the placeholder, that is, the data exceeding the dirty data threshold is marked, and when the data is subsequently cleaned, the marked dirty data can be filtered or treated as abnormal data for alarm.

[0072] S23, reading the marked preliminary processed data by referencing the corresponding custom read concurrency threshold variable and the custom read concurrency function in the custom script of the read concurrency strategy through the placeholder to obtain the read data;

[0073] In this embodiment, if Figure 2 As shown, in the custom script of the read concurrency strategy, the marked preliminary processed data is read by referencing the corresponding custom read concurrency threshold variable and the custom read concurrency function through the placeholder, that is, the number of read concurrencies is limited to speed up the reading efficiency.

[0074] S24. In the custom script of the query strategy, the read data is queried by referencing corresponding custom query variables and custom query functions by placeholders to obtain processed data.

[0075] S3. Partition the processed data according to the partitioning strategy to obtain partitioned data, clean the partitioned data according to the cleaning strategy to obtain cleaned data, and store the cleaned data to the target data source according to the storage format.

[0076] In this embodiment, if Figure 2 As shown, data partitioning is combined with data cleaning, the processed data is partitioned according to the partitioning strategy, and then the partitioned data is cleaned according to the cleaning strategy, so that the cleaned data is stored in the target data source according to the storage format.

[0077] At this time, the processed data is partitioned according to the partition strategy in step S3 to obtain partitioned data including:

[0078] S31, obtaining a partition category from the partition strategy, performing a primary partition on the processed data according to the partition category to obtain preliminary partition data, obtaining a partition parameter, a partition size and a partition key from the partition category, performing a secondary partition on the preliminary partition data according to the partition parameter, the partition size and the partition key to obtain partitioned data;

[0079] The partition categories include timestamp, geographic location and user ID.

[0080] In this embodiment, if Figure 2As shown, the processed data is partitioned at multiple levels. The partition category is first obtained from the partition strategy. The partition category includes timestamp, geographic location and user ID. The processed data is first partitioned at the primary level according to the partition category to obtain preliminary partition data. Then, the preliminary partition data is partitioned at the secondary level according to specific partition parameters, partition size and partition key to obtain partitioned data.

[0081] At this time, the cleaning strategy in step S3 includes a missing value processing strategy, a data conversion strategy and a filtering strategy, and the partitioned data is cleaned according to the cleaning strategy to obtain the cleaned data including:

[0082] S32, filling the missing values ​​in the partitioned data using the mean, median or mode according to the missing value processing strategy to obtain filled data;

[0083] In this embodiment, if Figure 2 As shown, according to the missing value processing strategy, the missing values ​​in the partitioned data are filled using the mean, median, mode and preset value, which can be specifically set according to the actual needs of the user to obtain the filled data.

[0084] S33, performing standardization processing on the filled data according to the data conversion strategy to obtain standardized data, wherein the standardization processing includes standardizing the date, standardizing the timestamp format and standardizing the currency symbol;

[0085] At this time, the cleaning strategy in step S33 includes an exception handling strategy, and obtaining standardized data includes:

[0086] S331. Issue an alarm for abnormal values ​​in the standardized data according to an abnormality handling strategy.

[0087] In this embodiment, if Figure 2 As shown, the filled data is standardized according to the data conversion strategy, the date, timestamp, currency symbol and telephone number format are standardized to obtain standardized data, and an alarm is issued for abnormal values ​​in the standardized data according to the abnormal processing strategy.

[0088] S34, creating a temporary buffer, performing deduplication processing on the standardized data in the temporary buffer according to the filtering strategy, obtaining the deduplicated standardized data and releasing the temporary buffer at the same time, performing word segmentation and stem extraction on the deduplicated standardized data through Apache Lucene, and obtaining cleaned data.

[0089] In this embodiment, if Figure 2As shown, when the standardized data is processed according to the filtering strategy, a temporary buffer is created, and the standardized data is deduplicated in the temporary buffer. The temporary buffer is released after deduplication, and Apache Lucene is used to perform word segmentation and stem extraction on the deduplicated standardized data, and stop words are removed. The word segmentation and stop word removal can be combined with the word segmentation vocabulary and stop word vocabulary provided by the user to obtain cleaned data.

[0090] Embodiment 2

[0091] Please refer to Figure 3 The present invention provides a data processing system 1, comprising: an acquisition module 2, a processing module 3, a target module 4, a partitioning module 41 and a cleaning module 42.

[0092] Wherein, the acquisition module 2 is used to acquire first configuration information and second configuration information, wherein the first configuration information includes a source data source and a data processing strategy, and the second configuration information includes a target data source, a storage format, a partitioning strategy, and a cleaning strategy, and the data processing strategy supports custom scripts, custom variables, and custom functions;

[0093] Processing module 3, used for collecting corresponding real-time data according to the source data source, combining the custom script with the custom variables and the custom functions, and processing the real-time data by referencing the custom variables and the custom functions by placeholders in the custom script to obtain processed data;

[0094] The target module 4 is used to partition the processed data according to the partitioning strategy to obtain partitioned data, clean the partitioned data according to the cleaning strategy to obtain cleaned data, and store the cleaned data to the target data source according to the storage format.

[0095] Specifically, the data processing strategy includes a throughput strategy, a dirty data strategy, a read concurrency strategy and a query strategy. The processing module 3 is specifically:

[0096] In the custom script of the throughput strategy, the transmission rate of the real-time data is processed by referencing the corresponding custom throughput threshold variable and the custom throughput function by placeholders to obtain preliminary processing data;

[0097] In the custom script of the dirty data strategy, dirty data in the preliminary processed data is marked by referencing the corresponding custom dirty data threshold variable and the custom dirty data function by placeholders to obtain marked preliminary processed data;

[0098] In the custom script of the read concurrency strategy, the marked preliminary processing data is read by referencing the corresponding custom read concurrency threshold variable and the custom read concurrency function through the placeholder to obtain the read data;

[0099] In the custom script of the query strategy, the read data is queried by referring to the corresponding custom query variables and custom query functions through placeholders to obtain processed data.

[0100] Specifically, the target module 4 includes:

[0101] A partitioning module 41 is used to obtain a partition category from the partition strategy, perform primary partitioning on the processed data according to the partition category to obtain preliminary partition data, obtain a partition parameter, a partition size and a partition key from the partition category, perform secondary partitioning on the preliminary partition data according to the partition parameter, the partition size and the partition key to obtain partitioned data;

[0102] The partition categories include timestamp, geographic location and user ID.

[0103] Specifically, the cleaning strategy includes a missing value processing strategy, a data conversion strategy and a filtering strategy, and the target module 4 includes:

[0104] A cleaning module 42 is used to fill the missing values ​​in the partitioned data using the mean, median or mode according to the missing value processing strategy to obtain filled data;

[0105] Performing standardization processing on the filled data according to the data conversion strategy to obtain standardized data, wherein the standardization processing includes standardizing the date, standardizing the timestamp format and standardizing the currency symbol;

[0106] A temporary buffer is created, and the standardized data is deduplicated in the temporary buffer according to the filtering strategy to obtain the deduplicated standardized data and release the temporary buffer at the same time, and the deduplicated standardized data is segmented and stemmed by Apache Lucene to obtain cleaned data.

[0107] Specifically, the cleaning module is:

[0108] An alarm is issued for abnormal values ​​in the standardized data according to an abnormality handling strategy.

[0109] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0110] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0112] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0113] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0114] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0115] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A method for data processing, characterized in that: include: Obtaining first configuration information and second configuration information, wherein the first configuration information includes a source data source and a data processing strategy, and the second configuration information includes a target data source, a storage format, a partitioning strategy, and a cleaning strategy, and the data processing strategy supports custom scripts, custom variables, and custom functions; Collecting corresponding real-time data according to the source data source, combining the custom script with the custom variables and the custom functions, and processing the real-time data by referencing the custom variables and the custom functions by placeholders in the custom script to obtain processed data; The processed data is partitioned according to the partitioning strategy to obtain partitioned data, the partitioned data is cleaned according to the cleaning strategy to obtain cleaned data, and the cleaned data is stored in the target data source according to the storage format.

2. A data processing method according to claim 1, characterized in that: The data processing strategy includes a throughput strategy, a dirty data strategy, a read concurrency strategy and a query strategy. The real-time data is processed by referencing the custom variables and the custom functions by placeholders in the custom script, and the processed data obtained includes: In the custom script of the throughput strategy, the transmission rate of the real-time data is processed by referencing the corresponding custom throughput threshold variable and the custom throughput function by placeholders to obtain preliminary processing data; In the custom script of the dirty data strategy, dirty data in the preliminary processed data is marked by referencing the corresponding custom dirty data threshold variable and the custom dirty data function by placeholders to obtain marked preliminary processed data; In the custom script of the read concurrency strategy, the marked preliminary processing data is read by referencing the corresponding custom read concurrency threshold variable and the custom read concurrency function through the placeholder to obtain the read data; In the custom script of the query strategy, the read data is queried by referring to the corresponding custom query variables and custom query functions through placeholders to obtain processed data.

3. A data processing method according to claim 1, characterized in that: Partitioning the processed data according to the partition strategy to obtain partitioned data includes: Obtaining a partition category from the partition strategy, performing a primary partition on the processed data according to the partition category to obtain preliminary partition data, obtaining a partition parameter, a partition size and a partition key from the partition category, performing a secondary partition on the preliminary partition data according to the partition parameter, the partition size and the partition key to obtain partitioned data; The partition categories include timestamp, geographic location and user ID.

4. A data processing method according to claim 1, characterized in that: The cleaning strategy includes a missing value processing strategy, a data conversion strategy and a filtering strategy, and the partitioned data is cleaned according to the cleaning strategy to obtain the cleaned data including: Fill the missing values ​​in the partitioned data using the mean, median or mode according to the missing value processing strategy to obtain filled data; Performing standardization processing on the filled data according to the data conversion strategy to obtain standardized data, wherein the standardization processing includes standardizing the date, standardizing the timestamp format and standardizing the currency symbol; A temporary buffer is created, and the standardized data is deduplicated in the temporary buffer according to the filtering strategy to obtain the deduplicated standardized data and release the temporary buffer at the same time, and the deduplicated standardized data is segmented and stemmed by Apache Lucene to obtain cleaned data.

5. A data processing method as claimed in claim 4, characterized in that: The cleaning strategy includes an exception handling strategy, and obtaining standardized data includes: An alarm is issued for abnormal values ​​in the standardized data according to an abnormality handling strategy.

6. A data processing system, characterized in that: include: An acquisition module, used to acquire first configuration information and second configuration information, wherein the first configuration information includes a source data source and a data processing strategy, and the second configuration information includes a target data source, a storage format, a partition strategy, and a cleaning strategy, and the data processing strategy supports custom scripts, custom variables, and custom functions; a processing module, configured to collect corresponding real-time data according to the source data source, combine the custom script with the custom variables and the custom functions, and process the real-time data by referencing the custom variables and the custom functions by placeholders in the custom script to obtain processed data; The target module is used to partition the processed data according to the partitioning strategy to obtain partitioned data, clean the partitioned data according to the cleaning strategy to obtain cleaned data, and store the cleaned data to the target data source according to the storage format.

7. A data processing system as claimed in claim 6, characterized in that: The data processing strategy includes throughput strategy, dirty data strategy, read concurrency strategy and query strategy. The processing module is specifically: In the custom script of the throughput strategy, the transmission rate of the real-time data is processed by referencing the corresponding custom throughput threshold variable and the custom throughput function by placeholders to obtain preliminary processing data; In the custom script of the dirty data strategy, dirty data in the preliminary processed data is marked by referencing the corresponding custom dirty data threshold variable and the custom dirty data function by placeholders to obtain marked preliminary processed data; In the custom script of the read concurrency strategy, the marked preliminary processing data is read by referencing the corresponding custom read concurrency threshold variable and the custom read concurrency function through the placeholder to obtain the read data; In the custom script of the query strategy, the read data is queried by referring to the corresponding custom query variables and custom query functions through placeholders to obtain processed data.

8. A data processing system as claimed in claim 6, characterized in that: The target module includes: A partitioning module, used to obtain a partitioning category from the partitioning strategy, perform a primary partitioning on the processed data according to the partitioning category to obtain preliminary partitioning data, obtain a partitioning parameter, a partitioning size and a partitioning key from the partitioning category, perform a secondary partitioning on the preliminary partitioning data according to the partitioning parameter, the partitioning size and the partitioning key to obtain partitioned data; The partition categories include timestamp, geographic location and user ID.

9. A data processing system as claimed in claim 6, characterized in that: The cleaning strategy includes missing value processing strategy, data conversion strategy and filtering strategy, and the target module includes: A cleaning module, used for filling the missing values ​​in the partitioned data using the mean, median or mode according to the missing value processing strategy to obtain filled data; Performing standardization processing on the filled data according to the data conversion strategy to obtain standardized data, wherein the standardization processing includes standardizing the date, standardizing the timestamp format and standardizing the currency symbol; A temporary buffer is created, and the standardized data is deduplicated in the temporary buffer according to the filtering strategy to obtain the deduplicated standardized data and release the temporary buffer at the same time, and the deduplicated standardized data is segmented and stemmed by Apache Lucene to obtain cleaned data.

10. A data processing system as claimed in claim 9, characterized in that: The cleaning module is specifically: An alarm is issued for abnormal values ​​in the standardized data according to an abnormality handling strategy.

Images