Big data processing method implemented based on spark

Abstract

The invention discloses a big data processing method based on spark, and the method comprises the following steps: S1, database authentication information configuration: configuring database authentication information, and storing the database authentication information in a background through employing an encryption algorithm; s2, data cleaning operation setting, wherein cleaning comprises value replacement, blank character processing and abnormal data format conversion, data are read into a memory in batches, and data cleaning is carried out; s3, executing the operation: executing the operation regularly and circularly; and S4, log monitoring: logging in real time in the job execution process, monitoring the job execution progress, and positioning the job execution problem. According to the big data processing method implemented on the basis of the spark, data stored in different databases can be extracted, cleaned and exported, the problem of data islands is solved, and the big data processing method implemented on the basis of the spark is suitable for data warehouse construction scenes on the basis of the big data.

Description

Technical Field

[0001] The present invention relates to the field of big data processing methods, and in particular to a big data processing method based on Spark. Background Art

[0002] There are numerous new methods for processing big data, all with the core goal of building higher-quality data, enabling data interoperability, and creating value from data. This process generally involves the following steps: data extraction, data conversion, and data export. Data extraction refers to extracting the required data from various sources, such as common databases, text data, data files in a specified format, and data streams. Data conversion involves processing collected data according to a standard, aiming to improve data quality and facilitate analysis. Data export involves exporting the processed data to target software for final analysis and visualization.

[0003] However, current big data processing methods have certain performance issues and cannot meet users' real-time requirements. Traditional data processing is mostly done offline, with little real-time processing. This approach is inefficient and cannot meet current users' requirements for accuracy and efficiency. It is not suitable for data scenarios with large data volumes. Summary of the Invention

[0004] The main purpose of the present invention is to provide a big data processing method based on Spark, which can effectively solve the problems in the background technology.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for processing big data based on Spark is described as follows:

[0007] S1: Database authentication information configuration: Configure database authentication information, and use encryption algorithm to store it in the background;

[0008] S2: Data cleaning job settings: Cleaning includes: value replacement, blank character processing, abnormal data

[0009] Format conversion reads data into memory in batches for data cleaning;

[0010] S3: Execute jobs: jobs can be executed in a scheduled or cyclic manner;

[0011] S4: Log monitoring: Real-time log of the job execution process, monitoring the job execution progress, and locating job execution problems;

[0012] Preferably, in step S1, the database authentication information is configured as follows: the authentication information is encrypted using a dynamically generated static key, where the static key refers to the key encrypted and stored after the user enters the database username and password, and the authentication is obtained by decrypting the key during authentication.

[0013] Preferably, in step S2, the in-memory computing feature of Spark is utilized to solve the problem of low efficiency in the ETL process, and the data is read into the memory in batches for cleaning and then imported to achieve the purpose of data intercommunication between different data sources and data cleaning and aggregation.

[0014] Preferably, in step S3, the configuration jobs are: source database, data cleaning, target database, and different execution strategies such as cyclic execution and timed execution are provided according to the usage standard of the data warehouse T+N.

[0015] Preferably, in step S4, real-time feedback is provided on the job execution progress and resource consumption to ensure transparency of the job progress.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] This big data processing method based on spark can ensure that data stored in different databases can be extracted, cleaned and exported, solving the problem of data islands. It is suitable for data warehouse construction scenarios based on big data, wherein the authentication information is encrypted using a dynamically generated static key. The static key refers to the encrypted storage of the key after the user enters the database username and password, and the authentication is obtained by decrypting the key during authentication. The in-memory computing feature of spark is used to solve the problem of low efficiency in the ETL process. The data is read into the memory in batches for cleaning and then imported to achieve the purpose of data intercommunication between different data sources and data cleaning and aggregation. The in-memory computing feature of spark is used to solve the problem of low efficiency in the ETL process. The data is read into the memory in batches for cleaning and then imported to achieve the purpose of data intercommunication between different data sources and data cleaning and aggregation. The progress of job execution and resource consumption are fed back in real time to ensure the transparency of job progress. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a big data processing method based on Spark implemented in the present invention; DETAILED DESCRIPTION

[0019] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0020] like Figure 1As shown, a big data processing method based on Spark is implemented. The steps of the big data processing method based on Spark are as follows:

[0021] S1: Database authentication information configuration: Configure database authentication information, and use encryption algorithm to store it in the background;

[0022] S2: Data cleaning job settings: Cleaning includes: value replacement, blank character processing, abnormal data

[0023] Format conversion reads data into memory in batches for data cleaning;

[0024] S3: Execute jobs: jobs can be executed in a scheduled or cyclic manner;

[0025] S4: Log monitoring: Real-time log of the job execution process, monitoring the job execution progress, and locating job execution problems.

[0026] In step S1, the database authentication information is configured: the authentication information is encrypted using a dynamically generated static key. The static key refers to the encrypted storage of the key after the user enters the database username and password, and the authentication is obtained by decrypting the key during authentication; in step S2, the in-memory computing characteristics of Spark are used to solve the problem of low efficiency in the ETL process. By reading the data into the memory in batches, cleaning it, and then importing it, the purpose of data interoperability and data cleaning and aggregation between different data sources is achieved; in step S3, the in-memory computing characteristics of Spark are used to solve the problem of low efficiency in the ETL process. By reading the data into the memory in batches, cleaning it, and then importing it, the purpose of data interoperability and data cleaning and aggregation between different data sources is achieved; in step S4, real-time feedback is provided on the job execution progress and resource consumption to ensure the transparency of the job progress.

[0027] It should be noted that the big data processing method based on spark described in the present invention can ensure that data stored in different databases are extracted, cleaned and exported, solve the problem of data islands, and is suitable for data warehouse construction scenarios based on big data, wherein the authentication information is encrypted using a dynamically generated static key. The static key refers to the encrypted storage of the key after the user enters the database username and password, and the authentication is obtained by decrypting the key during authentication. The characteristics of spark based on memory computing are used to solve the problem of low efficiency in the ETL process. The data is read into the memory in batches for cleaning and then imported to achieve the purpose of data intercommunication between different data sources and data cleaning and aggregation. The characteristics of spark based on memory computing are used to solve the problem of low efficiency in the ETL process. The data is read into the memory in batches for cleaning and then imported to achieve the purpose of data intercommunication between different data sources and data cleaning and aggregation. Real-time feedback is provided on the progress of job execution and resource consumption to ensure the transparency of job progress.

[0028] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A big data processing method based on Spark, characterized by: The steps of the big data processing method based on Spark are as follows: S1: Database authentication information configuration: Configure database authentication information, and use encryption algorithm to store it in the background; S2: Data cleaning job settings: Cleaning includes: value replacement, blank character processing, abnormal data format conversion, reading data into memory in batches, and performing data cleaning; S3: Execute jobs: jobs can be executed in a scheduled or cyclic manner; S4: Log monitoring: Real-time log of the job execution process, monitoring the job execution progress, and locating job execution problems.

2. The method for processing big data based on Spark according to claim 1, wherein: In step S1, the database authentication information is configured: the authentication information is encrypted using a dynamically generated static key. The static key refers to the key encrypted and stored after the user enters the database username and password, and the authentication is obtained by decrypting the key during authentication.

3. The method for processing big data based on Spark according to claim 1, wherein: In step S2, the in-memory computing feature of Spark is utilized to solve the problem of low efficiency in the ETL process. By reading data into memory in batches, cleaning it, and then importing it, the purpose of data intercommunication between different data sources and data cleaning and aggregation is achieved.

4. The method for processing big data based on Spark according to claim 1, wherein: In step S3, the jobs are configured: source database, data cleaning, and target database, and different execution strategies such as cyclic execution and timed execution are provided according to the T+N usage standard of the data warehouse.

5. A method for processing big data based on Spark according to claim 4 Its characteristics are: In step S4, real-time feedback is provided on the progress of job execution and resource consumption to ensure transparency of job progress.

Images