Distributed system data processing method based on Hadoop technology
By adopting secure encryption and detailed data processing processes in the distributed system of Hadoop technology, the problems of leakage and repeated operations during data acquisition and processing are solved, and the secure transmission and efficient processing of data are realized, ensuring the accuracy of data and reliability.
Patent Information
- Application Number
- CN202410932284.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art has problems of data leakage and repeated operations in the data acquisition and processing process, especially in the distributed systems of Hadoop technology, where effective data encryption and processing process optimization are lacking.
Using secure network protocols and encryption algorithms to encrypt sensitive data, and perform pre-processing, division, task allocation, map and reduce specific operations during data collection and processing to ensure data security and processing efficiency, including Data cleaning, conversion, normalization and storage.
The security of data transmission and processing efficiency are realized, data leakage and repeated operations are avoided, data accuracy and reliability are ensured, calculation overhead for subsequent analysis is reduced, and the same type of data is realized by saving processing flow configuration files. Quick processing.
Smart Images

Figure CN119004524A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of distributed system data processing, and in particular to a distributed system data processing method based on Hadoop technology. Background Art
[0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without understanding the underlying details of the distribution, and make full use of the power of the cluster for high-speed computing and storage. Hadoop implements a distributed file system, one of which is HDFS. HDFS has the characteristics of high fault tolerance and is designed to be deployed on low-cost hardware. It also provides high throughput to access application data, making it suitable for applications with very large data sets. HDFS relaxes the requirements of POSIX and can access data in the file system in the form of streams.
[0003] In the existing technology, when collecting data, most of them first collect all the data, then select the required data, and then encrypt the required data. The screened data is not encrypted, which is easy to cause leakage. At the same time, the distributed system data processing method based on Hadoop technology is not saved. When processing the same type of data later, the same operation needs to be performed again, which is time-consuming. For this reason, we propose a distributed system data processing method based on Hadoop technology. Summary of the invention
[0004] The purpose of the present invention is to provide a distributed system data processing method based on Hadoop technology.
[0005] In order to solve the problems raised in the above background technology, the present invention provides the following technical solutions: a distributed system data processing method based on Hadoop technology, comprising the following steps:
[0006] Step 1: Data collection: Collect data from various data sources to ensure that it will not be illegally obtained or tampered with during transmission. Sensitive data must be encrypted and imported into the Hadoop distributed file system.
[0007] Step 2: Data preprocessing: Clean, convert, and normalize data in a distributed environment to ensure data quality and availability.
[0008] Step 3: Data partitioning: Divide the collected large-scale data into multiple data blocks for parallel processing on distributed nodes;
[0009] Step 4: Task allocation: Allocate data processing tasks to each node through Hadoop’s task scheduling mechanism;
[0010] Step 5: Map phase: the original data is divided into several key-value pairs, and then processed by the Map function to generate intermediate results. The Map function can process each key-value pair and output a new key-value pair.
[0011] Step 6: In the Shuffle phase, the output of the Map function is sorted and grouped by key so that data with the same key can be sent to the same Reduce node;
[0012] Step 7: Reduce phase: execute the user-defined Reduce function on each node to summarize and aggregate the shuffled data to obtain the final processing result;
[0013] Step 8: Result storage: store the processed results back to HDFS or other storage systems for subsequent analysis and use, and store the entire processing flow for subsequent calls.
[0014] As a further solution of the present invention: in the step one, for different types of data sources, corresponding appropriate collection methods should be adopted. During the data collection process, secure network protocols are used, virtual private networks are established, and access control is implemented to ensure the security of transmission. A symmetric encryption algorithm is used to encrypt sensitive data, and only the recipient with the correct key can decrypt it. The key needs to be securely stored and updated regularly. The data is digitally signed before transmission, and the recipient can verify the source and integrity of the data to prevent the data from being tampered with. At the same time, the format conversion of the data needs to be considered, because the original data may exist in multiple formats. In order to facilitate subsequent processing and analysis, it may be necessary to convert it into a unified and easy-to-process format. At the same time, the real-time and integrity of data collection must be ensured. For data with high real-time requirements, appropriate technologies and architectures need to be adopted to ensure that the data can be collected and processed in a timely manner. For integrity, a data verification mechanism must be established to ensure that the collected data is not lost or damaged.
[0015] As a further solution of the present invention: in the step 2, data cleaning can be used to check whether there are missing values in the data, and different processing methods can be used to reasonably fill them. Data conversion can uniformly convert data in different formats into a format suitable for subsequent processing. Data normalization can ensure that the data is within a reasonable range. Data screening and filtering can remove irrelevant or non-compliant data according to specific conditions or rules. Statistical analysis is performed on the data to understand the basic characteristics and distribution of the data, which is helpful for the subsequent selection of appropriate processing methods and models.
[0016] As a further solution of the present invention: in step three, in the data partitioning stage, a suitable partitioning strategy is determined based on the characteristics of the data, the distribution of computing resources and the needs of subsequent processing. When partitioning, the correlation and locality of the data should be considered, and related data should be partitioned into the same or adjacent data blocks as much as possible to reduce data movement and communication overhead in subsequent processing. After the partitioning is completed, the data blocks need to be marked and managed so that each data block can be accurately identified and operated in subsequent task allocation and processing.
[0017] As a further solution of the present invention: In the step four, first of all, tasks must be reasonably divided and arranged according to the computing resource status, clarifying which tasks can be executed in parallel and which tasks have dependencies and need to be performed sequentially, and establishing an effective task monitoring mechanism to understand the execution status, progress and resource usage of each task in real time. If a task fails or an abnormal situation occurs, it can be processed and reallocated in a timely manner.
[0018] As a further solution of the present invention: in step five, at the beginning of the Map stage, the system will divide the input data into multiple independent small data blocks and distribute them to different processing nodes to execute the Map function. During the processing, the Map task will generate a large number of intermediate key-value pairs and cache them locally.
[0019] As a further solution of the present invention: in step six, a large number of intermediate key-value pairs generated in the Map stage are reorganized and distributed, and data related to the same key originally scattered on various nodes are integrated together to prepare for the subsequent Reduce stage.
[0020] As a further solution of the present invention: in step seven, the Reduce function is applied to all values related to the specific key after shuffling, and is processed according to a predetermined logic to output a more concise and meaningful data result.
[0021] As a further solution of the present invention: in the step eight, a suitable storage medium and storage method are selected. First, some data verification and validation operations are performed to ensure the integrity and accuracy of the data to prevent data damage or loss during the storage process. A large amount of data results are compressed to save storage space and improve the efficiency of storage and reading. Appropriate access rights and encryption measures are set to protect the data from unauthorized access or tampering. Corresponding data indexes or metadata information are established to quickly locate and retrieve result data. The entire processing flow, including the data reading method, the specific functions and logic of each stage, is saved in the form of code or configuration files. When encountering the same type of data processing in the future, these stored methods and configurations are directly loaded and processed according to the established process.
[0022] By adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are:
[0023] 1. The present invention ensures the security of transmission by using secure network protocols, establishing a virtual private network and implementing access control. It uses a symmetric encryption algorithm to encrypt sensitive data. Only the recipient with the correct key can decrypt it. The key needs to be securely stored and updated regularly. The data is digitally signed before transmission. The recipient can verify the source and integrity of the data, prevent data from being tampered with, and avoid data leakage during data collection;
[0024] 2. The present invention makes data more accurate and reliable through data preprocessing, provides a solid foundation for subsequent analysis and model building, unifies data formats and coding, reduces data inconsistency and ambiguity, ensures that data from different parts can be better integrated and understood, removes outliers and unreasonable data, and avoids their misleading impact on the overall analysis results. After proper conversion and organization, data is easier to understand and interpret, which is convenient for analysts to work, reduces unnecessary computing overhead and time waste in subsequent processing, and avoids repeated debugging and corrections caused by data problems, thereby speeding up the progress of overall analysis;
[0025] 3. The present invention saves the entire processing flow, including the data reading method, the specific functions and logic of each stage, in the form of code or configuration files. When encountering the same type of data processing in the future, these stored methods and configurations are directly loaded and processed according to the established process, avoiding repeated development and debugging of the same processing logic, saving time and energy, and ensuring that the same specifications and methods are followed each time the same type of data is processed, and the results are comparable and consistent. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 Schematic diagram of the method steps in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The specific embodiments of the present invention will be further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.
[0028] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0029] Please see attached Figure 1 The present invention provides a distributed system data processing method based on Hadoop technology, comprising the following steps:
[0030] Step 1: Data collection: Collect data from various data sources to ensure that it will not be illegally obtained or tampered with during transmission. Sensitive data must be encrypted and imported into the Hadoop distributed file system.
[0031] Step 2: Data preprocessing: Clean, convert, and normalize data in a distributed environment to ensure data quality and availability.
[0032] Step 3: Data partitioning: Divide the collected large-scale data into multiple data blocks for parallel processing on distributed nodes;
[0033] Step 4: Task allocation: Allocate data processing tasks to each node through Hadoop’s task scheduling mechanism;
[0034] Step 5: Map phase: the original data is divided into several key-value pairs, and then processed by the Map function to generate intermediate results. The Map function can process each key-value pair and output a new key-value pair.
[0035] Step 6: In the Shuffle phase, the output of the Map function is sorted and grouped by key so that data with the same key can be sent to the same Reduce node;
[0036] Step 7: Reduce phase: execute the user-defined Reduce function on each node to summarize and aggregate the shuffled data to obtain the final processing result;
[0037] Step 8: Result storage: store the processed results back to HDFS or other storage systems for subsequent analysis and use, and store the entire processing flow for subsequent calls.
[0038] In one embodiment of the present invention: In step one, for different types of data sources, corresponding appropriate collection methods should be adopted. During the data collection process, a secure network protocol is used, a virtual private network is established, and access control is implemented to ensure the security of transmission. A symmetric encryption algorithm is used to encrypt sensitive data, and only the recipient with the correct key can decrypt it. The key needs to be securely stored and updated regularly. The data is digitally signed before transmission, and the recipient can verify the source and integrity of the data to prevent the data from being tampered with. At the same time, the format conversion of the data needs to be considered, because the original data may exist in multiple formats. In order to facilitate subsequent processing and analysis, it may be necessary to convert it into a unified and easy-to-process format. At the same time, the real-time and integrity of data collection must be ensured. For data with high real-time requirements, appropriate technologies and architectures need to be adopted to ensure that the data can be collected and processed in a timely manner. For integrity, a data verification mechanism must be established to ensure that the collected data is not lost or damaged.
[0039] In one embodiment of the present invention: In step 2, data cleaning can be used to check whether there are missing values in the data, and different processing methods can be used to reasonably fill them. Data conversion can uniformly convert data in different formats into a format suitable for subsequent processing. Data normalization can ensure that the data is within a reasonable range. Data screening and filtering can remove irrelevant or non-compliant data according to specific conditions or rules. Statistical analysis is performed on the data to understand the basic characteristics and distribution of the data, which is helpful for the subsequent selection of appropriate processing methods and models.
[0040] In one embodiment of the present invention: In step three, in the data partitioning stage, a suitable partitioning strategy is determined based on the characteristics of the data, the distribution of computing resources, and the needs of subsequent processing. When partitioning, the correlation and locality of the data should be considered, and related data should be partitioned into the same or adjacent data blocks as much as possible to reduce data movement and communication overhead in subsequent processing. After the partitioning is completed, the data blocks need to be marked and managed so that each data block can be accurately identified and operated in subsequent task allocation and processing.
[0041] In one embodiment of the present invention: In step four, first, tasks must be reasonably divided and arranged according to the computing resource status, clarifying which tasks can be executed in parallel and which tasks have dependencies and need to be performed sequentially, and establishing an effective task monitoring mechanism to understand the execution status, progress and resource usage of each task in real time. If a task fails or an abnormal situation occurs, it can be processed and reallocated in a timely manner.
[0042] In one embodiment of the present invention: In step five, at the beginning of the Map phase, the system will divide the input data into multiple independent small data blocks and distribute them to different processing nodes to execute the Map function. During the processing, the Map task will generate a large number of intermediate key-value pairs and cache them locally.
[0043] In one embodiment of the present invention: In step six, a large number of intermediate key-value pairs generated in the Map phase are reorganized and distributed, and data related to the same key originally scattered on various nodes are integrated together to prepare for the subsequent Reduce phase.
[0044] In one embodiment of the present invention: In step seven, the Reduce function is applied to all values associated with a specific key after shuffling, and is processed according to a predetermined logic to output a more concise and meaningful data result.
[0045] In one embodiment of the present invention: In step eight, a suitable storage medium and storage method are selected. First, some data verification and validation operations are performed to ensure the integrity and accuracy of the data to prevent data damage or loss during the storage process. A large amount of data results are compressed to save storage space and improve storage and reading efficiency. Appropriate access rights and encryption measures are set to protect data from unauthorized access or tampering. Corresponding data indexes or metadata information are established to quickly locate and retrieve result data. The entire processing flow, including the data reading method, the specific functions and logic of each stage, is saved in the form of code or configuration files. When encountering the same type of data processing in the future, these stored methods and configurations are directly loaded and processed according to the established process.
[0046] Example 1, please refer to the attached Figure 1 , determine the required encryption strength based on the sensitivity and security requirements of the data. When selecting an encryption method, you need to consider its consumption of data processing speed and system resources to ensure that the encryption process does not have too much impact on the business. When selecting an encryption method, you need to ensure that it complies with relevant regulations and standards. Choose an encryption method that is easy to implement, configure, and maintain to reduce system complexity and management costs. Built-in encryption modules in devices or collection programs ensure that data is encrypted at the source. Combined with the identity authentication mechanism, only authorized collection sources can use the correct key for encryption. Update encryption keys regularly to enhance security. Monitor and audit the encryption process to ensure the normal execution and security of encryption operations.
[0047] Example 2, please refer to the attached Figure 1For missing values, different processing methods can be used, such as filling them with default values, means, medians, or making reasonable guesses through other relevant data. Duplicate data records should also be removed to ensure the uniqueness and accuracy of the data.
[0048] Example 3, please refer to the attached Figure 1 If the processing flow needs to be modified or optimized, it is only necessary to make adjustments in the stored method without modifying each specific processing task separately. In actual operation, the relevant code can be packaged into a library or module, or the configuration file can be saved in a specific location. When it is needed, the stored content can be imported or loaded in an appropriate manner, and then the data processing flow can be started. In this way, the efficient reuse of existing mature processing methods can be achieved, and the efficiency and quality of data processing work can be improved.
[0049] Specifically, the security of transmission is guaranteed by using secure network protocols, establishing virtual private networks and implementing access control. Sensitive data is encrypted using symmetric encryption algorithms, and only the recipient with the correct key can decrypt it. The key needs to be securely stored and updated regularly. The data is digitally signed before transmission. The recipient can verify the source and integrity of the data, prevent data tampering, and avoid data leakage during data collection.
[0050] Specifically, through data preprocessing, the data can be made more accurate and reliable, providing a solid foundation for subsequent analysis and model building, unifying the data format and coding, reducing data inconsistency and ambiguity, ensuring that data from different parts can be better integrated and understood, eliminating outliers and unreasonable data to avoid their misleading impact on the overall analysis results. After proper conversion and organization, the data is easier to understand and interpret, facilitating the work of analysts, reducing unnecessary computing overhead and time waste in subsequent processing, and avoiding repeated debugging and corrections due to data problems, thereby speeding up the progress of the overall analysis.
[0051] Specifically, by saving the entire processing flow, including the data reading method, specific functions and logic of each stage, in the form of code or configuration files, when encountering the same type of data processing in the future, these stored methods and configurations are directly loaded and processed according to the established process, avoiding repeated development and debugging of the same processing logic, saving time and energy, and ensuring that each time the same type of data is processed, the same specifications and methods are followed, and the results are comparable and consistent.
[0052] Working principle:
[0053] First, data is collected from various data sources. For different types of data sources, appropriate collection methods should be used to ensure the security of transmission, use secure network protocols, establish virtual private networks and implement access control, and use symmetric encryption algorithms to encrypt sensitive data. Then, pre-processing operations such as cleaning, conversion, and normalization are performed on the data in a distributed environment to ensure data quality and availability. Large-scale data is divided into multiple data blocks for parallel processing on distributed nodes. Through Hadoop's task scheduling mechanism, data processing tasks are assigned to each node. The original data is divided into several key-value pairs, and then processed by the Map function to generate intermediate results. The Map function The map function can process each key-value pair and output a new key-value pair. The output results of the Map function will be sorted and grouped by key so that data with the same key can be sent to the same Reduce node. The Reduce function will process each group of data with the same key to generate the final result, and store the processed results back to HDFS or other storage systems for subsequent analysis and use. The entire processing flow, including the data reading method, the specific functions and logic of each stage, is saved in the form of code or configuration files. When encountering the same type of data processing in the future, these stored methods and configurations are directly loaded and processed according to the established process. At this point, the entire workflow ends.
[0054] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.
[0055] Some aspects of the present application can be completely executed by hardware, can be completely executed by software (including firmware, resident software, microcode, etc.), or can be executed by a combination of hardware and software. The above hardware or software can all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". The processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. In addition, various aspects of the present application may be expressed as computer products located in one or more computer-readable media, which include computer-readable program codes. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, tapes...), optical disks (e.g., compact disks CDs, digital versatile disks DVDs...), smart cards, and flash memory devices (e.g., cards, sticks, key drives...).
[0056] A computer-readable medium may include a propagated data signal containing computer program code, such as in baseband or as part of a carrier wave. The propagated signal may have a variety of manifestations, including electromagnetic, optical, etc., or a suitable combination. A computer-readable medium may be any computer-readable medium other than a computer-readable storage medium, which may be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit a program for use. The program code on the computer-readable medium may be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or a combination of any of the above mediums.
[0057] Similarly, it should be noted that in order to simplify the description disclosed in this application, so as to help understand one or more embodiments of the invention, in the above description of the embodiments of the present application, sometimes multiple features are merged into one embodiment, drawings or descriptions thereof. However, this disclosure method does not mean that the features required by the object of the present application are more than the features mentioned in the claims. In fact, the features of the embodiment are less than all the features of the single embodiment disclosed above. In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "approximately", "approximately" or "substantially" in some examples. Unless otherwise specified, "approximately", "approximately" or "substantially" indicate that the number allows a variation of ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the characteristics required by individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining the digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of the present application are approximate values, in specific embodiments, such numerical settings are as accurate as possible within the feasible range.
[0058] Although the present invention is disclosed as above in terms of preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection defined by the claims of the present invention.
Claims
1. A distributed system data processing method based on Hadoop technology, Features: The following steps are involved: Step 1: Data collection: Collect data from various data sources to ensure that it will not be illegally obtained or tampered with during transmission. Sensitive data must be encrypted and imported into the Hadoop distributed file system. Step 2: Data preprocessing: Clean, convert, and normalize data in a distributed environment to ensure data quality and availability. Step 3: Data partitioning: Divide the collected large-scale data into multiple data blocks for parallel processing on distributed nodes; Step 4: Task allocation: Allocate data processing tasks to each node through Hadoop’s task scheduling mechanism; Step 5: Map phase: the original data is divided into several key-value pairs, and then processed by the Map function to generate intermediate results. The Map function can process each key-value pair and output a new key-value pair. Step 6: In the Shuffle phase, the output of the Map function is sorted and grouped by key so that data with the same key can be sent to the same Reduce node; Step 7: Reduce phase: execute the user-defined Reduce function on each node to summarize and aggregate the shuffled data to obtain the final processing result; Step 8: Result storage: store the processed results back to HDFS or other storage systems for subsequent analysis and use, and store the entire processing flow for subsequent calls.
2. A distributed system data processing method based on Hadoop technology according to claim 1, Features: In the step 1, for different types of data sources, corresponding appropriate collection methods should be adopted. During the data collection process, secure network protocols should be used, virtual private networks should be established, and access control should be implemented to ensure the security of transmission. A symmetric encryption algorithm should be used to encrypt sensitive data, and only the recipient with the correct key can decrypt it. The key needs to be securely stored and updated regularly. The data should be digitally signed before transmission so that the recipient can verify the source and integrity of the data and prevent the data from being tampered with. At the same time, data format conversion should also be considered, because the original data may exist in multiple formats. In order to facilitate subsequent processing and analysis, it may be necessary to convert it into a unified, easy-to-process format. At the same time, the real-time and integrity of data collection should be ensured. For data with high real-time requirements, appropriate technologies and architectures need to be adopted to ensure that the data can be collected and processed in a timely manner. For integrity, a data verification mechanism should be established to ensure that the collected data is not lost or damaged.
3. The distributed system data processing method based on Hadoop technology according to claim 1, Features: In the step 2, data cleaning can be used to check whether there are missing values in the data, and different processing methods can be used to fill them reasonably. Data conversion can convert data in different formats into a format suitable for subsequent processing. Data normalization can ensure that the data is within a reasonable range. Data screening and filtering can remove irrelevant or non-compliant data according to specific conditions or rules. Statistical analysis of the data is performed to understand the basic characteristics and distribution of the data, which is helpful for the subsequent selection of appropriate processing methods and models.
4. The distributed system data processing method based on Hadoop technology according to claim 1, Features: In the step three, in the data partitioning stage, the appropriate partitioning strategy is determined based on the characteristics of the data, the distribution of computing resources, and the needs of subsequent processing. When partitioning, the correlation and locality of the data should be considered, and related data should be partitioned into the same or adjacent data blocks as much as possible to reduce the movement and communication overhead of data in subsequent processing. After the partitioning is completed, the data blocks need to be marked and managed so that each data block can be accurately identified and operated in subsequent task allocation and processing.
5. The distributed system data processing method based on Hadoop technology according to claim 1, Features: In step 4, we must first reasonably divide and arrange tasks according to the computing resource status, clarify which tasks can be executed in parallel and which tasks have dependencies and need to be performed sequentially, and establish an effective task monitoring mechanism to understand the execution status, progress and resource usage of each task in real time. If a task fails or an abnormal situation occurs, it can be processed and reallocated in a timely manner.
6. The distributed system data processing method based on Hadoop technology according to claim 1, Features: In step 5, at the beginning of the Map phase, the system will divide the input data into multiple independent small data blocks and distribute them to different processing nodes to execute the Map function. During the processing, the Map task will generate a large number of intermediate key-value pairs and cache them locally.
7. The distributed system data processing method based on Hadoop technology according to claim 1, Features: In the step six, a large number of intermediate key-value pairs generated in the Map phase are reorganized and distributed, and the data related to the same key originally scattered on various nodes are integrated together to prepare for the subsequent Reduce phase.
8. The distributed system data processing method based on Hadoop technology according to claim 1, Features: In step seven, the Reduce function is applied to all values associated with a specific key after shuffling, and is processed according to a predetermined logic to output a more concise and meaningful data result.
9. The distributed system data processing method based on Hadoop technology according to claim 1, Features: In step eight, a suitable storage medium and storage method are selected. First, some data verification and validation operations are performed to ensure the integrity and accuracy of the data to prevent data damage or loss during storage. A large amount of data results are compressed to save storage space and improve storage and reading efficiency. Appropriate access rights and encryption measures are set to protect data from unauthorized access or tampering. Corresponding data indexes or metadata information are established to quickly locate and retrieve result data. The entire processing flow, including the data reading method, specific functions and logic of each stage, is saved in the form of code or configuration files. When encountering the same type of data processing in the future, these stored methods and configurations are directly loaded and processed according to the established process.
