Data processing method, system, electronic device and computer readable storage medium
By converting the pending data in the data processing system into column storage file format, reducing the encryption and decryption operations, and using HttpFS for data transmission, the resource consumption and complexity problems caused by encrypted storage and decryption of text format files in distributed systems are solved. , realizes efficient data transmission.
Patent Information
- Application Number
- CN202210759076.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-06-29
AI Technical Summary
In a data processing system with a distributed system architecture, the encryption storage and decryption operations of text format files lead to high consumption of network and disk resources and the data transmission process is complex.
The data processing method is adopted to convert the to-processed data into column storage file format, reduce encryption and decryption operations, and data transmission is carried out through HttpFS, directly interacting between multi-level nodes, and reducing intermediate server cache.
It improves data transmission efficiency, reduces the occupation of network and storage resources, and simplifies the data interaction process.
Smart Images

Figure CN115378958B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to data processing methods, systems, electronic devices, and computer-readable storage media. Background Art
[0002] In distributed system architecture data processing systems, multiple nodes typically work together to complete data processing. The relevant data platform analyzes and stores the data generated by the upstream system. Because the data platform has numerous parsing strategies and metrics for different types of data to be processed, the upstream system needs to provide large amounts of complete offline data to the data platform.
[0003] In the relevant data transmission scheme, the node used to store the data needs to encrypt the text file. In the subsequent data exchange process, the next node needs to interact with SFTP multiple times to obtain the encrypted data, which results in a large consumption of network and disk resources and a complicated process. Summary of the Invention
[0004] This application provides data processing methods, systems, electronic devices, and computer-readable storage media that can improve data transmission efficiency.
[0005] To address the aforementioned technical problems, this application provides a data processing method. This method is applied to a data processing system comprising multiple hierarchical nodes. The method includes: if the current hierarchical node is a first-level node, then the current hierarchical node acquires data to be processed; if the current hierarchical node is not a first-level node, then the current hierarchical node acquires data to be processed sent by the previous hierarchical node; the current hierarchical node filters the data to be processed and converts the file format of the filtered data to a columnar storage file format for storage; the current hierarchical node sends a data capture notification to the next hierarchical node; in response to the data capture instruction from the next hierarchical node, the current hierarchical node sends the data to be processed, converted to a columnar storage file format, to the next hierarchical node; wherein the data capture instruction is generated based on the data capture notification.
[0006] The process involves the current-level node filtering the data to be processed and converting the file format of the filtered data to a columnar storage file format for storage. This includes: the current-level node filtering the data to be processed; converting the file format of the filtered data to a columnar storage file format; obtaining the compression format; and compressing and storing the data to be processed in the converted columnar storage file format according to the compression format.
[0007] If the current level node is not the first level node, the method further includes: the current level node responds to the data fetching notification and obtains the file list of filtered data to be processed from the previous level node; the current level node uses multi-threading to fetch the filtered data to be processed based on the file list.
[0008] Specifically, the current-level node will convert the data to be processed into columnar storage file format and send it to the next-level node. This includes: the current-level node storing the data to be processed into columnar storage file format in the embedded database; the current-level node traversing the data to be processed into columnar storage file format in the embedded database and adding the data to be processed into the message queue; and the current-level node distributing the data to be processed into columnar storage file format to the next-level node through the message queue.
[0009] Specifically, the current-level node will convert the data to be processed into a columnar storage file format and store it in the embedded database. This includes: the current-level node parsing the data to be processed into a columnar storage file format; and the current-level node converting the parsed data to be processed into an object spectrum file format and storing it in the embedded database.
[0010] Specifically, the current-level node will convert the data to be processed into columnar storage file format and store it in the embedded database. This includes: the current-level node will clean the data to be processed into columnar storage file format; and the current-level node will store the cleaned data to be processed in the embedded database.
[0011] If the current level node is a first-level node, the current level node obtains the data to be processed, including: the current level node uses a data synchronization tool to obtain the data to be processed from multiple databases.
[0012] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a data processing system, characterized in that the data processing system includes multiple hierarchical nodes; each hierarchical node is used to implement the method provided by any of the above technical solutions.
[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide an electronic device, which includes a processor and a memory coupled to the processor. The memory stores a computer program, and the processor executes the computer program to implement the method provided by any of the above technical solutions.
[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium that stores a computer program, which, when executed by a processor, implements the method provided by any of the above technical solutions.
[0015] The beneficial effects of the embodiments of this application are as follows: Unlike the prior art, the data processing method, system, electronic device, and computer-readable storage medium provided in this application are applied to a data processing system. The data processing system includes multiple hierarchical nodes. The method includes: if the current hierarchical node is a first-level node, then the current hierarchical node acquires data to be processed; if the current hierarchical node is not a first-level node, then the current hierarchical node acquires data to be processed sent by the previous hierarchical node; the current hierarchical node filters the data to be processed and converts the file format of the filtered data to be processed into a columnar storage file format for storage; the current hierarchical node sends a data capture notification to the next hierarchical node; in response to the data capture instruction from the next hierarchical node, the current hierarchical node sends the data to be processed, converted to a columnar storage file format, to the next hierarchical node; wherein, the data capture instruction is generated based on the data capture notification. Compared to the cumbersome plaintext encryption and decryption operations in related technologies during data interaction, the current-level node directly converts the filtered data to be processed into a columnar storage file format for storage, eliminating the need for cumbersome encryption and decryption operations. It then directly transmits data with the next-level node. Compared to the use of intermediate servers for data caching in related technologies, this approach enables the current-level node to interact directly with the next-level node, reducing data caching on intermediate servers and improving data transmission efficiency. Attached Figure Description
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:
[0017] Figure 1 This is a flowchart illustrating the first embodiment of the data processing method provided in this application;
[0018] Figure 2 This is a flowchart illustrating the second embodiment of the data processing method provided in this application;
[0019] Figure 3 This is a schematic diagram illustrating an application scenario of the data processing method provided in this application;
[0020] Figure 4 This is a flowchart illustrating the third embodiment of the data processing method provided in this application;
[0021] Figure 5 This is a flowchart illustrating the third embodiment of the data processing method provided in this application;
[0022] Figure 6This is a flowchart illustrating the fourth embodiment of the data processing method provided in this application;
[0023] Figure 7 This is a flowchart illustrating an embodiment of step 65 provided in this application;
[0024] Figure 8 This is a schematic diagram of the structure of an embodiment of the data processing system provided in this application;
[0025] Figure 9 This is a schematic diagram illustrating an application scenario of an embodiment of the data processing system provided in this application;
[0026] Figure 10 This is a schematic diagram illustrating another application scenario of an embodiment of the data processing system provided in this application;
[0027] Figure 11 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;
[0028] Figure 12 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation
[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0030] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0031] The following explains the terms used in this application:
[0032] 1. Hadoop: Hadoop is a distributed system infrastructure developed by the Apache Software Foundation. Users can develop distributed programs without understanding the underlying details of distributed systems, fully leveraging the power of clusters for high-speed computation and storage.
[0033] 2. Spark: Spark (Apache Spark) is a fast and general-purpose computing engine designed for large-scale data processing.
[0034] 3. MapReduce (MR): MapReduce is a programming model for parallel computation of large datasets (greater than 1TB). It greatly simplifies the process for programmers who lack distributed parallel programming skills to run their programs on distributed systems.
[0035] 4. HDFS: HDFS (Hadoop Distributed File System) is a distributed file system designed to run on general-purpose hardware. HDFS is a highly fault-tolerant system, suitable for deployment on inexpensive machines. HDFS provides high-throughput data access, making it ideal for applications with large-scale datasets.
[0036] 5. Parquet: Parquet (Apache Parquet) is a new columnar storage format in the Hadoop ecosystem. It is compatible with most computing frameworks in the Hadoop ecosystem, supported by a variety of query engines, and is language- and platform-independent.
[0037] 6. HttpFS: HttpFS is a gateway server that provides support for all HDFS file system operations (read and write). It can be used to transfer data between clusters running different versions of Hadoop.
[0038] 7. JSON: JSON (JavaScript Object Notation) is a lightweight data interchange format.
[0039] 8. RocksDB: RocksDB is an embedded, single-machine key-value store engine that can efficiently persist data to disk.
[0040] 9. MQ: MQ (Message Queue) is a container that stores messages during the transmission of messages between systems; it is a common middleware.
[0041] 10. SFTP: FTP (File Transfer Protocol) is a standard protocol for transferring files over a network. SFTP adds the SSH (Secure Shell) security protocol to FTP, making file transfers more secure.
[0042] 11. Upstream Systems: In the loan lifecycle, there are three phases: pre-loan, during-loan, and post-loan. The collection platform operates in the post-loan phase. Therefore, the "upstream systems" mentioned in this article refer to several systems related to the pre-loan and post-loan phases.
[0043] See Figure 1 , Figure 1 This is a schematic flowchart illustrating an embodiment of a data processing method provided in this application. The method includes:
[0044] Step 101: The upstream system updates today's data and saves the data to the databases of various business systems.
[0045] Step 102: Use data synchronization tools to pull data from various business systems to Hadoop's HDFS.
[0046] Step 103: Use the Spark engine to filter data that meets the criteria for the day and save it to HDFS.
[0047] Step 104: Convert the data into a JSON string using a custom MapReduce program and save it to a text file.
[0048] The text file will be temporarily stored in HDFS.
[0049] Step 105: Encrypt the text file for storage and generate a new encrypted file for transmission over the network.
[0050] Step 106: Upload the encrypted files in batches to the SFTP server.
[0051] Step 107: Notify the next level node.
[0052] Meanwhile, the upstream system's offline data has been fully prepared.
[0053] Step 108: The next-level node downloads encrypted files in batches from the SFTP server.
[0054] Step 109: The next level node decrypts the encrypted file.
[0055] Step 110: The next level node reads the decrypted encrypted file, cleans up incomplete and redundant data, and stores it in RocksDB.
[0056] Step 111: The next-level node traverses and reads the data from RocksDB, and distributes the data to the corresponding child nodes of the next-level node via MQ.
[0057] In the above embodiments, the applicant found that because the files contain plaintext information, both SFTP and the application server cache the files during transmission. Therefore, in this context, it is necessary to encrypt and store the files. However, when the data volume is large, the encryption and decryption operations are very time-consuming and place high demands on the server's computing resources. Furthermore, the interaction is complex and cumbersome. In the context of large amounts of data, frequent interactions with the file system are required, and frequent encryption and decryption are also needed for file operations, which increases both processing time and storage burden.
[0058] Based on this, this application proposes the following technical solution: by converting the data of the upper-level node into Parquet file format for storage, the encryption and decryption operations during data interaction are reduced, and HttpFS is used for data transmission. This enables the current-level node to interact directly with the next-level node, reduces data caching on intermediate servers, and improves data interaction efficiency, thereby solving at least one of the above-mentioned problems. Specifically, see [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating one embodiment of the data processing method provided in this application. The method is applied to a data processing system, which includes multiple hierarchical nodes, each hierarchical node comprising at least one hierarchical node. The method includes:
[0059] Step 21: The current level node obtains the data to be processed.
[0060] In some embodiments, if the current level node is the first level node, then the current level node obtains the data to be processed.
[0061] The data to be processed can be generated by upstream systems. For example, in a data processing system, data generated by upstream systems needs to be analyzed and saved every morning. Because the data processing system has many analysis strategies and indicators for different types of data to be processed, the upstream system needs to provide a large amount of offline data to the data processing system completely. Each level of nodes can include at least one node.
[0062] It is understandable that the upstream system can include different subsystems, each of which can perform corresponding functions and generate corresponding data independently. Examples include newly added user information, user operation information, and logs from relevant systems.
[0063] Each node can be an electronic device, which has a processor and a memory.
[0064] In one application scenario, the upstream system sends data to be processed according to a preset period. This data can be generated within the preset period, and it can be incremental data or log data formed from modifications to existing data.
[0065] In some embodiments, the upstream system may include different subsystems, each of which stores data in a corresponding database when it generates data. Therefore, the first-level node can use data synchronization tools to obtain data to be processed from multiple databases.
[0066] In some embodiments, if the current level node is not the first level node, then the current level node obtains the data to be processed sent by the previous level node.
[0067] Step 22: The current level node filters the data to be processed and converts the file format of the filtered data to columnar storage file format for storage.
[0068] In step 22, the current-level node filters the data to be processed, selecting data that meets preset requirements. For example, it filters out user A's data. Another example is filtering out overdue account data.
[0069] Column-oriented storage file format: Data is grouped and stored one column at a time. This allows for faster read speeds for the following reasons:
[0070] (1) Only the columns that are needed need to be read. Therefore, the amount of data read from disk into memory and then moved from memory into the processor cache is reduced.
[0071] (2) The columns are highly compressed. This will reduce the number of bytes that must be read and moved.
[0072] (3) Most queries do not involve all columns in a table. Therefore, many columns never enter memory. This, combined with excellent compression methods, improves buffer pool utilization, thereby reducing total I / O.
[0073] (4) Advanced query execution techniques process column blocks in a simplified manner (called "batch processing"), thereby reducing CPU usage.
[0074] Column-oriented storage file formats can include Parquet, which converts plaintext data into binary data using a unique compilation method. This reduces the need for encryption and subsequent decryption of the plaintext data, thus improving data transmission efficiency.
[0075] In some embodiments, the current level node is the first level node. The file formats in which the upstream system nodes store data may differ. Therefore, the data to be processed received by the first level node needs to be converted to a unified file format at the first level node.
[0076] Step 23: The current level node sends a data fetching notification to the next level node.
[0077] After storing the filtered data to be processed in a columnar storage file format at the current level node, the current level node needs to use the data for the next step of the process. For example, sending a data fetching notification to the next level node.
[0078] In some embodiments, when the current level node is a second level node instead of a first level node, a data capture notification is sent to the third level node. It should be noted that the second level node described in this embodiment refers to an intermediate level node; that is, a second level node is neither a first level node nor the last level node.
[0079] In some embodiments, the current level node is a first-level node, which sends a data capture notification to the second-level node.
[0080] Step 24: In response to the data fetching instruction from the next level node, the current level node will convert the data to be processed into a columnar storage file format and send it to the next level node; wherein, the data fetching instruction is generated based on the data fetching notification.
[0081] In some embodiments, when the current level node is a first-level node, after the second-level node receives the data capture notification, it will generate a corresponding data capture instruction and send the instruction to the first-level node. The first-level node, in response to the second-level node's data capture instruction, will convert the data to be processed into a column-stored file format and send it to the second-level node. The data to be processed can be captured using the Hypertext Transfer Protocol (HTTP). It should be noted that if the current level node is the last level node, after completing steps 21 and 22, the current level node will send the corresponding filtered data to be processed in column-stored file format only after receiving a data acquisition request from another data processing system. In the following description, the current level node or the second level node will be assumed to be not the last level node.
[0082] In one application scenario, refer to Figure 3 Explanation:
[0083] The first-level node receives the data to be processed within a preset period. When the first-level node receives the data to be processed, it filters the data according to preset conditions, removes useless data, keeps the filtered data, and converts the file format of the filtered data to column storage file format for storage.
[0084] After storing all the filtered data to be processed, the first-level node sends a data retrieval notification to the second-level node, so that the second-level node and the first-level node can transmit data through the Hypertext Transfer Protocol.
[0085] In some embodiments, when the current level node is not a first-level node but a second-level node, a data fetching notification is sent to the third-level node. The second-level node then distributes the data to be processed, converted to a column-store file format, to the third-level node. It can be understood that the third-level node may include multiple child nodes, each capable of using the distributed data to perform further operations, such as data display.
[0086] In some embodiments, the third-level nodes include at least two nodes. Each node has a different function and requires different data. Therefore, the second-level nodes need to classify the data to be processed according to the needs of each node in the third level and distribute the classified data to the third-level nodes.
[0087] In this embodiment, compared to the cumbersome plaintext encryption and decryption operations in related technologies during data interaction, the current-level node directly converts the file format of the filtered data to be processed into a columnar storage file format for storage, eliminating the need for cumbersome encryption and decryption operations, and directly transmits data with the next-level node. Compared to the use of intermediate servers for data caching in related technologies, this allows the current-level node to interact directly with the next-level node, reducing data caching on intermediate servers and improving data transmission efficiency.
[0088] See Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of the data processing method provided in this application. The method is applied to a data processing system, which includes multiple hierarchical nodes. The method includes:
[0089] Step 41: The current level node obtains the data to be processed.
[0090] In this embodiment, the current level node can be either the first level node or the second level node described above.
[0091] Step 42: The current level node filters the data to be processed.
[0092] Step 43: The current level node converts the file format of the filtered data to be processed into a columnar storage file format.
[0093] Steps 41 and 42 have the same or similar technical solutions as any of the above embodiments, and will not be described in detail here.
[0094] Step 44: Obtain the preset compression format for the current level node.
[0095] The preset compression format can be configured in advance, such as lzo, gzip, snappy, or uncompressed.
[0096] Snappy is a C++ development package used for compression and decompression. Its goal is not to maximize compression or be compatible with other compression formats, but rather to provide high-speed compression and a reasonable compression ratio. Snappy is faster than zlib, but files are 20% to 100% larger. On a 64-bit Core i7 processor, it can achieve compression speeds of 250 to 500 megabytes per second.
[0097] Step 45: The current level node compresses and stores the data to be processed, which has been converted to columnar storage file format according to the compression format.
[0098] In this embodiment, compression reduces the file size of the data to be processed, thereby reducing the storage space occupied by the current level node and enabling the current level node to store more data.
[0099] Step 46: The current level node sends a data fetching notification to the next level node.
[0100] Step 47: In response to the data fetching command from the next level node, the current level node will convert the data to be processed into a columnar storage file format and send it to the next level node.
[0101] Steps 45-47 have the same or similar technical solutions as any of the above embodiments, and will not be described in detail here.
[0102] In some embodiments, when the current level node is a first-level node and the next level node is a second-level node, the second-level node needs to decompress the compressed data to obtain the complete data after receiving the compressed data to be processed.
[0103] In this embodiment, compared to the cumbersome plaintext encryption and decryption operations during data interaction in related technologies, the current-level node directly converts the filtered data to be processed into a columnar storage file format for storage, eliminating the need for cumbersome encryption and decryption operations. It then directly transmits data with the next-level node. Compared to the use of an intermediate server for data caching in related technologies, this allows the current-level node to interact directly with the next-level node, reducing intermediate server data caching and improving data transmission efficiency. Furthermore, compression reduces the file size of the data to be processed, thereby reducing the storage space occupied by the current-level node and enabling it to store more data.
[0104] See Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of the data processing method provided in this application. The method is applied to a data processing system, which includes multiple hierarchical nodes. The method includes:
[0105] Step 51: The first-level node obtains the data to be processed.
[0106] Step 52: The first-level node filters the data to be processed and converts the file format of the filtered data to columnar storage file format for storage.
[0107] Step 53: The first-level node sends a data capture notification to the second-level node.
[0108] Steps 51 to 53 have the same or similar technical solutions as any of the above embodiments, and will not be described in detail here.
[0109] Step 54: The second-level node responds to the data fetching notification by obtaining the file list of filtered data to be processed from the first-level node via the Hypertext Transfer Protocol.
[0110] Step 55: The second-level node uses multi-threading to retrieve the filtered data to be processed based on the file list.
[0111] Enabling multi-threading to crawl filtered data can improve data crawling efficiency.
[0112] In some embodiments, the file list is typically numbered, allowing threads to be allocated based on these numbers. For example, if there are 1000 files to be processed in the file list, 10 threads can be enabled, with each thread responsible for 100 files. This allows the 10 threads to process data simultaneously, improving data retrieval efficiency.
[0113] Step 56: The second-level node will distribute the data to be processed, which has been converted into a columnar storage file format, to the third-level node.
[0114] In this embodiment, compared to the cumbersome plaintext encryption and decryption operations during data interaction in related technologies, the first-level node directly converts the filtered data to be processed into a columnar storage file format for storage, eliminating the need for cumbersome encryption and decryption operations. It then directly transmits data with the second-level node using the Hypertext Transfer Protocol. Compared to the use of an intermediate server for data caching in related technologies, this allows the first-level node to interact directly with the second-level node, reducing data caching on the intermediate server and improving data transmission efficiency. Furthermore, the second-level node uses multi-threading to retrieve the filtered data, further improving data retrieval efficiency.
[0115] See Figure 6 , Figure 6 This is a flowchart illustrating another embodiment of the data processing method provided in this application. The method is applied to a data processing system, which includes multiple hierarchical nodes. The method includes:
[0116] Step 61: The first-level node obtains the data to be processed.
[0117] Step 62: The first-level node filters the data to be processed and converts the file format of the filtered data to columnar storage file format for storage.
[0118] Step 63: The first-level node sends a data capture notification to the second-level node.
[0119] Step 64: The second-level node responds to the data fetching notification by fetching the filtered data to be processed via the Hypertext Transfer Protocol; wherein, the data fetching instruction is generated based on the data fetching notification.
[0120] Step 65: The second-level node will convert the data to be processed into columnar storage file format and store it in the embedded database.
[0121] The embedded database can be RocksDB. Considering that the amount of data to be processed may be large, the storage space in the node's memory is usually small. Using the node's memory would cause data overflow. Therefore, an embedded database is used, utilizing part of the node's disk storage space as an embedded database to store the filtered data to be processed.
[0122] In some embodiments, see Figure 7 Step 65 can be the following process:
[0123] Step 71: The second-level nodes will convert the data to be processed into columnar storage file format for data cleaning.
[0124] In some embodiments, data cleaning can remove redundant and incomplete data from the data to be processed.
[0125] It is understandable that the data to be processed obtained from different nodes may be the same, such as basic user information.
[0126] Furthermore, some data may be incomplete or lack key information, such as missing identity information in basic user information. Such data has no special function and should also be deleted.
[0127] Step 72: The second-level node stores the cleaned data to be processed into the embedded database.
[0128] In this embodiment, data cleaning can further filter the data to be processed, further reduce the file size of the data, and alleviate the storage pressure on the embedded database.
[0129] In other embodiments, step 65 may be that the second-level node parses and converts the data to be processed into a columnar storage file format; the second-level node then converts the parsed data to be processed into a JavaScript object musical notation file format and stores it in the embedded database.
[0130] In other embodiments, the second-level node may perform data cleaning on the filtered data to be processed, and convert the cleaned data to be processed into a JavaScript object musical notation file format and store it in the embedded database.
[0131] Step 66: The second-level node traverses the data to be processed in the embedded database that has been converted to columnar storage file format, and adds the data to be processed in columnar storage file format to the message queue.
[0132] In some embodiments, after storing all the data to be processed into the embedded database, the second-level node needs to add the filtered data to the message queue in a traversal manner.
[0133] In some embodiments, the message queue may be at least one of Redis Stream message queue, ZeroMQ, RabbitMQ, ActiveMQ, and Kafka / Jafka.
[0134] Step 67: The second-level node distributes the data to be processed to the next third-level node through the message queue.
[0135] In some embodiments, the data to be processed can be identified, with each identifier corresponding to one of the nodes in the third-level nodes.
[0136] Therefore, when data to be processed is detected in the message queue, the identifier of the data to be processed is obtained, and the corresponding node transmission path is determined based on the identifier. The data to be processed is then sent to the corresponding node in the third-level node through this node transmission path.
[0137] In this embodiment, compared to the cumbersome plaintext encryption and decryption operations during data interaction in related technologies, the first-level node directly converts the filtered data to be processed into a columnar storage file format for storage, eliminating the need for cumbersome encryption and decryption operations. It then directly transmits data with the second-level node using the Hypertext Transfer Protocol. Compared to the use of an intermediate server for data caching in related technologies, this allows the first-level node to interact directly with the second-level node, reducing data caching on the intermediate server and improving data transmission efficiency. Furthermore, the use of an embedded database provides greater storage space, avoiding data overflow issues caused by using memory for data storage.
[0138] See Figure 8 , Figure 8 This is a schematic diagram of the structure of an embodiment of the data processing system provided in this application. The data processing system 100 includes multiple hierarchical nodes, such as... Figure 8 The first-level node 10, the second-level node 20, and at least one third-level node 30 are shown.
[0139] The first-level node 10 is used to acquire data to be processed; to filter the data to be processed, and to convert the file format of the filtered data to column storage file format for storage; and to generate a data capture notification.
[0140] The second-level node 20 is used to respond to a data fetching notification and fetch the filtered data to be processed via the Hypertext Transfer Protocol; wherein, the data fetching instruction is generated based on the data fetching notification and the filtered data to be processed is distributed to the third-level node 30.
[0141] In one application scenario, refer to Figure 9 The following explanation will be based on first-level node 10, second-level node 20, and third-level node 30:
[0142] Each upstream system compiles the data and sends the resulting data to be processed to the first-level node 10.
[0143] The first-level node 10 filters the data to be processed, converts the file format of the filtered data to Parquet file format for storage, and generates a data fetching notification.
[0144] First-level node 10 sends a data capture notification to second-level node 20.
[0145] The second-level node 20 generates data fetching instructions based on the data fetching notification.
[0146] The second-level node 20 sends the data fetching command to the first-level node 10, and uses HttpFS to fetch the filtered data to be processed.
[0147] The second-level node 20 distributes the filtered data to the third-level node 30.
[0148] In another application scenario, combined with Figure 10 Explanation:
[0149] The accounting system and customer system in the upstream system summarize the data of the day according to business logic and merge it into their respective business line databases. The databases are basically common relational databases, such as Oracle, SQL Server and MySQL.
[0150] Once the upstream system has prepared the data, it is extracted into the Hadoop distributed storage system, stored in Parquet format, and compressed using the snappy compression format.
[0151] The data platform uses HttpFS for data extraction.
[0152] Specifically, the system retrieves a list of Parquet files and uses multi-threading to download them. Then, it parses the downloaded Parquet files, formats the columnar data into JSON according to business rules, and stores the JSON data in RocksDB.
[0153] Then, it loops through RocksDB, sending formatted JSON data to the message queue. Consumers listening to the message queue will receive the corresponding data and perform the corresponding business processing.
[0154] In this context, consumers can be sub-platforms of the data platform, such as external data platforms.
[0155] In some embodiments, the data processing system 100 described above can implement the methods of any of the above embodiments.
[0156] See Figure 11 , Figure 11 This is a schematic diagram of an embodiment of the electronic device provided in this application. The electronic device 130 includes a processor 131 and a memory 132 coupled to the processor 131. The memory 132 stores a computer program, and the processor 131 executes the computer program to implement the following method:
[0157] If the current level node is a first-level node, it retrieves the data to be processed. If the current level node is not a first-level node, it retrieves the data to be processed sent by the previous level node. The current level node filters the data to be processed and converts the file format of the filtered data to columnar storage file format for storage. The current level node sends a data fetching notification to the next level node. In response to the data fetching instruction from the next level node, the current level node sends the data to be processed, converted to columnar storage file format, to the next level node. The data fetching instruction is generated based on the data fetching notification.
[0158] It is understood that the processor 131 is also used to execute computer programs to implement the methods of any of the above embodiments.
[0159] See Figure 12 , Figure 12This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 140 stores a computer program 141, which, when executed by a processor, implements the following method:
[0160] If the current level node is a first-level node, it retrieves the data to be processed. If the current level node is not a first-level node, it retrieves the data to be processed sent by the previous level node. The current level node filters the data to be processed and converts the file format of the filtered data to columnar storage file format for storage. The current level node sends a data fetching notification to the next level node. In response to the data fetching instruction from the next level node, the current level node sends the data to be processed, converted to columnar storage file format, to the next level node. The data fetching instruction is generated based on the data fetching notification.
[0161] It is understood that when computer program 141 is executed by a processor, it is also used to implement the methods of any of the above embodiments.
[0162] In summary, the data processing method, system, electronic device, and computer-readable storage medium provided in this application are applied to a data processing system comprising multiple hierarchical nodes. The method includes: if the current hierarchical node is a first-level node, the current hierarchical node acquires data to be processed; if the current hierarchical node is not a first-level node, the current hierarchical node acquires data to be processed sent by the previous hierarchical node; the current hierarchical node filters the data to be processed and converts the file format of the filtered data to a columnar storage file format for storage; the current hierarchical node sends a data capture notification to the next hierarchical node; in response to the data capture instruction from the next hierarchical node, the current hierarchical node sends the data to be processed, converted to a columnar storage file format, to the next hierarchical node; wherein, the data capture instruction is generated based on the data capture notification. Compared to the cumbersome plaintext encryption and decryption operations in related technologies during data interaction, the current-level node directly converts the filtered data to be processed into a columnar storage file format for storage, eliminating the need for cumbersome encryption and decryption operations. It then directly transmits data with the second-level node. Compared to the use of intermediate servers for data caching in related technologies, this approach allows the current-level node to interact directly with the next-level node, reducing data caching on intermediate servers, improving data transmission efficiency, reducing the occupation of network and storage resources during transmission, alleviating transmission and storage pressure, and enabling throughput of large amounts of data.
[0163] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of circuits or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
[0164] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.
[0165] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0166] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A data processing method, characterized in that: The method is applied to a data processing system, the data processing system includes multiple hierarchical nodes, and the method includes: If the current level node is a first level node, the current level node obtains the data to be processed; if the current level node is not a first level node, the current level node obtains the data to be processed sent by the previous level node; The current level node filters the data to be processed, and converts the file format of the filtered data to be processed into a column storage file format to store it in the current level node; wherein the filtered data to be processed stored in the column storage file format is stored in an embedded database, and the embedded database is part of the storage space of the disk of the current level node; The current level node sends a data crawling notification to the next level node; In response to the data capture instruction of the next-level node, the current-level node sends the to-be-processed data converted into a column storage file format to the next-level node; wherein the data capture instruction is generated based on the data capture notification.
2. The method according to claim 1, characterized in that The current level node filters the data to be processed, and converts the file format of the filtered data to be processed into a column storage file format for storage, including: The current level node filters the data to be processed; Convert the file format of the filtered data to be processed into a column storage file format; Get the preset compression format; The data to be processed converted into the column storage file format is compressed and stored according to the preset compression format.
3. The method according to claim 1, characterized in that: If the current level node is not a first level node, the method further includes: The current level node obtains the file list of the to-be-processed data filtered in the previous level node in response to the data capture notification; The current level node enables multi-threading to capture the filtered data to be processed according to the file list.
4. The method according to claim 1, characterized in that: The current level node sends the to-be-processed data converted into a column storage file format to the next level node, including: The current level node traverses the embedded database, obtains the to-be-processed data converted into a column storage file format, and adds the to-be-processed data converted into a column storage file format to a message queue; The current level node sends the to-be-processed data converted into a column storage file format to the next level node through the message queue.
5. The method according to claim 4, characterized in that The current level node stores the to-be-processed data converted into a column storage file format in an embedded database, including: The current level node parses and converts the to-be-processed data into a column storage file format; The current level node converts the parsed data to be processed into an object notation file format and stores it in the embedded database.
6. The method according to claim 4, characterized in that The current level node stores the to-be-processed data converted into a column storage file format in an embedded database, including: The current level node converts the to-be-processed data into a column storage file format for data cleaning; The current level node stores the cleaned data to be processed in an embedded database.
7. The method according to claim 1, characterized in that If the current level node is a first level node, the current level node obtains data to be processed, including: The current level node obtains the data to be processed from multiple databases using a data synchronization tool.
8. A data processing system, characterized in that: The data processing system includes a plurality of hierarchical nodes; each hierarchical node is used to implement the method according to any one of claims 1-7.
9. An electronic device, characterized in that: The electronic device comprises a processor and a memory coupled to the processor, wherein a computer program is stored in the memory, and the processor is configured to execute the computer program to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.
