A method and system for ensuring consistency of data synchronization of a distributed file system

Abstract

The application provides a method and system for ensuring consistency of data synchronization of a distributed file system. The method comprises the following steps: responding to an update period of acquired business data; performing secondary processing on the data according to an actual update period of the acquired business data; and responding to periodic full data processing to ensure that all data is up-to-date. The method is used in an offline data warehouse based on HDFS storage+HIVE analysis, is applicable to consistency of business data in various industries, and is particularly suitable for a scenario in which the amount of synchronized data is large and the actual update period of data is short. The method ensures data consistency by performing secondary processing on large amounts of business data according to the actual update period of the business data and combining periodic full data processing. Since the amount of data read from a source library is reduced, the read pressure of the source library is reduced.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology, specifically relating to a method and system for ensuring data synchronization consistency in a distributed file system. Background Technology

[0002] With the explosive growth of data in recent years, the concept of data warehouses has been mentioned more and more in the field of data governance. In most scenarios, offline data warehouses based on the distributed file system HDFS are used. They adopt the HDFS+Hive technical architecture and use the SPARK computing engine for data processing. They can solve the problem of storing large amounts of data and perform relational database-like query analysis. They are widely used in T+1 scenarios.

[0003] In some companies and organizations, the consistency of business data is of particular importance because if the consistency of data cannot be guaranteed, downstream business will inevitably experience delays.

[0004] In data warehouse construction, based on HDFS distributed file system storage, it is not possible to directly update the file content. Data synchronization is performed by accumulation. Data in the data warehouse may be duplicated in business logic. Alternatively, a partitioning method is used to extract all data. After extraction, the file path is switched when creating tables in HIVE.

[0005] In the HDFS+Hive architecture, HDFS can effectively handle continuously growing log data through partitioning. For business data, which is subject to updates, a full data extraction plus partition switching approach can be used when the data volume is small. However, when the data volume is large, scheduled full extractions can hardly guarantee data timeliness, and even T+1 may not be able to guarantee timely updates. Furthermore, scheduled full extractions also put significant pressure on the source database. If incremental partitioning is used for storage, data consistency cannot be guaranteed. Multiple records with the same business primary key may exist in HDFS, and downstream applications must retrieve the latest version before continuing data flow. This often leads to data analysis departments criticizing the ETL department for failing to guarantee data consistency.

[0006] Therefore, it is of great significance to propose a method and system for ensuring data synchronization consistency in distributed file systems. Summary of the Invention

[0007] To address the problem of existing methods failing to guarantee data consistency, this invention provides a method and system for ensuring data synchronization consistency in a distributed file system, thereby resolving the aforementioned technical deficiencies.

[0008] In a first aspect, this invention proposes a method for ensuring data synchronization consistency in a distributed file system, the method comprising the following steps:

[0009] Responding to the update cycle of acquiring business data;

[0010] The data is then processed a second time based on the actual update cycle of the acquired business data.

[0011] The system periodically processes all data to ensure that all data is up-to-date.

[0012] Preferred options include:

[0013] Retrieve data files within the update cycle from the target database based on the current task time;

[0014] The calculation is performed using the SPARK computing engine, and the data is written back to the original partition.

[0015] Move the calculated data file to the backup directory and refresh the HIVE table.

[0016] Further preferred methods include: during data synchronization, selecting appropriate partitions based on the size of the incremental data, processing partitions by year, month, day, and hour, and ensuring data timeliness by controlling the scheduling frequency of data synchronization.

[0017] Further preferred, the secondary processing specifically includes:

[0018] Using the SPARK computing engine, HDFS file data is written into memory, and SPARK-SQL is used to remove duplicates and retrieve the latest data.

[0019] The data is written back to the corresponding HDFS partition to ensure data consistency.

[0020] Furthermore, the secondary processing specifically includes:

[0021] The calculation is based on the actual update cycle of the business data, and each task only takes the data from the most recent period for calculation and retrieves the latest data.

[0022] Secondly, embodiments of the present invention also provide a system for ensuring data synchronization consistency in a distributed file system, comprising:

[0023] The acquisition module is used to respond to the update cycle of the acquired business data;

[0024] The secondary processing module is used to process the data a second time according to the actual update cycle of the acquired business data.

[0025] The full data processing module is used to periodically process all data to ensure that all data is up-to-date.

[0026] Further preferred options include:

[0027] The computation engine module is used to perform calculations using the SPARK computation engine and write the data back to the original partition.

[0028] The data backup module is used to move the calculated data files to the backup directory and refresh the HIVE table.

[0029] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0030] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0031] Compared with the prior art, the beneficial results of the present invention are as follows:

[0032] (1) This invention is used in offline data warehouses based on HDFS storage and HIVE analysis. It is applicable to ensuring business data consistency in various industries, especially for scenarios with large-scale synchronized data and short actual data update cycles. By performing secondary processing on large-scale business data according to the actual update cycle of business data, combined with regular full processing, data consistency is ensured. As the amount of data read from the source database is reduced, the read pressure on the source database is reduced.

[0033] (2) Each task pulls business data within the actual update cycle and performs secondary processing, which reduces the overall data volume and saves computing resources. The processing can be performed using the client server memory or the SPARK cluster memory, which is flexible and convenient.

[0034] (3) Downstream data users no longer need to perform deduplication and retrieve the latest data, which improves data production efficiency and also prevents data users from criticizing the ETL provider for data inconsistency. Attached Figure Description

[0035] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0036] Figure 1 This is an exemplary device architecture diagram in which an embodiment of the present invention can be applied;

[0037] Figure 2 This is a flowchart illustrating a method for ensuring data synchronization consistency in a distributed file system according to an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of the overall processing logic of a method for ensuring data synchronization consistency in a distributed file system according to an embodiment of the present invention.

[0039] Figure 4 This is a flowchart illustrating the specific implementation steps of the method for ensuring data synchronization consistency in a distributed file system according to an embodiment of the present invention.

[0040] Figure 5 A schematic diagram of a system for ensuring data synchronization consistency in a distributed file system, as described in an embodiment of the present invention.

[0041] Figure 6 This is a schematic diagram of the structure of a computer device suitable for implementing electronic devices according to embodiments of the present invention. Detailed Implementation

[0042] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and are illustrated by specific illustrative embodiments in which the invention may be practiced. In this regard, directional terms such as “top,” “bottom,” “left,” “right,” “up,” “down,” etc., are used with reference to the orientation of the described figures. Because components of the embodiments can be positioned in several different orientations, directional terms are used for illustrative purposes and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the invention is defined by the appended claims.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] Figure 1An exemplary system architecture 100 for processing information, or for processing information, to which embodiments of the present invention can be applied, is shown.

[0045] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0046] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0047] Terminal devices 101, 102, and 103 can be various electronic devices with communication functions, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0048] Server 105 can be a server that provides various services, such as a background information processing server that processes verification request information sent by terminal devices 101, 102, and 103. The background information processing server can analyze and process the received verification request information and obtain processing results (such as verification success information used to indicate that the verification request is a valid request).

[0049] It should be noted that the information processing method provided in the embodiments of the present invention is generally executed by server 105, and correspondingly, the device for processing information is generally disposed in server 105. Furthermore, the information sending method provided in the embodiments of the present invention is generally executed by terminal devices 101, 102, and 103, and correspondingly, the device for sending information is generally disposed in terminal devices 101, 102, and 103.

[0050] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, used to provide distributed services), or as a single software program or multiple software modules; no specific limitations are made here.

[0051] In the HDFS+Hive architecture, HDFS can effectively handle continuously growing log data through partitioning. For business data, which is subject to updates, a full data extraction plus partition switching approach can be used when the data volume is small. However, when the data volume is large, scheduled full extractions can hardly guarantee data timeliness, and even T+1 may not be able to guarantee timely updates. Furthermore, scheduled full extractions also put significant pressure on the source database. If incremental partitioning is used for storage, data consistency cannot be guaranteed. Multiple records with the same business primary key may exist in HDFS, and downstream applications must retrieve the latest version before continuing data flow. This often leads to data analysis departments criticizing the ETL department for failing to guarantee data consistency.

[0052] To address the above issues, in business data scenarios with large data volumes, while ensuring data timeliness, this invention performs secondary processing on the data files written to HDFS based on the actual update cycle of the business data to ensure that the data stored in HDFS is always the latest in terms of business logic.

[0053] Figure 2 An embodiment of the present invention discloses a method for ensuring data synchronization consistency in a distributed file system, such as... Figure 2 As shown, the method includes the following steps:

[0054] S1, responding to the update cycle of acquiring business data;

[0055] S2. Perform secondary processing on the data based on the actual update cycle of the acquired business data;

[0056] S3 responds by periodically processing all data to ensure that all data is up-to-date.

[0057] Specifically, the earning processing logic in this embodiment of the invention is as follows: Figure 3 As shown.

[0058] 1) Perform secondary processing on the data according to the actual update cycle of the business data.

[0059] During data synchronization, appropriate partitions are selected based on the size of the incremental data. Partitions can be processed by year, month, day, or hour. Data timeliness is ensured by controlling the scheduling frequency of data synchronization. Since HDFS is stored in file format, it cannot be updated using primary key comparison via SQL like relational databases, nor can it be overwritten by specifying _id like full-text indexes. However, the SPARK computing engine can be used to write HDFS file data into memory, use SPARK-SQL to remove duplicates and retrieve the latest data, and then write the data back to the corresponding HDFS partition. This secondary processing method ensures data consistency.

[0060] Furthermore, the secondary processing should be based on the actual update cycle of the business data. The actual update cycle refers to the time period from the generation of a piece of business data to its change. The purpose of this method is to save computing resources. If the actual update cycle of the data is not considered and the entire table data is used for calculation directly, it will lead to a waste of resources. In fact, the data that needs to be updated is only the data within a certain cycle.

[0061] For example, in the public security sector, common guest accommodation information tables typically have a maximum interval of six months between check-in and check-out, and most of this data is updated within a shorter period, often changing within a week. Calculating and retrieving the latest data for each task saves significant computational resources compared to retrieving the entire dataset each time.

[0062] 2) Perform full data processing regularly.

[0063] From a rigorous perspective, calculating based on the actual update cycle of business data only satisfies 95% of data updates. However, some special data may have longer actual update cycles and cannot be covered by conventional practices. For example, hotel guests may stay for extended periods, with check-in and check-out intervals exceeding one year. Therefore, the remaining 5% of data requires periodic full table calculations to ensure all data is up-to-date.

[0064] The specific implementation steps of this invention are as follows: Figure 4 As shown in the diagram. Retrieving data based on the current task time ensures that data files in the partition that are being written but not yet completed are not read, thus preventing data loss. Moving the data files after calculation ensures uninterrupted data access downstream.

[0065] This invention employs a secondary processing method based on the actual update cycle of large-volume business data, combined with periodic full processing, to ensure data consistency. As the amount of data read from the source database is reduced, the read pressure on the source database is decreased.

[0066] Each task retrieves data within the actual update cycle of the business data and performs secondary processing, reducing the overall data volume and saving computing resources. The processing can be performed using the client server's memory or the SPARK cluster's memory, offering flexibility and convenience. Downstream data users no longer need to perform deduplication and fetch the latest data, improving data production efficiency and preventing data users from criticizing the ETL provider for data inconsistencies.

[0067] This invention is used in offline data warehouses based on HDFS storage and HIVE analytics, and is applicable to ensuring business data consistency across various industries, especially for scenarios with large-scale synchronized data and short actual data update cycles.

[0068] Secondly, embodiments of the present invention also disclose a system for ensuring data synchronization consistency in a distributed file system, such as... Figure 5 As shown, it includes: an acquisition module 51, a secondary processing module 52, a full processing module 53, a calculation engine module 54, and a data backup module 55.

[0069] In one specific embodiment, the acquisition module 51 is used to acquire the update cycle of the business data; the secondary processing module 52 is used to perform secondary processing on the data according to the actual update cycle of the acquired business data; the full processing module 53 is used to periodically process the full data to ensure that all data is up-to-date; the calculation engine module 54 is used to perform calculations using the SPARK calculation engine and write the data back to the original partition; and the data backup module 55 is used to move the calculated data file to the backup directory and refresh the HIVE table.

[0070] The following is for reference. Figure 6 It illustrates an electronic device suitable for implementing embodiments of the present invention (e.g., Figure 1 The diagram shows the structure of a computer device 600 (a server or terminal device). Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0071] like Figure 6 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 603 or programs loaded from storage section 609 into random access memory (RAM) 606. Various programs and data required for the operation of device 600 are also stored in RAM 604. The CPU 601, GPU 602, ROM 603, and RAM 604 are interconnected via bus 605. Input / output (I / O) interface 606 is also connected to bus 605.

[0072] The following components are connected to I / O interface 606: an input section 607 including a keyboard, mouse, etc.; an output section 608 including an LCD, speakers, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card, such as a LAN card or modem. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to I / O interface 606 as needed. A removable medium 612, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 611 as needed so that computer programs read from it can be installed into storage section 609 as needed.

[0073] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 610, and / or installed from removable medium 612. When the computer program is executed by central processing unit (CPU) 601 and graphics processing unit (GPU) 602, the functions defined in the methods of this invention are performed.

[0074] It should be noted that the computer-readable medium described in this invention can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution apparatus, device, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0075] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0076] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based devices that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0077] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor.

[0078] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: respond to the acquisition of the update cycle of business data; perform secondary processing on the data according to the actual update cycle of the acquired business data; and periodically process all data to ensure that all data is up-to-date.

[0079] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for ensuring data synchronization consistency in a distributed file system, characterized in that, The method includes the following steps: Responding to the update cycle of acquiring business data; The data is processed in a second time according to the actual update cycle of the acquired business data. The second processing specifically includes: during data synchronization, appropriate partition selection is performed according to the size of incremental data. The partitions are processed by year, month, day, and hour. Data timeliness is ensured by controlling the scheduling frequency of data synchronization. The HDFS file data is written to memory using the SPARK computing engine. After deduplication and retrieving the latest data using SPARK-SQL, the data is written back to the corresponding HDFS partition. In response to periodically performing full data processing to ensure all data is up-to-date, and after the secondary processing and periodic full processing, the calculated data files are moved to the backup directory and the HIVE table is refreshed.

2. The method for ensuring data synchronization consistency in a distributed file system according to claim 1, characterized in that, Specifically, it includes: Retrieve data files within the update cycle from the target database based on the current task time; The calculations are performed using the SPARK computing engine, and the data is written back to the original partition.

3. The method for ensuring data synchronization consistency in a distributed file system according to claim 2, characterized in that, The secondary processing also includes: The calculation is based on the actual update cycle of the business data, and each task only takes the data from the most recent period for calculation and retrieves the latest data.

4. A system for ensuring data synchronization consistency in a distributed file system, characterized in that, Including the method as described in any one of claims 1-3, further comprising: The acquisition module is used to respond to the update cycle of the acquired business data; The secondary processing module is used to process the data a second time according to the actual update cycle of the acquired business data. The full data processing module is used to periodically process all data to ensure that all data is up-to-date.

5. The system for ensuring data synchronization and consistency in a distributed file system according to claim 4, characterized in that, Also includes: The computation engine module is used to perform calculations using the SPARK computation engine and write the data back to the original partition. The data backup module is used to move the calculated data files to the backup directory and refresh the HIVE table.

6. An electronic device, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 3.

Images