Data deduplication merging method, system, electronic device and storage medium

Abstract

The application provides a data deduplication merging method and system, an electronic device and a storage medium. The method comprises the following steps: a selection module selects a master deduplication server from a plurality of deduplication servers, and takes other deduplication servers except the master deduplication server as slave deduplication servers; the master deduplication server determines the priority of the plurality of deduplication servers, takes a deduplication server with the highest priority as a target deduplication server, obtains to-be-processed data, and transmits the to-be-processed data to the target deduplication server; and the target deduplication server performs deduplication merging processing on the to-be-processed data. The scheme can realize timely processing of the to-be-processed data, and improves the deduplication merging efficiency.

Description

Technical Field

[0001] This application relates to big data technology, and in particular to a data deduplication and merging method, system, electronic device and storage medium. Background Technology

[0002] With the rapid development of information technology, massive amounts of data require deduplication and merging. Data deduplication and merging refers to removing duplicate data from the data to be processed and updating the entire dataset based on the incremental data. Currently, data deduplication and merging processes are mainly handled in centralized and distributed manner.

[0003] In the centralized data deduplication method, after receiving the data to be processed, the data to be processed is inserted into the data queue and processed sequentially. In the distributed data deduplication method, after receiving the data to be processed, the data to be processed is transmitted to the management server. The management server selects the target deduplication server to perform deduplication and merging of the data to be processed. This involves multiple data interactions between the deduplication server and the management server, making the data deduplication and merging process complex.

[0004] The aforementioned existing technologies cannot achieve timely processing of data to be processed, and the efficiency of data deduplication and merging is low. Summary of the Invention

[0005] This application provides a data deduplication and merging method, system, electronic device, and storage medium, aiming to solve the problem of low efficiency in data deduplication and merging.

[0006] In a first aspect, this application provides a data deduplication and merging method applied to a data deduplication and merging system. The data deduplication and merging system includes multiple deduplication servers and an election module. The method includes: the election module electing a master deduplication server from the multiple deduplication servers, and designating the other deduplication servers besides the master deduplication server as slave deduplication servers; the master deduplication server determining the priority of the multiple deduplication servers, designating the deduplication server with the highest priority as the target deduplication server, obtaining the data to be processed, and transmitting the data to be processed to the target deduplication server; and the target deduplication server performing deduplication and merging processing on the data to be processed.

[0007] Optionally, the primary deduplication server determines the priority of the plurality of deduplication servers and selects the deduplication server with the highest priority as the target deduplication server, including: the primary deduplication server acquiring heartbeat data and status data of each secondary deduplication server; wherein, the heartbeat data includes the data processing speed and the length of the data queue to be processed of the secondary deduplication server; the status data includes the data processing speed and the length of the data queue to be processed of the primary deduplication server; the primary deduplication server calculates a priority coefficient for each deduplication server based on the heartbeat data and status data of the secondary deduplication servers; the priority coefficient of the deduplication server is the ratio of the data processing speed of the deduplication server to the length of the data queue to be processed of the deduplication server; the primary deduplication server selects the deduplication server with the highest priority coefficient among the plurality of deduplication servers as the target deduplication server.

[0008] Optionally, the target deduplication server performs deduplication merging processing on the data to be processed, including: the target deduplication server performs deduplication merging processing on the data to be processed and the full sample data of the current target deduplication server to obtain the full sample data and incremental sample data of the target deduplication server after deduplication merging processing.

[0009] Optionally, the target deduplication server performs deduplication and merging processing on the data to be processed, and then further includes: the target deduplication server sending its current incremental sample data to an incremental server; the incremental server saving the received incremental sample data from the target deduplication server as its current full sample data, and transmitting the full sample data of the incremental server to the full server and the plurality of deduplication servers; the full server updating its full sample data based on using the received full sample data from the incremental server as its current incremental sample data; and the plurality of deduplication servers updating their full sample data based on using the received full sample data from the incremental server as their current incremental sample data.

[0010] Optionally, the method further includes: when the update cycle arrives, the plurality of deduplication servers update the full sample data of the plurality of deduplication servers based on the full sample data of the full server and the full sample data of the incremental server.

[0011] Optionally, the election module obtains a master deduplication server from multiple deduplication servers through election, and designates the other deduplication servers besides the master deduplication server as slave deduplication servers. This includes: performing an election process for each deduplication server until a master deduplication server is obtained; wherein the election process is as follows: the deduplication server sends an election request to other deduplication servers; if the number of election votes returned by the other deduplication servers received by the deduplication server is greater than a first threshold, then the deduplication server is designated as the master deduplication server; otherwise, the deduplication server is designated as a slave deduplication server.

[0012] Secondly, this application provides a data deduplication and merging system, the system comprising: multiple deduplication servers and an election module; the election module is used to obtain a master deduplication server from the multiple deduplication servers through election, and to designate the other deduplication servers among the multiple deduplication servers besides the master deduplication server as slave deduplication servers; the master deduplication server is used to determine the priority of the multiple deduplication servers, to designate the deduplication server with the highest priority as the target deduplication server, and to obtain the data to be processed and transmit the data to be processed to the target deduplication server; the target deduplication server is used to perform deduplication and merging processing on the data to be processed.

[0013] Optionally, the primary deduplication server is used to determine the priority of the plurality of deduplication servers. When selecting the deduplication server with the highest priority as the target deduplication server, it specifically involves: acquiring heartbeat data of each secondary deduplication server and status data of the primary deduplication server; wherein the heartbeat data includes the data processing speed of the secondary deduplication server and the length of the data queue to be processed; the status data includes the data processing speed of the primary deduplication server and the length of the data queue to be processed; calculating the priority coefficient of each deduplication server based on the heartbeat data of the secondary deduplication server and the status data of the primary deduplication server; the priority coefficient of the deduplication server is the ratio of the data processing speed of the deduplication server to the length of the data queue to be processed of the deduplication server; and selecting the deduplication server with the highest priority coefficient among the plurality of deduplication servers as the target deduplication server.

[0014] Optionally, the target deduplication server is specifically used to: perform deduplication merging processing on the data to be processed and the full sample data of the current target deduplication server to obtain the full sample data and incremental sample data of the target deduplication server after deduplication merging processing.

[0015] Optionally, the system further includes: an incremental server and a full server; the target deduplication server is further configured to send the incremental sample data of the current target deduplication server to the incremental server; the incremental server is configured to save the received incremental sample data of the target deduplication server as the full sample data of the current incremental server, and transmit the full sample data of the incremental server to the full server and the plurality of deduplication servers; the full server is configured to update the full sample data of the full server according to using the received full sample data of the incremental server as the incremental sample data of the current full server; the plurality of deduplication servers are configured to update the full sample data of the deduplication server according to using the received full sample data of the incremental server as the incremental sample data of the current deduplication server.

[0016] Optionally, the plurality of deduplication servers are further configured to: update the full sample data of the plurality of deduplication servers based on the full sample data of the full server and the full sample data of the incremental server when the update cycle arrives.

[0017] Optionally, the election module is specifically used for: performing an election process for each deduplication server until a master deduplication server is obtained; wherein, the election process is as follows: the deduplication server sends an election request to other deduplication servers; if the number of election votes returned by the other deduplication servers received by the deduplication server is greater than a first threshold, then the deduplication server is designated as the master deduplication server; otherwise, the deduplication server is designated as the slave deduplication server.

[0018] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described above.

[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described above.

[0020] In the data deduplication and merging method, system, electronic device, and storage medium provided in this application, the election module elects a master deduplication server from multiple deduplication servers, and designates the other deduplication servers besides the master server as slave deduplication servers. The master deduplication server determines the priority of the multiple deduplication servers, designates the deduplication server with the highest priority as the target deduplication server, obtains the data to be processed, and transmits the data to be processed to the target deduplication server. The target deduplication server performs deduplication and merging processing on the data to be processed. The scheme of this application, by electing a master deduplication server from multiple deduplication servers and determining the priority of the multiple deduplication servers, with a higher priority indicating more timely data processing, enables timely processing of the data to be processed and improves the efficiency of data deduplication and merging. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1 This is a flowchart illustrating a data deduplication and merging method provided in Embodiment 1 of this application;

[0023] Figure 2 A schematic diagram illustrating the election process of a primary deduplication server provided in Embodiment 1 of this application;

[0024] Figure 3 This is a flowchart illustrating a data deduplication and merging method provided in Embodiment 2 of this application;

[0025] Figure 4 This is a schematic diagram of the structure of a data deduplication and merging system provided in Embodiment 3 of this application;

[0026] Figure 5 This is a schematic diagram of the structure of the electronic device provided in Embodiment 4 of this application.

[0027] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0030] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, to implement the application in a sequence other than those given in the embodiments illustrated or described herein.

[0031] Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover but not exclude inclusion. For example, a product or device that includes a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device. As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0032] With the rapid development of information technology, massive amounts of data require deduplication and merging. For example, recommending products to customers, sending marketing text messages to users, and merging data collected from multiple data sources often require data deduplication and merging. Data deduplication and merging refers to removing duplicate data from the data to be processed and updating the entire dataset based on incremental data. Currently, data deduplication and merging processes are mainly handled in centralized and distributed manner.

[0033] The centralized data deduplication and merging method stores all sample data in a database. After receiving the data to be processed, it adds the data to the queue. The deduplication server processes the data in the queue in order, but it cannot achieve timely processing of the data to be processed.

[0034] The distributed data deduplication and merging method involves receiving data to be processed, transmitting the data to a management server, which then distributes it to distributed deduplication servers for processing. The management server adds the data to a main queue, while the deduplication servers retrieve their sub-queues from the management server and perform deduplication and merging on the data within each sub-queue. Each time a deduplication server performs deduplication and merging, it needs to retrieve the full sample data from the management server and transmit the updated full sample data back to the management server. Understandably, this distributed data deduplication and merging method involves multiple data interactions between the deduplication servers and the management server, each involving a large amount of data transmission. The complex deduplication and merging process makes it impossible to achieve timely processing of data.

[0035] The data deduplication and merging method provided in this application determines a master deduplication server from multiple deduplication servers through election. The master deduplication server determines the priority of multiple deduplication servers. The higher the priority of the deduplication server, the more timely the deduplication server processes the data. The data to be processed is transferred to the deduplication server with the highest priority for processing, which can realize the timely processing of the data to be processed and improve the efficiency of data deduplication and merging.

[0036] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In the description of this application, unless otherwise expressly specified and limited, the terms should be broadly understood within the art. The embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Example 1

[0038] Figure 1 This is a flowchart illustrating a data deduplication and merging method provided in Embodiment 1 of this application. The method in this embodiment is applied to a data deduplication and merging system, which includes multiple deduplication servers and an election module, such as... Figure 1 As shown, the method includes the following steps:

[0039] S101. The election module selects the master deduplication server from multiple deduplication servers through election, and designates the other deduplication servers besides the master deduplication server as slave deduplication servers.

[0040] S102. The main deduplication server determines the priority of multiple deduplication servers and selects the deduplication server with the highest priority as the target deduplication server.

[0041] S103. The main deduplication server obtains the data to be processed and transmits the data to the target deduplication server.

[0042] S104. The target deduplication server performs deduplication and merging processing on the data to be processed.

[0043] Among them, the deduplication server is a server used for data deduplication and merging. The deduplication server can compare the data to be processed with the sample data stored on the deduplication server, delete the data that is duplicated in the data to be processed and the sample data, merge the data that is not duplicated in the data to be processed and the sample data with the sample data, update to obtain new sample data and store it.

[0044] In practical applications, a data deduplication and merging system includes multiple deduplication servers. When these servers start, an election module elects a master deduplication server, and the other servers become slave deduplication servers. For example, if the deduplication and merging system includes deduplication server 1, deduplication server 2, and deduplication server 3, and the election module elects deduplication server 2 as the master, then deduplication server 1 and deduplication server 2 become slave deduplication servers.

[0045] It should be noted that the election of the primary deduplication server only occurs when the deduplication server starts up. Once the primary deduplication server is determined, subsequent data deduplication and merging processes do not involve another primary deduplication server election.

[0046] In this embodiment, the priority of the deduplication server can be used to prioritize the deduplication and merging of data to be processed. The higher the priority of the deduplication server, the more timely the data processing. For example, when multiple deduplication servers have the same data processing speed, the priority of the deduplication server can be represented by the queue length of the data to be processed. The length of the queue is equal to the number of data items to be processed. The shorter the queue length, the higher the priority of the deduplication server. For example, if the queue length of deduplication server 1 is 5, the queue length of deduplication server 2 is 6, and the queue length of deduplication server 3 is 2, then deduplication server 3 has the shortest queue length and therefore the highest priority.

[0047] It should be noted that both the primary and secondary deduplication servers can serve as target deduplication servers. The primary deduplication service obtains the priority of each secondary deduplication server and the priority of the primary deduplication server, compares the priorities of multiple deduplication servers, and selects the deduplication server with the highest priority as the target deduplication server. Referring to the example above, deduplication server 2 is the primary deduplication server. Deduplication server 2 obtains the queue length of the data to be processed from deduplication servers 2, 1, and 3. It compares the queue lengths of the data to be processed from multiple deduplication servers, determines that deduplication server 3 has the shortest queue length, and selects deduplication server 3 as the target deduplication server.

[0048] In practical applications, both the primary and secondary deduplication servers may receive data to be processed. After receiving the data, the secondary server transmits it to the primary server. The primary server then transmits the data to the target server, which performs deduplication and merging on the data. Essentially, the primary server allocates data to higher-priority target servers, ensuring timely processing of the data.

[0049] Optionally, after receiving the data to be processed, the main deduplication server determines the priority of multiple deduplication servers, selects the deduplication server with the highest priority as the target deduplication server, and transmits the data to be processed to the target deduplication server for deduplication and merging processing.

[0050] Optionally, the priority of the deduplication server can be determined based on the data processing speed of the deduplication server and the length of the queue of data to be processed. In one possible implementation, S102 includes:

[0051] The primary deduplication server obtains heartbeat data from each secondary deduplication server and status data from the primary deduplication server.

[0052] The primary deduplication server calculates the priority coefficient of each deduplication server based on the heartbeat data of the secondary deduplication servers and the status data of the primary deduplication server.

[0053] The primary deduplication server selects the deduplication server with the highest priority coefficient among multiple deduplication servers as the target deduplication server.

[0054] The heartbeat data mechanism serves as a status synchronization tool between the secondary deduplication server and the primary deduplication server. A heartbeat period can be set, and upon arrival of the scheduled period, the secondary deduplication server sends heartbeat data to the primary server. The heartbeat data includes the data processing speed of the secondary deduplication server and the length of the pending data queue. The status data includes the data processing speed of the primary deduplication server and the length of the pending data queue. In the example above, the length of the pending data queue represents the number of data items to be processed. It can be understood that by obtaining the heartbeat data from each secondary deduplication server and the status data from the primary server, the primary deduplication server can determine the data processing speed and the number of pending data items across multiple deduplication servers.

[0055] For example, the primary deduplication server is deduplication server 2. Deduplication server 2 obtains heartbeat data 1 from deduplication server 1 and heartbeat data 3 from deduplication server 3. Deduplication server 2 also obtains status data 2 from deduplication server 2. Specifically, based on heartbeat data 1, deduplication server 2 determines that the data processing speed of deduplication server 1 is 5 data entries per second, and the number of data entries to be processed is 20. Based on status data 2, deduplication server 2 determines that its data processing speed is 10 data entries per second, and the number of data entries to be processed is 25. Based on heartbeat data 3, deduplication server 2 determines that the data processing speed of deduplication server 3 is 4 data entries per second, and the number of data entries to be processed is 20.

[0056] In this embodiment, the priority coefficient of a deduplication server is the ratio of its data processing speed to the length of its pending data queue. Specifically, the deduplication server calculates its priority coefficient based on the heartbeat data from the slave deduplication server and the status data from the master deduplication server. Using the example above, calculating the ratio of deduplication server 1's data processing speed to its pending data queue yields a priority coefficient of 0.25; calculating the ratio of deduplication server 2's data processing speed to its pending data queue yields a priority coefficient of 0.4; and calculating the ratio of deduplication server 3's data processing speed to its pending data queue yields a priority coefficient of 0.2.

[0057] It can be understood that the ratio of the length of the data queue to the data processing speed of the deduplication server is the time required for the deduplication server to process the data. Since the priority of the deduplication server can be used to prioritize the deduplication and merging of data, the shorter the time required for the deduplication server to process the data queue, the higher its priority. Correspondingly, the priority coefficient is the reciprocal of the time required for the deduplication server to process the data queue; the larger the priority coefficient, the higher the priority of the deduplication server. Specifically, the primary deduplication server selects the deduplication server with the highest priority coefficient among multiple deduplication servers as the target deduplication server. Referring to the example above, deduplication server 2 has the highest priority coefficient, so deduplication server 2 is selected as the target deduplication server.

[0058] Optionally, in one example, when a heartbeat cycle arrives, the primary deduplication server, based on the received heartbeat data, designates the secondary deduplication server that sent the heartbeat data and the primary deduplication server as candidate deduplication servers. After receiving the data to be processed, the primary deduplication server determines the highest priority deduplication server from the candidate servers as the target deduplication server and transmits the data to be processed to the target deduplication server. It can be understood that if the primary deduplication server does not receive heartbeat data from the secondary deduplication server when the heartbeat cycle arrives, the primary deduplication server determines that the secondary deduplication server is temporarily out of contact and does not add it to the candidate deduplication servers.

[0059] In this embodiment, the master deduplication server obtains the heartbeat data of each slave deduplication server and the status data of the master deduplication server. Based on the heartbeat data of the slave deduplication servers and the status data of the master deduplication server, the priority coefficient of each deduplication server is calculated. The higher the priority coefficient, the higher the priority of the deduplication server. The deduplication server with the highest priority coefficient is selected as the target deduplication server. This can accurately determine the deduplication server with the highest priority. Applying the target deduplication server to perform deduplication and merging processing on the data to be processed can achieve timely processing of the data to be processed and improve the efficiency of data deduplication and merging.

[0060] In practical applications, when multiple deduplication servers start up, a primary deduplication server election will take place. In one example, S101 includes:

[0061] An election process is performed for each deduplication server until a master deduplication server is obtained. The election process is as follows: a deduplication server sends an election request to other deduplication servers. If the number of election votes returned by other deduplication servers to the deduplication server is greater than a first threshold, then the deduplication server is designated as the master deduplication server; otherwise, the deduplication server is designated as the slave deduplication server.

[0062] The election request is used by a deduplication server to request to become the primary deduplication server from other deduplication servers. For example, deduplication server A sends a deduplication request to other deduplication servers asking, "Do you agree to let deduplication server A become the primary deduplication server?". Upon receiving the election request, each deduplication server returns an election response to the server that sent the request; the response can be either "agree" or "disagree". If a deduplication server receives the election request but does not return an election response, it indicates that it disagrees with the election. The number of votes received by the deduplication server represents the number of "agree" votes received.

[0063] In practical applications, the election of the primary deduplication server can be based on a certain order, such as the startup time order of multiple deduplication servers, the size of the deduplication server's numerical identifier, or the size of the last digit of the deduplication server's data identifier. The election of the deduplication server can be random and is not limited here.

[0064] Specifically, the deduplication server sends election requests to other deduplication servers. If the number of votes returned by other deduplication servers to a deduplication server is greater than a first threshold, then that deduplication server is designated as the primary deduplication server; otherwise, it is designated as a secondary deduplication server. For example, a data deduplication and merging system includes deduplication servers A, B, C, and D, with a first threshold of 1. Deduplication server A sends election requests to deduplication servers B, C, and D. Deduplication server A receives election responses from deduplication servers B and C, both of which agree to the election. It does not receive an election response from deduplication server D, determining the number of votes to be 2, which is greater than the first threshold. Therefore, deduplication server A is designated as the primary deduplication server, and deduplication servers B, C, and D are designated as secondary deduplication servers.

[0065] To facilitate understanding the election of the primary deduplication server, combined with Figure 2 An example is provided to illustrate the election of a primary deduplication server. Figure 2 This is a schematic diagram of the election process of a primary deduplication server provided in Embodiment 1 of this application, as shown below. Figure 2As shown, the data deduplication and merging system includes deduplication server 20, deduplication server 21, deduplication server 22, and deduplication server 23. Deduplication server 20 sends election requests to deduplication servers 21, 22, and 23 respectively. The election requests are used by deduplication server 20 to request to become the primary deduplication server from the other deduplication servers. Deduplication servers 21, 22, and 23 send election responses to the deduplication servers based on the received election requests. The election responses may agree to make deduplication server 20 the primary deduplication server, or they may disagree. If the number of votes returned by other deduplication servers received by deduplication server 20 is greater than a first threshold, then deduplication server 20 is made the primary deduplication server; otherwise, deduplication server 20 is made a secondary deduplication server.

[0066] In this embodiment, the deduplication server sends an election request to other deduplication servers. If the number of election votes returned by other deduplication servers is greater than a first threshold, then the deduplication server is designated as the master deduplication server; otherwise, the deduplication server is designated as the slave deduplication server. The master deduplication server is determined through election.

[0067] In the data deduplication and merging method provided in this application, an election module selects a master deduplication server from multiple deduplication servers, and designates the other deduplication servers besides the master server as slave deduplication servers. The master server determines the priority of the multiple deduplication servers, designates the deduplication server with the highest priority as the target deduplication server, obtains the data to be processed, and transmits the data to the target deduplication server. The target deduplication server then performs deduplication and merging processing on the data to be processed. In the data deduplication and merging process, this application's embodiment determines the master deduplication server from multiple deduplication servers through election, and the master server determines the priority of the multiple deduplication servers. The higher the priority of a deduplication server, the more timely it processes the data. By transmitting the data to be processed to the highest priority deduplication server, timely processing of the data to be processed can be achieved, improving the efficiency of data deduplication and merging.

[0068] Example 2

[0069] Figure 3 This is a flowchart illustrating a data deduplication and merging method provided in Embodiment 2 of this application, as shown below. Figure 3 As shown, the method includes the following steps:

[0070] S301, The election module obtains the master deduplication server from multiple deduplication servers through election, and regards the other deduplication servers besides the master deduplication server as slave deduplication servers.

[0071] S302. The main deduplication server determines the priority of multiple deduplication servers and selects the deduplication server with the highest priority as the target deduplication server.

[0072] S303. The main deduplication server obtains the data to be processed and transmits the data to the target deduplication server.

[0073] S304. The target deduplication server performs deduplication merging processing on the data to be processed and the full sample data of the current target deduplication server to obtain the full sample data and incremental sample data of the target deduplication server after deduplication merging processing.

[0074] S305. The target deduplication server sends the incremental sample data of the current target deduplication server to the incremental server.

[0075] S306. The incremental server saves the received incremental sample data from the target deduplication server as the full sample data of the current incremental server, and transmits the full sample data of the incremental server to the full server and multiple deduplication servers.

[0076] S307. The full server updates its full sample data based on the full sample data received from the incremental server as the current incremental sample data of the full server.

[0077] S308. Multiple deduplication servers update their full sample data based on the received full sample data from the incremental server as the current incremental sample data for that deduplication server.

[0078] S301 to S303 in this embodiment are the same as S101 to S103 in the above embodiment. Please refer to the above embodiment for details. They will not be repeated in this embodiment.

[0079] In this embodiment, the full sample data of the current target deduplication server refers to the sample data stored locally on the target deduplication server. The incremental sample data of the target deduplication server after deduplication merging is data that is not duplicated between the data to be processed and the full sample data of the current target deduplication server; that is, data added to the data to be processed compared to the full sample data of the current target deduplication server. The full sample data of the target deduplication server after deduplication merging is the merged data of the incremental sample data of the target deduplication server after deduplication merging and the full sample data of the current target deduplication server. It can be understood that the full sample data of the target deduplication server includes the incremental sample data of the target deduplication server.

[0080] In practical applications, deduplication and merging processing involves two steps: deduplication and merging. Specifically, after the target deduplication server obtains the data to be processed, it first performs deduplication processing on the data to be processed and its full sample data. This involves comparing the data to be processed with the full sample data of the target deduplication server, removing duplicate data from the data to be processed, and using the data from the data to be processed that is not duplicated as incremental sample data for the target deduplication server. After obtaining the incremental sample data, the target deduplication server merges this incremental sample data with its full sample data to obtain the deduplicated and merged full sample data for the target deduplication server. In essence, after deduplication and merging, the sample data stored locally on the target deduplication server is updated to reflect the deduplicated and merged full sample data.

[0081] Specifically, in S304, the target deduplication server performs deduplication merging processing on the data to be processed and the full sample data of the current target deduplication server to obtain the full sample data and incremental sample data of the target deduplication server after deduplication merging. For example, if the full sample data of the current target deduplication server is a, b, c, d, and the data to be processed is a, e, the data to be processed a and e are compared with the full sample data a, b, c, d of the current target deduplication server. Data 'a' that is duplicated in the full sample data of the current target deduplication server is removed, and data 'e' that is not duplicated in the full sample data of the current target deduplication server is used as the incremental sample data of the target deduplication server. The incremental sample data 'e' of the target deduplication server is then merged with the full sample data a, b, c, d of the current target deduplication server to obtain the full sample data a, b, c, d, e of the target deduplication server after deduplication merging.

[0082] It can be understood that the full sample data of the incremental server is the sample data stored locally on the incremental server. In practical applications, after receiving the incremental sample data from the target deduplication server, the incremental server updates its locally stored sample data with the incremental sample data received from the target deduplication server.

[0083] In practical applications, the primary deduplication server may select multiple target deduplication servers to process multiple datasets simultaneously. Optionally, after receiving incremental sample data from the target deduplication servers, the incremental server performs deduplication processing on its current full sample data and the received incremental sample data from the target deduplication servers. Specifically, it saves the data in the received incremental sample data from the target deduplication servers that does not overlap with the current full sample data as the full sample data of the incremental data. For example, if the current full sample data of the incremental server is 'e', ​​and the incremental sample data received from the target deduplication servers are 'e' and 'f', the server performs deduplication processing on the current full sample data and the received incremental sample data from the target deduplication servers, saving the data in the received incremental sample data from the target deduplication servers that does not overlap with the current full sample data as the full sample data of the incremental data.

[0084] Optionally, after receiving the full sample data from the incremental server, the full server uses the incremental server's full sample data as its incremental sample data, merges the incremental sample data of the full server with the current full sample data of the full server, and updates the full sample data of the full server.

[0085] Specifically, in S307, the full server updates its full sample data by using the received full sample data from the incremental server as its current incremental sample data. For example, if the full server receives full sample data 'e' from the incremental server, and the current full sample data is 'a', 'b', 'c', and 'd', it uses the incremental server's full sample data as its incremental sample data and merges it with the current full sample data to obtain the full sample data 'a', 'b', 'c', 'd', and 'e' for the full server.

[0086] Optionally, after receiving the full sample data from the incremental server, multiple deduplication servers use the full sample data from the incremental server as the current incremental sample data for the deduplication server, perform deduplication merging processing on the current incremental sample data and the current full sample data for the deduplication server, and update the full sample data of multiple deduplication servers.

[0087] Specifically, in S308, multiple deduplication servers update their full sample data based on the received full sample data from the incremental server. For example, if multiple deduplication servers receive full sample data e and f from the incremental server, and the current full sample data of deduplication server 1 is a, b, c, d, and e, the full sample data from the incremental server is used as the incremental sample data for the full server. The incremental sample data from the full server is then merged with the current full sample data of the full server to obtain the full sample data a, b, c, d, e, and f for the full server. Similarly, the current full sample data of deduplication server 2 is a, b, c, d, e, and f. The full sample data from the incremental server is then used as the incremental sample data for the full server. The incremental sample data of the full server is merged with the full sample data of the current full server to obtain the full sample data a, b, c, d, e, f of the full server; the full sample data of the current deduplication server 3 is a, b, c, d. The full sample data of the incremental server is used as the incremental sample data of the full server. Since the full sample data of the current deduplication server 3 and the incremental sample data of the full server do not overlap, the incremental sample data of the full server is merged with the full sample data of the current full server to obtain the full sample data a, b, c, d, e, f of the full server.

[0088] In this embodiment, after the target deduplication server performs deduplication merging processing, it sends the incremental sample data of the target deduplication server to the incremental server. The incremental server saves the received incremental sample data of the target deduplication server as the full sample data of the current incremental server, and transmits the full sample data of the incremental server to the full server and multiple deduplication servers. The full server and multiple deduplication servers update their full sample data according to the full sample data received from the incremental server. Compared with the method of multiple deduplication servers obtaining the full sample data of the full server to update their full sample data, this avoids large-scale data interaction between multiple deduplication servers and the full server, saving resources.

[0089] Optionally, to ensure the consistency and accuracy of the full sample data across multiple deduplication servers, the multiple deduplication servers periodically update the full sample data. In one possible implementation, the update process of the full sample data across the multiple deduplication servers further includes:

[0090] When the update cycle arrives, multiple deduplication servers update the full sample data of multiple deduplication servers based on the full sample data of the full server and the full sample data of the incremental server.

[0091] Based on the above example, the full sample data of the incremental server is obtained after the target deduplication server completes the deduplication and merging. The incremental server sends the full sample data of the incremental server to the full server. The full server uses the full sample data of the incremental server as the incremental sample data of the full server, and merges the incremental sample data of the full server with the current full sample data of the full server to update the full sample data of the full server.

[0092] It is understandable that when the full sample data of the full server is updated, the full sample data of the full server includes the full sample data of the incremental server; when the full sample data of the full server is not updated, the full sample data of the full server does not include the full sample data of the incremental server. Therefore, when the update cycle arrives, multiple deduplication servers, based on the full sample data of the full server and the full sample data of the incremental server, merge the obtained full sample data of the full server and the full sample data of the incremental server to update the full sample data of multiple deduplication servers, which can ensure the consistency and accuracy of the full sample data of multiple deduplication servers.

[0093] It should be noted that this implementation differs from the method in S308 where multiple deduplication servers update the full sample data. In S308, after the full sample data of the incremental server is updated, the multiple deduplication servers only obtain the full sample data of the incremental server. The multiple deduplication servers then merge the full sample data of the incremental server with the full sample data currently stored locally on the deduplication server to update the full sample data of the multiple deduplication servers. In this implementation, when the update cycle arrives, the multiple deduplication servers obtain the full sample data of the full server and the full sample data of the incremental server, and merge the obtained full sample data of the full server and the full sample data of the incremental server to remove duplicates. The result of the deduplication merging is used as the updated full sample data of the multiple deduplication servers.

[0094] In this embodiment, when the update cycle arrives, multiple deduplication servers update the full sample data of multiple deduplication servers based on the full sample data of the full server and the full sample data of the incremental server. This ensures the consistency and accuracy of the full sample data of multiple deduplication servers, thereby improving the accuracy of deduplication merging of the target deduplication server.

[0095] In the data deduplication and merging method provided in this application, the election module obtains a master deduplication server from multiple deduplication servers through election; the master deduplication server determines the priority of the multiple deduplication servers, selects the deduplication server with the highest priority as the target deduplication server, obtains the data to be processed, and transmits the data to be processed to the target deduplication server; the target deduplication server performs deduplication and merging processing on the data to be processed; the target deduplication server sends its incremental sample data to the incremental server; the incremental server saves the received incremental sample data from the target deduplication server as its full sample data and transmits the full sample data to the full server and multiple deduplication servers; the full server updates its full sample data based on the received full sample data from the incremental server as its current incremental sample data; multiple deduplication servers update their full sample data based on the received full sample data from the incremental server as their current incremental sample data. In the data deduplication and merging process of this application embodiment, after the target deduplication server performs the deduplication and merging process, the full server and multiple deduplication servers update the full sample data of the full server and multiple deduplication servers according to the full sample data received from the incremental server, thereby avoiding large data interaction between multiple deduplication servers and the full server and saving resources.

[0096] Example 3

[0097] Figure 4 This is a schematic diagram of a data deduplication and merging system provided in Embodiment 3 of this application. An example is a data deduplication and merging system including three deduplication servers. Figure 4 As shown, the device includes: multiple deduplication servers 40 and an election module 41;

[0098] The election module 41 is used to obtain a master deduplication server from multiple deduplication servers 40 through election, and to designate the other deduplication servers among the multiple deduplication servers 40, excluding the master deduplication server, as slave deduplication servers.

[0099] The primary deduplication server is used to determine the priority of multiple deduplication servers 40, select the deduplication server with the highest priority as the target deduplication server, obtain the data to be processed, and transfer the data to be processed to the target deduplication server.

[0100] The target deduplication server is used to perform deduplication and merging processing on the data to be processed.

[0101] Among them, the deduplication server is a server used for data deduplication and merging. The deduplication server can compare the data to be processed with the sample data stored on the deduplication server, delete the data that is duplicated in the data to be processed and the sample data, merge the data that is not duplicated in the data to be processed and the sample data with the sample data, update to obtain new sample data and store it.

[0102] In practical applications, the data deduplication and merging system includes multiple deduplication servers 40. When the multiple deduplication servers 40 are started, the election module elects a master deduplication server and uses the other deduplication servers among the multiple deduplication servers 40, excluding the master deduplication server, as slave deduplication servers.

[0103] It should be noted that the election of the primary deduplication server only occurs when the deduplication server starts up. Once the primary deduplication server is determined, subsequent data deduplication and merging processes do not involve another primary deduplication server election.

[0104] In this embodiment, the priority of the deduplication server can be used to prioritize the deduplication and merging of data to be processed. The higher the priority of the deduplication server, the more timely the data processing. For example, when multiple deduplication servers 40 have the same data processing speed, the priority of the deduplication server can be represented by the queue length of the data to be processed. The length of the queue is equal to the number of data items to be processed. The shorter the queue length, the higher the priority of the deduplication server.

[0105] It should be noted that both the primary and secondary deduplication servers can serve as target deduplication servers. The primary deduplication service obtains the priority of each secondary deduplication server and the priority of the primary deduplication server, compares the priorities of multiple deduplication servers, and selects the deduplication server with the highest priority as the target deduplication server.

[0106] In practical applications, both the primary and secondary deduplication servers may receive data to be processed. After receiving the data, the secondary server transmits it to the primary server. The primary server then transmits the data to the target server, which performs deduplication and merging on the data. Essentially, the primary server allocates data to higher-priority target servers, ensuring timely processing of the data.

[0107] Optionally, after receiving the data to be processed, the main deduplication server determines the priority of multiple deduplication servers 40, selects the deduplication server with the highest priority as the target deduplication server, and transmits the data to be processed to the target deduplication server for deduplication and merging processing.

[0108] Optionally, in one possible implementation, the primary deduplication server is used to determine the priority of multiple deduplication servers 40, and when the deduplication server with the highest priority is selected as the target deduplication server, it is specifically used for:

[0109] Obtain the heartbeat data of each slave deduplication server and the status data of the master deduplication server;

[0110] The priority coefficient of each deduplication server is calculated based on the heartbeat data from the primary deduplication server and the status data from the master deduplication server.

[0111] The deduplication server with the highest priority coefficient among the 40 deduplication servers is selected as the target deduplication server.

[0112] The heartbeat data mechanism serves as a status synchronization tool between the primary and secondary deduplication servers. A heartbeat period can be set, and upon arrival of the scheduled period, each secondary deduplication server sends heartbeat data to the primary server. The heartbeat data includes the data processing speed of the secondary server and the length of the queue of data to be processed. The status data includes the data processing speed of the primary deduplication server and the length of the queue of data to be processed. In the example above, the length of the queue of data to be processed represents the number of data items to be processed. It can be understood that by obtaining the heartbeat data from each secondary server and the status data from the primary server, the primary deduplication server can determine the data processing speed and the number of data items to be processed across multiple deduplication servers.

[0113] In this embodiment, the priority coefficient of a deduplication server is the ratio of its data processing speed to the length of its pending data queue. Specifically, the deduplication server calculates the priority coefficient of each server based on the heartbeat data from slave servers and the status data from the master server.

[0114] It can be understood that the ratio of the length of the queue of data to be processed by the deduplication server to the data processing speed is the time required for the deduplication server to process the data. Since the priority of the deduplication server can be used to prioritize the deduplication and merging of data, the shorter the time required for the deduplication server to process the queue of data, the higher the priority of the deduplication server. Correspondingly, the priority coefficient is the reciprocal of the time required for the deduplication server to process the queue of data; the larger the priority coefficient, the higher the priority of the deduplication server.

[0115] Optionally, in one example, when a heartbeat cycle arrives, the primary deduplication server, based on the received heartbeat data, designates the secondary deduplication server that sent the heartbeat data and the primary deduplication server as candidate deduplication servers. After receiving the data to be processed, the primary deduplication server determines the highest priority deduplication server from the candidate servers as the target deduplication server and transmits the data to be processed to the target deduplication server. It can be understood that if the primary deduplication server does not receive heartbeat data from the secondary deduplication server when the heartbeat cycle arrives, the primary deduplication server determines that the secondary deduplication server is temporarily out of contact and does not add it to the candidate deduplication servers.

[0116] In this embodiment, the master deduplication server obtains the heartbeat data of each slave deduplication server and the status data of the master deduplication server. Based on the heartbeat data of the slave deduplication servers and the status data of the master deduplication server, the priority coefficient of each deduplication server is calculated. The higher the priority coefficient, the higher the priority of the deduplication server. The deduplication server with the highest priority coefficient is selected as the target deduplication server. This can accurately determine the deduplication server with the highest priority. Applying the target deduplication server to perform deduplication and merging processing on the data to be processed can achieve timely processing of the data to be processed and improve the efficiency of data deduplication and merging.

[0117] In addition, in one possible implementation, the election module 41 is specifically used to: perform election processing for each deduplication server until a master deduplication server is obtained; wherein, the election processing is: the deduplication server sends an election request to other deduplication servers, and if the number of election votes returned by other deduplication servers received by the deduplication server is greater than a first threshold, then the deduplication server is designated as the master deduplication server; otherwise, the deduplication server is designated as the slave deduplication server.

[0118] The election request is used by the deduplication server to request to become the primary deduplication server from other deduplication servers. It can be understood that after receiving an election request, the deduplication server returns an election response to the server that sent the request; the response can be either in favor of the election or against it. If a deduplication server receives an election request but does not return an election response, it indicates that it does not agree to the election. The election vote count represents the number of votes received by the deduplication server indicating agreement to the election.

[0119] In practical applications, the election of the primary deduplication server can be based on a certain order, such as the startup time order of multiple deduplication servers 40, the size of the deduplication server's numerical identifier, and the size of the last digit of the deduplication server's data identifier. The election of the deduplication server can be random and is not limited here.

[0120] In this embodiment, the deduplication server sends an election request to other deduplication servers. If the number of election votes returned by other deduplication servers is greater than a first threshold, then the deduplication server is designated as the master deduplication server; otherwise, the deduplication server is designated as the slave deduplication server. The master deduplication server is determined through election.

[0121] Optionally, in one possible implementation, the target deduplication server is specifically used to: perform deduplication merging processing on the data to be processed and the full sample data of the current target deduplication server to obtain the full sample data and incremental sample data of the target deduplication server after deduplication merging processing.

[0122] The full sample data of the current target deduplication server refers to the sample data stored locally on the target deduplication server. The incremental sample data of the target deduplication server after deduplication merging is the data to be processed that is not duplicated in the full sample data of the current target deduplication server; that is, the data added to the data to be processed compared to the full sample data of the current target deduplication server. The full sample data of the target deduplication server after deduplication merging is the merged data of the incremental sample data of the target deduplication server after deduplication merging and the full sample data of the current target deduplication server. It can be understood that the full sample data of the target deduplication server includes the incremental sample data of the target deduplication server.

[0123] In practical applications, deduplication and merging processing involves two steps: deduplication and merging. Specifically, after the target deduplication server obtains the data to be processed, it first performs deduplication processing on the data to be processed and its full sample data. This involves comparing the data to be processed with the full sample data of the target deduplication server, removing duplicate data from the data to be processed, and using the data from the data to be processed that is not duplicated as incremental sample data for the target deduplication server. After obtaining the incremental sample data, the target deduplication server merges this incremental sample data with its full sample data to obtain the deduplicated and merged full sample data for the target deduplication server. In essence, after deduplication and merging, the sample data stored locally on the target deduplication server is updated to reflect the deduplicated and merged full sample data.

[0124] Optionally, in one possible implementation, the system further includes: an incremental server and a full server;

[0125] The target deduplication server is also used to send incremental sample data from the current target deduplication server to the incremental server;

[0126] The incremental server is used to save the received incremental sample data from the target deduplication server as the full sample data of the current incremental server, and to transmit the full sample data of the incremental server to the full server and multiple deduplication servers 40.

[0127] The full server is used to update the full sample data of the full server by using the full sample data of the received incremental server as the incremental sample data of the current full server.

[0128] Multiple deduplication servers 40 are used to update the full sample data of the deduplication server based on the full sample data of the received incremental server as the current incremental sample data of the deduplication server.

[0129] It can be understood that the full sample data of the incremental server is the sample data stored locally on the incremental server. In practical applications, after receiving the incremental sample data from the target deduplication server, the incremental server updates its locally stored sample data with the incremental sample data received from the target deduplication server.

[0130] In practical applications, the primary deduplication server may select multiple target deduplication servers to process multiple datasets simultaneously. Optionally, after receiving incremental sample data from the target deduplication servers, the incremental server performs deduplication processing by comparing the current incremental server's full sample data with the received incremental sample data from the target deduplication servers. Specifically, it saves the data in the received incremental sample data from the target deduplication servers that does not overlap with the current incremental server's full sample data as the full sample data of the incremental data.

[0131] Optionally, after receiving the full sample data from the incremental server, the full server uses the incremental server's full sample data as its incremental sample data, merges the incremental sample data of the full server with the current full sample data of the full server, and updates the full sample data of the full server.

[0132] Optionally, after receiving the full sample data from the incremental server, the multiple deduplication servers 40 use the full sample data from the incremental server as the current incremental sample data of the deduplication server, perform deduplication merging processing on the current incremental sample data of the deduplication server and the current full sample data of the deduplication server, and update the full sample data of the multiple deduplication servers 40.

[0133] In this embodiment, after the target deduplication server performs deduplication merging processing, it sends the incremental sample data of the target deduplication server to the incremental server. The incremental server saves the received incremental sample data of the target deduplication server as the full sample data of the current incremental server, and transmits the full sample data of the incremental server to the full server and multiple deduplication servers. The full server and multiple deduplication servers update their full sample data according to the full sample data received from the incremental server. Compared with the method of multiple deduplication servers obtaining the full sample data of the full server to update their full sample data, this avoids large data interaction between multiple deduplication servers and the full server, saving resources.

[0134] Optionally, in one example, the multiple deduplication servers 40 are also used to: update the full sample data of the multiple deduplication servers 40 based on the full sample data of the full server and the full sample data of the incremental server when the update cycle arrives.

[0135] Based on the above example, the full sample data of the incremental server is obtained after the target deduplication server completes the deduplication and merging. The incremental server sends the full sample data of the incremental server to the full server. The full server uses the full sample data of the incremental server as the incremental sample data of the full server, and merges the incremental sample data of the full server with the current full sample data of the full server to update the full sample data of the full server.

[0136] It is understandable that when the full sample data of the full server is updated, the full sample data of the full server includes the full sample data of the incremental server; when the full sample data of the full server is not updated, the full sample data of the full server does not include the full sample data of the incremental server. Therefore, when the update cycle arrives, multiple deduplication servers 40, based on the full sample data of the full server and the full sample data of the incremental server, perform deduplication and merge the obtained full sample data of the full server and the incremental server, and update the full sample data of multiple deduplication servers 40, which can ensure the consistency and accuracy of the full sample data of multiple deduplication servers 40.

[0137] In this embodiment, when the update cycle arrives, multiple deduplication servers update the full sample data of multiple deduplication servers based on the full sample data of the full server and the full sample data of the incremental server. This ensures the consistency and accuracy of the full sample data of multiple deduplication servers, thereby improving the accuracy of deduplication merging of the target deduplication server.

[0138] In the data deduplication and merging system provided in this application, the election module elects a master deduplication server from multiple deduplication servers, and designates the other deduplication servers besides the master server as slave deduplication servers. The master deduplication server determines the priority of the multiple deduplication servers, designates the deduplication server with the highest priority as the target deduplication server, obtains the data to be processed, and transmits the data to the target deduplication server. The target deduplication server then performs deduplication and merging processing on the data to be processed. In the data deduplication and merging process, this application's embodiment determines the master deduplication server from multiple deduplication servers through election, and the master deduplication server determines the priority of the multiple deduplication servers. The higher the priority of a deduplication server, the more timely it processes the data. By transmitting the data to be processed to the highest priority deduplication server, timely processing of the data to be processed can be achieved, improving the efficiency of data deduplication and merging.

[0139] Example 5

[0140] Figure 5 This is a schematic diagram of the structure of the electronic device provided in Embodiment 5 of this application, as shown below. Figure 5 As shown, the electronic device includes:

[0141] The processor 51 and the main control device also include a memory 52; it may also include a communication interface 53 and a bus 54. The processor 51, memory 52, and communication interface 53 can communicate with each other via the bus 54. The communication interface 53 can be used for information transmission. The processor 51 can call logical instructions stored in the memory 52 to execute the methods of the above embodiments.

[0142] Furthermore, the logic instructions in the aforementioned memory 52 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0143] The memory 52, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 51 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 52, thereby implementing the methods in the above-described method embodiments.

[0144] The memory 52 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 52 may include high-speed random access memory and may also include non-volatile memory.

[0145] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods in any of the embodiments. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0146] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0147] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data deduplication and merging method, characterized in that, Applied to a data deduplication and merging system, the data deduplication and merging system includes multiple deduplication servers and an election module, the method includes: The election module selects a master deduplication server from multiple deduplication servers through election, and designates the other deduplication servers besides the master deduplication server as slave deduplication servers. The master deduplication server acquires heartbeat data from each slave deduplication server and status data from the master deduplication server; wherein, the heartbeat data includes the data processing speed of the slave deduplication server and the length of the data queue to be processed; the status data includes the data processing speed of the master deduplication server and the length of the data queue to be processed. The master deduplication server calculates the priority coefficient of each deduplication server based on the heartbeat data of the slave deduplication servers and the status data of the master deduplication server; the priority coefficient of the deduplication server is the ratio of the data processing speed of the deduplication server to the length of the data queue to be processed of the deduplication server. The main deduplication server selects the deduplication server with the highest priority coefficient among the multiple deduplication servers as the target deduplication server, obtains the data to be processed, and transmits the data to be processed to the target deduplication server. The target deduplication server performs deduplication merging processing on the data to be processed and the full sample data of the current target deduplication server to obtain the full sample data and incremental sample data of the target deduplication server after deduplication merging processing. The target deduplication server sends the incremental sample data of the current target deduplication server to the incremental server; The incremental server saves the received incremental sample data from the target deduplication server as the full sample data of the current incremental server, and transmits the full sample data of the incremental server to the full server and the multiple deduplication servers. The full-sample server updates its full-sample data by using the full-sample data received from the incremental server as the current incremental sample data of the full-sample server; the multiple deduplication servers update their full-sample data by using the full-sample data received from the incremental server as the current incremental sample data of the deduplication server.

2. The method according to claim 1, characterized in that, The method further includes: When the update cycle arrives, the multiple deduplication servers update the full sample data of the multiple deduplication servers based on the full sample data of the full server and the full sample data of the incremental server.

3. The method according to claim 1 or 2, characterized in that, The election module selects a master deduplication server from multiple deduplication servers through election, and designates the other deduplication servers besides the master server as slave deduplication servers, including: An election process is performed for each deduplication server until a master deduplication server is obtained. The election process is as follows: the deduplication server sends an election request to other deduplication servers. If the number of election votes returned by the other deduplication servers to the deduplication server is greater than a first threshold, then the deduplication server is designated as the master deduplication server; otherwise, the deduplication server is designated as the slave deduplication server.

4. A data deduplication and merging system, characterized in that, The system includes: multiple deduplication servers and an election module; The election module is used to obtain a master deduplication server from multiple deduplication servers through election, and to designate the other deduplication servers besides the master deduplication server as slave deduplication servers. The master deduplication server is used to acquire heartbeat data from each slave deduplication server and status data from the master deduplication server. The heartbeat data includes the data processing speed of each slave deduplication server and the length of its pending data queue. The status data includes the data processing speed of the master deduplication server and the length of its pending data queue. A priority coefficient is calculated for each deduplication server based on its heartbeat data and status data. The priority coefficient is the ratio of the server's data processing speed to the length of its pending data queue. The server with the highest priority coefficient among the multiple deduplication servers is selected as the target deduplication server, and the pending data is acquired and transmitted to the target deduplication server. The target deduplication server is used to perform deduplication merging processing on the data to be processed and the full sample data of the current target deduplication server to obtain the full sample data and incremental sample data of the target deduplication server after deduplication merging processing. The system also includes: an incremental server and a full server; The target deduplication server is also used to send the incremental sample data of the current target deduplication server to the incremental server; The incremental server is used to save the received incremental sample data from the target deduplication server as the full sample data of the current incremental server, and to transmit the full sample data of the incremental server to the full server and the multiple deduplication servers. The full server is used to update the full sample data of the full server by using the full sample data of the received incremental server as the current incremental sample data of the full server. The plurality of deduplication servers are used to update the full sample data of the deduplication server based on the full sample data received from the incremental server as the current incremental sample data of the deduplication server.

5. The system according to claim 4, characterized in that, The multiple deduplication servers are also used for: When the update cycle arrives, the full sample data of the multiple deduplication servers is updated based on the full sample data of the full server and the full sample data of the incremental server.

6. The system according to claim 4 or 5, characterized in that, The election module is specifically used for: An election process is performed for each deduplication server until a master deduplication server is obtained. The election process is as follows: the deduplication server sends an election request to other deduplication servers. If the number of election votes returned by the other deduplication servers to the deduplication server is greater than a first threshold, then the deduplication server is designated as the master deduplication server; otherwise, the deduplication server is designated as the slave deduplication server.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-3.

Images