Data comparison method, electronic device and storage medium

Through the hash ring, the relationship between data files and servers is managed, and the load balancing problem caused by server exceptions in large-scale data comparison is solved, the comparison efficiency and system stability are improved, and the task is quickly restored and efficient resource utilization is achieved.

CN118394719BActive Publication Date: 2024-11-29ZHEJIANG HUAQI INTELLIGENT TECHNOLOGY CO LTD +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410842858.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-26
Publication Date
2024-11-29
Estimated Expiration
2044-06-26

AI Technical Summary

Technical Problem

In large-scale data comparison scenarios, if any server abnormality occurs during the data comparison process in the prior art, it will lead to data load balancing problems, affecting the comparison efficiency and system stability of multiple servers.

Method used

Through the hash ring, manage the correspondence between the data file and the server, determine the first target node corresponding to the target data file, and quickly reselect the second target node when the server is abnormal, ensuring the continuity and efficiency of the comparison task and avoiding the The data is evenly distributed to multiple normal servers.

Benefits of technology

It improves the execution efficiency of data comparison tasks, reduces the stability impact of the server cluster, and ensures rapid task recovery and resource utilization in server abnormal situations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118394719B_ABST
    Figure CN118394719B_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of data comparison, and specifically to a data comparison method, electronic device and storage medium, the method comprising: responding to a comparison instruction, determining a first target node in a hash ring corresponding to a target data file to be compared; selecting a second target node from the hash ring based on the position of the first target node, and triggering a first server corresponding to the second target node to perform a comparison task corresponding to the comparison instruction; if an exception occurs during the first server's comparison of the target data file, reselecting a second target node from a plurality of second nodes in the hash ring based on the position of the first target node corresponding to the target data file, and triggering a second server of the reselected second target node to perform a comparison task corresponding to the comparison instruction. When some servers are abnormal, the impact on the execution efficiency of servers that normally perform comparison tasks can be reduced, thereby improving the stability of the system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data comparison method, an electronic device and a storage medium. Background Art

[0002] Data comparison is to use the server to compare the data in multiple data files according to preset rules. For example, count the vehicles that have passed through both point 1 and point 2 in the data file of the vehicle passing through point 1 and the data file of the vehicle passing through point 2.

[0003] For large-scale data scenarios, in order to improve the comparison efficiency of data files, the existing technology distributes data to multiple servers in a server cluster for processing, that is, first load the data in the data file into multiple servers so that multiple servers can perform data file comparison tasks at the same time. If any server is abnormal during the data comparison process, the data in the abnormal server will be evenly distributed to multiple other normal servers to achieve load balancing. This processing method will cause multiple servers to divide server resources to obtain data from the database and perform comparison tasks on the newly obtained data during the data comparison process, which will affect the execution efficiency of the comparison tasks of multiple servers. The impact range is large and the stability of the server cluster is low. Summary of the invention

[0004] The embodiments of the present application provide a data comparison method, an electronic device, and a storage medium, which utilize a hash ring to manage the correspondence between data files and servers, thereby reducing the impact on the execution efficiency of servers that normally perform comparison tasks and improving the stability of the system.

[0005] In a first aspect, an embodiment of the present application provides a data comparison method, the method comprising:

[0006] In response to the comparison instruction, determine a first target node in the hash ring corresponding to the target data file to be compared;

[0007] Selecting a second target node from the hash ring based on the position of the first target node, and triggering a first server corresponding to the second target node to execute a comparison task corresponding to the comparison instruction, wherein the hash ring includes a plurality of first nodes and a plurality of second nodes, a first node in the hash ring corresponds to a data file one-to-one, and a second node in the hash ring corresponds to a server one-to-one;

[0008] If an exception occurs during the target data file comparison process performed by the first server, a second target node is reselected from multiple second nodes of the hash ring based on the position of the first target node corresponding to the target data file, and the second server of the reselected second target node is triggered to execute the comparison task corresponding to the comparison instruction.

[0009] In the above embodiment, the target data file and each server are mapped to the nodes of the hash ring, and the hash ring is used to manage the correspondence between the target data file and the server. When an abnormality occurs in the first server corresponding to the target data file, the second server can be quickly reselected through the positional relationship between the first target node corresponding to the target data file on the hash ring and the second node corresponding to each server, so that the second server can continue to perform the comparison task in time to ensure the continuity of task execution; moreover, for the scenario where any first server is abnormal, the embodiment of the present application reselects a second server based on the positional relationship between the target data file identified on the hash ring and each server, instead of evenly distributing the data in the abnormal server to multiple other normal servers. This only affects the efficiency of one second server in performing the comparison task, and the impact range is relatively small.

[0010] In a possible implementation, determining a first target node in the hash ring corresponding to a target data file to be compared includes:

[0011] Determine first identification information of a first node in the hash ring corresponding to the target data file based on a hash value corresponding to the first identification of the target data file and the number of nodes in the hash ring;

[0012] A first target node is determined based on the first identification information.

[0013] In the above embodiment, different target data files can be mapped to different first nodes of the hash ring through the first identifier of the target data file and the number of nodes in the hash ring, so that the management node manages the target data files using the hash ring.

[0014] In a possible implementation manner, the corresponding relationship between the server and the second node is determined in the following manner:

[0015] For any one server, based on the hash value obtained from the second identifier of the any one server and the number of nodes in the hash ring, determine the second identifier information of the second node in the hash ring corresponding to the any one server;

[0016] A corresponding relationship between any one of the servers and the second node in the hash ring is established based on the second identification information.

[0017] In the above embodiment, different servers can be mapped to different second nodes of the hash ring through the second identifier of each server and the number of nodes in the hash ring, so that the management node manages the servers using the hash ring.

[0018] In a possible implementation, the method further includes:

[0019] If the distance between the second nodes corresponding to the multiple servers is less than the preset distance, the virtual machines corresponding to the multiple servers are set on the second nodes on the hash ring whose distance from the second nodes corresponding to the multiple servers is not less than the preset distance.

[0020] In the above embodiment, virtual machines are set for servers that are close to each other on the hash ring, so as to avoid a situation where a large amount of data is loaded onto the same server and the server is overloaded.

[0021] In a possible implementation, the selecting the second target node from the hash ring based on the position of the first target node includes:

[0022] A second node closest to the first target node is selected along a preset direction of the hash ring as the second target node.

[0023] In the above embodiment, the second node closest to the first target node is selected as the second target node, so that when the first server is abnormal, the server for the self-comparison task can be determined promptly and quickly, thereby improving the efficiency of data file comparison. In addition, for the scenario where multiple second target nodes need to be selected, if the preset direction is not set, multiple target data files may be closest to the same server. Therefore, selecting the second server along the preset direction can ensure the consistency of the selection process and avoid selecting the same server for multiple target data files.

[0024] In a possible implementation, the first server executing the comparison task includes multiple;

[0025] The triggering the first server corresponding to the second target node to execute the comparison task corresponding to the comparison instruction includes:

[0026] Splitting the comparison task into multiple subtasks;

[0027] For any subtask and any first server, trigger the any first server to execute the any subtask, and merge the results after receiving the comparison results of each subtask, and the number of the subtasks matches the number of the first servers.

[0028] In the above embodiment, for the scenario of large amounts of data, a distributed comparison method is set up to ensure the efficiency of the server in executing the comparison task, and after each server completes the corresponding subtask, the comparison results of each subtask are merged to obtain a complete comparison result corresponding to the comparison task.

[0029] In a possible implementation, the comparison task is performed by the server based on the generation time sequence of the data in the target data file;

[0030] The method further comprises:

[0031] During the process of the first server executing the comparison task corresponding to the comparison instruction, periodically receiving a first time range determined based on the generation time of the compared data sent by the first server;

[0032] The triggering of the reselected second server of the second target node to execute the comparison task corresponding to the comparison instruction also includes:

[0033] After determining that an exception occurs during the target data file comparison process performed by the first server, the second server is notified of the first time range received most recently, so that the second server performs a comparison task for unmatched data based on the first time range.

[0034] In the above embodiment, the comparison task is executed based on the generation time sequence of the data in the target data file, and the first server periodically receives the first time range determined based on the generation time of the compared data. This allows real-time and accurate determination of the progress of the comparison task executed by the first server, and further enables continuous comparison after an abnormality occurs in the first server, thereby avoiding repeated comparison of compared data and waste of server resources.

[0035] In a possible implementation, the comparison task is performed by the server based on the generation time sequence of the data in the target data file;

[0036] The method further comprises:

[0037] During the process of the first server executing the comparison task corresponding to the comparison instruction, if a second time range determined based on the generation time of the new data in the target data file is received, the second time range is notified to the second server so that the second server performs the comparison task for the new data based on the second time range, wherein the second time range is determined by the device to which the database storing the target data file belongs after determining that the target data file has new data.

[0038] If the device to which the database belongs determines that new data is added in the target data file, the second time range corresponding to the new data is sent to the management node, and the management node then notifies the server of the second time range, so that the server can determine the new data in the target data file, and then the server can perform a comparison task for the new data based on the second time range. The embodiment of the present application provides a comparison process for the scenario of new data in the target data file, which can ensure the integrity of the comparison of the target data file and obtain more accurate comparison results.

[0039] In a second aspect, an embodiment of the present application provides a data comparison method, which is applied to a server, and the method includes:

[0040] Receiving a comparison instruction sent by the management node, wherein the comparison instruction is sent after the management node determines a first target node in the hash ring corresponding to the target data file to be compared, and selects a second target node from the hash ring based on a position of the first target node;

[0041] Acquire the target data file corresponding to the comparison instruction from the database;

[0042] Based on the generation time sequence of the data in the target data file, executing the comparison task corresponding to the comparison instruction on the target data file to obtain the comparison result corresponding to the target data file;

[0043] The hash ring includes a plurality of first nodes and a plurality of second nodes, the first nodes in the hash ring correspond one-to-one to the data files, and the second nodes in the hash ring correspond one-to-one to the servers.

[0044] In the above embodiment, the server executes the comparison task corresponding to the comparison instruction on the target data file based on the generation time sequence of the data in the target data file, and can accurately determine the progress of its own comparison task execution, ensuring that in abnormal circumstances, the first time range representing the compared data can be fed back to the management node, so that other normal servers reselected by the management node can continue to perform the comparison task on the data that may not be correct, thereby avoiding repeated comparison of data.

[0045] In a possible implementation manner, executing the comparison task corresponding to the comparison instruction on the target data file includes:

[0046] During the execution of the comparison task corresponding to the comparison instruction, a first time range determined based on the generation time of the compared data is periodically sent to the management node.

[0047] In the above embodiment, by periodically sending the first time range determined based on the generation time of the compared data to the management node, the management node can clearly know the compared data and the unmatched data in the target data file. Then, when a server is abnormal, it can trigger the reselected server to continue the comparison task, thereby avoiding repeated comparison of data.

[0048] In a possible implementation, the comparison task is performed by multiple servers;

[0049] The performing the comparison task corresponding to the comparison instruction on the target data file includes:

[0050] For any server, receiving a subtask sent by the management node, wherein the comparison instruction corresponds to a subtask obtained by splitting the comparison task;

[0051] After completing the subtasks, the comparison results of the subtasks are fed back to the management node, so that the management node merges the comparison results of the subtasks.

[0052] In the above embodiment, for the scenario of large amounts of data, a distributed comparison method is set up to ensure the efficiency of the server in executing the comparison task, and after each server completes the corresponding subtask, the comparison results of each subtask are fed back to the management node so that the management node can merge them to ensure that a complete comparison result corresponding to the comparison task is obtained.

[0053] In a possible implementation manner, executing the comparison task corresponding to the comparison instruction on the target data file includes:

[0054] If the comparison instruction received includes a first time range, a comparison task is performed on the uncompared data based on the first time range, and the first time range is sent by the management node after determining that other servers are abnormal.

[0055] In the above embodiment, the first time range can be used to determine the data in the target data file that has not been compared after other servers are abnormal, and then the comparison can be continued, avoiding repeated comparison and resource waste, and improving resource utilization.

[0056] In a possible implementation manner, executing the comparison task corresponding to the comparison instruction on the target data file includes:

[0057] During the execution of the comparison task corresponding to the comparison instruction, if a second time range determined based on the generation time of the new data in the target data file is received, the comparison task is performed for the new data based on the second time range, wherein the second time range is determined by the device to which the database storing the target data file belongs after determining that the target data file has new data.

[0058] If the device to which the database belongs determines that new data is added to the target data file, the server will receive a second time range determined based on the generation time of the new data in the target data file, and then can perform a comparison task on the new data based on the second time range. The embodiment of the present application provides a comparison process for the scenario of new data in the target data file, which can ensure the integrity of the comparison of the target data file and obtain more accurate comparison results.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising:

[0060] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor performs any one of the methods of the first aspect or the second aspect described above.

[0061] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute any one of the methods in the first aspect or the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0063] Figure 1 This is a schematic diagram of the first hash ring applicable to the embodiment of the present application;

[0064] Figure 2 A schematic diagram of a data comparison system provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of a data comparison method for a management node provided in an embodiment of the present application;

[0066] Figure 4A schematic diagram of a second hash ring applicable to the embodiment of the present application;

[0067] Figure 5 This is a schematic diagram of a third hash ring applicable to the embodiment of the present application;

[0068] Figure 6 This is a schematic diagram of a fourth hash ring applicable to the embodiment of the present application;

[0069] Figure 7 This is a schematic diagram of a fifth hash ring applicable to the embodiment of the present application;

[0070] Figure 8 This is a schematic diagram of a sixth hash ring applicable to the embodiment of the present application;

[0071] Fig. 9 This is a schematic diagram of the seventh hash ring applicable to the embodiment of the present application;

[0072] Fig.10 A schematic diagram of a data comparison method applied to a server provided in an embodiment of the present application;

[0073] Fig.11 A target data file partitioning schematic diagram provided in an embodiment of the present application;

[0074] Fig.12 A schematic diagram of a server receiving a comparison instruction provided in an embodiment of the present application;

[0075] Fig.13 A schematic diagram of the first data file comparison process provided in an embodiment of the present application;

[0076] Fig.14 A schematic diagram of a second data file comparison process provided in an embodiment of the present application;

[0077] Fig.15 A schematic diagram of the first comparison result merging provided in the embodiment of the present application;

[0078] Fig.16 A schematic diagram of the second comparison result merging provided in the embodiment of the present application;

[0079] Fig.17 A schematic diagram of a third data file comparison process provided in an embodiment of the present application;

[0080] Fig.18 A schematic diagram of a fourth data file comparison process provided in an embodiment of the present application;

[0081] Fig.19 A schematic diagram of the overall flow of a data comparison method provided in an embodiment of the present application;

[0082] Fig. 20A schematic diagram of a data comparison device applied to a management node provided in an embodiment of the present application;

[0083] Fig.21 A schematic diagram of a data comparison device applied to a server provided in an embodiment of the present application;

[0084] Fig. 22 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0086] First, some concepts involved in the embodiments of the present application are introduced:

[0087] Hash ring: In the embodiment of the present application, the hash ring includes a first node corresponding to a data file one by one, and a second node corresponding to a server one by one, such as Figure 1 As shown, for example, the number of nodes on the hash ring is 2^32, that is, the node serial numbers include 0~2^32-1, data file 1 corresponds to node 80 on the hash ring (the node with serial number 80 on the hash ring), and the server corresponds to node 120 on the hash ring (the node with serial number 120).

[0088] Data comparison: It can also be called data collision, that is, multiple data files are compared according to preset comparison rules. The comparison rules can be the same, similar, etc., and the embodiments of the present application do not specifically limit them. For example, the data file of the vehicle passing through point 1 is compared with the data file of the vehicle passing through point 2, and the same vehicle information in the two data files is counted, that is, the vehicles passing through point 1 and point 2 are counted.

[0089] like Figure 2 The figure shows a data comparison system provided by an embodiment of the present application, which includes a user terminal 201, a management node 202 in a server cluster, at least one server in the server cluster (server 203_1, server 203_2, ..., server 203_N shown in the figure) and a device 204 to which the database belongs.

[0090] The user terminal 201 is used to send a comparison instruction for the target data file to the management node. The user terminal 201 can be a mobile phone, a tablet computer, a laptop computer, etc., or it can be software installed on the terminal device, such as an application (Application, APP), a browser, etc., or it can be a web page, a small program, etc. The embodiment of the application does not specifically limit the form of the user terminal. The management node 202 is used to Figure 1 The hash ring shown manages the correspondence between data files and servers, and also responds to the comparison instruction sent by the user terminal 201, and sends the comparison instruction to at least one server. After receiving the comparison instruction sent by the management node 202, at least one server obtains the target data file from the device 204 to which the database belongs, and executes the comparison task corresponding to the comparison instruction. The device 204 to which the above-mentioned database belongs is used to store data files.

[0091] Based on the problems of large impact range and low system stability in the current data file comparison process, the embodiment of the present application provides a data comparison method, which is applied to a management node, such as Figure 3 As shown, the method includes:

[0092] S301: In response to a comparison instruction, determine a first target node in the hash ring corresponding to a target data file to be compared.

[0093] The comparison command is as follows Figure 2 The user terminal shown sends the instruction to the management node 202 in the server cluster, and then the management node 202 instructs any one or more servers in the server cluster to execute the comparison task corresponding to the comparison instruction.

[0094] The target data file includes a complete data file and / or a fragmented data file, and the fragmented data file is obtained by dividing a complete data file whose data amount exceeds a preset amount.

[0095] Taking the preset number 550 as an example, the comparison instruction contains the file identification of the data file to be compared: "Compare file A and file B", where the amount of data in file A is 1000 (exceeding the preset number), and file A (a complete data file) can be further divided into two shard data files, "file A1" and "file A2", with the amount of data in each shard data file being 500, and the amount of data in file B being 500 (not exceeding the preset number), and there is no need to divide file B (the complete data file). The file identifications of the two shard data files need to be associated with the identifications of their corresponding complete data files to clarify the data files that actually perform the comparison task, that is, the data files to be compared indicated in the comparison instruction are file A and file B, and in the actual comparison process, the target data files to be compared are "file A1", "file A2" and "file B". The embodiments of the present application do not limit the preset number, and the specific sharding method (the number of shards, the amount of data in each shard data). When the user terminal 201 sends a comparison instruction to the management node, it does not pay attention to the amount of data in the data file. The management node determines whether the multiple data files to be compared in the response instruction need to be fragmented. If so, it instructs the corresponding server to fragment when obtaining data.

[0096] The hash ring provided in the embodiment of the present application includes multiple first nodes and multiple second nodes. The first nodes in the hash ring correspond one-to-one to the data files, and the second nodes correspond one-to-one to the servers. The hash ring is used to manage the loading relationship between the data files and the servers. The loading relationship means storing the data files in the memory of the server so that the server can obtain the data files locally and execute the comparison task corresponding to the comparison instruction. Before determining the loading relationship between the target data file and the server, it is necessary to first determine the target data file and the first target node corresponding to the hash ring after receiving the comparison instruction sent by the user terminal 201. The specific implementation method is as follows:

[0097] Based on the hash value corresponding to the first identifier of the target data file and the number of nodes in the hash ring, determine the first identification information of the first node in the hash ring corresponding to the target data file; and determine the first target node based on the first identification information.

[0098] The first identifier of the target data file can be a file identifier, such as "file A". The number of nodes in the hash ring is 2^32, and the node sequence number is as follows: Figure 1The direction of the arrow (clockwise) is from 0 to 2^32. Perform a hash calculation on "File A" to obtain the hash value corresponding to "File A", and then use the hash value to modulo the number of nodes to obtain the first identification information, that is, through the formula hash (File A) % (2^32) = 80, it is determined that the target data file corresponds to the node with sequence number 80 in the hash ring. In this regard, the embodiment of the present application does not specifically limit the form of the first identification and the number of nodes in the hash ring.

[0099] The correspondence between the server and the second node is pre-set, specifically: for any server, based on the hash value obtained from the second identifier of the any server and the number of nodes in the hash ring, determine the second identification information of the second node in the hash ring corresponding to the any server; based on the second identification information, establish the correspondence between the any server and the second node in the hash ring.

[0100] The second identifier of the server can be the Internet Protocol (IP) address of the server, such as "10.34.6.21". The number of nodes in the hash ring is 2^32, and the node sequence number is as follows: Figure 1 The direction of the arrow (clockwise) is from 0 to 2^32-1. Perform a hash calculation on "10.34.6.21" to obtain the hash value corresponding to "10.34.6.21", and then use the hash value to modulo the number of nodes to obtain the second identification information, that is, through the formula hash (10.34.6.21) % (2^32) = 120, it is determined that the server corresponds to the node with the serial number 120 in the hash. In this regard, the embodiment of the present application does not specifically limit the form of the second identification.

[0101] If, in the process of determining the first target node corresponding to the target data file, it is found that the first target node obtained based on the first identifier and the number of nodes is occupied, the first target node needs to be re-determined. For example, according to the formula hash (file A) % (2^32) = 80, it is determined that the 80th node of the hash ring corresponding to file A is occupied by server 1 (or file B), then the first target node can be re-determined using the result obtained last time and the number of nodes on the hash ring, that is, according to the formula hash (80) % (2^32) = 160, the first target node corresponding to file A is the node with serial number 160.

[0102] S302: Select a second target node from the hash ring based on the position of the first target node, and trigger a first server corresponding to the second target node to execute a comparison task corresponding to the comparison instruction.

[0103] After determining the first target node corresponding to the target data file, the embodiment of the present application needs to further determine the first server for loading the target data file, which can be specifically implemented as follows:

[0104] A second node closest to the first target node is selected as the second target node along the preset direction of the hash ring, that is, a server closest to the target data file is selected to load the target data file and perform subsequent comparison tasks.

[0105] like Figure 4 As shown, the preset direction is clockwise, and the hash ring includes target data file 1, target data file 2, target data file 3 and target data file 4; server 1, server 2 and server 3. Based on the selection rules in the embodiment of the present application, it is determined to use server 1 to load target data file 1, use server 2 to load target data file 2 and target data file 3, and use server 3 to load data file 4. In the process of selecting the server, the load of the server can be further considered. If the remaining load is small after server 2 loads target data file 2, server 3 (the second closest to target data file 3) can be selected to load target data file 3. Among them, the preset direction can also be counterclockwise, which is not specifically limited in the embodiment of the present application.

[0106] by Figure 4 For example, if the target data files include 4, the management node needs to send the comparison instruction to the first servers corresponding to all the target data files, that is, server 1, server 2 and server 3 will all receive the comparison instruction.

[0107] Since the server cluster has specification restrictions, for example, the amount of data that can be processed is 10^10, that is, the amount of data in the database will not exceed 10^10, therefore, in addition to the above 301, after receiving the comparison instruction, the process of determining the loading of the target data file can also be pre-loaded in advance for all or part of the data files in the database, so that the server can directly obtain the data files from its own memory, thereby improving the efficiency of subsequent execution of the comparison task. Based on this, the user can choose the appropriate loading process according to his own needs for memory or efficiency, and the embodiment of the present application does not make specific limitations.

[0108] In the above preloading process, if Figure 5As shown, after determining the loading relationship between data file 1 and server 1, data file 2, data file 3 and server 2, and data file 4 and server 3, if a new server 4 is added to the server cluster, and server 4 is closer to data file 2, the loading relationship between data file 2 and server 2 is modified to the loading relationship between data file 2 and server 4, and then based on the loading relationship, each server obtains the corresponding data file from the database and stores it in the memory. If data file 2 has been stored in server 2, and a new server 4 is determined, then it can be determined whether to use server 4 to reload data file 2 based on the load capacity of server 2. If the remaining storage space is small after server 2 loads data file 2, server 4 can be used to reload data file 2, and this embodiment of the application does not specifically limit this.

[0109] Ideally, servers should be evenly distributed in the hash ring to achieve uniform data loading, such as Figure 6 As shown in the figure, server 1, server 2 and server 3 are evenly distributed on the hash ring. However, in actual situations, when setting the corresponding relationship between the server and the second node in the hash ring, a "biased ring" phenomenon may occur, such as Figure 7 As shown, the distances between server 1, server 2, and server 3 are less than the preset distance, that is, they are all concentrated in the "upper right part" of the hash ring. According to the loading relationship provided in S302, a large amount of data may be loaded onto server 1, thereby causing server 1 to be overloaded. The embodiment of the present application solves the above problem by constructing a virtual server, such as Figure 8 As shown, virtual machines corresponding to the multiple servers are set on a second node on the hash ring that is not less than a preset distance from the second node corresponding to the multiple servers, wherein, when establishing a corresponding relationship between the server and the second node, it is necessary to select an idle second node, that is, a second node that is not occupied by other servers, server 1_1 and server 1_2 are virtual machines corresponding to server 1; server 2_1 and server 2_2 are virtual machines corresponding to server 2; server 3_1 and server 3_2 are virtual machines corresponding to server 3.

[0110] After the first server completes loading the target data file, it can further perform a comparison task on the target data file based on the comparison rule in the comparison instruction. During the process of the first server performing the comparison task, at least one of the following situations may occur:

[0111] (1) It is detected that the comparison instruction carries a data identifier.

[0112] When generating a data file, the embodiment of the present application will record the generation time of each data to identify the data using the generation time. For example, the time range of the generation time of the data in any target data file is 10:00 on December 1, 2023 to 10:00 on December 31, 2023, and the generation time can be accurate to the day or to the second, which is not specifically limited here. Based on this, the user end 201 can also further carry a data identifier in the response instruction so that the server can clearly identify the data that needs to be compared in the data file, for example, the data in the time range of 10:00 on December 1, 2023 to 10:00 on December 31, 2023 in file A needs to be compared. After receiving the data identifier, the server can perform a comparison task on the data between 10:00 on December 1, 2023 and 10:00 on December 11, 2023 in file A.

[0113] If the comparison instruction does not contain a data identifier, but only contains the file identifier of the data file to be compared, such as comparing file A and file B, all data in file A and file B need to be compared.

[0114] (2) The management node periodically receives the first time range determined based on the generation time of the compared data sent by the first server to determine the data that has been compared by the first server and the data that has not been compared in the target data file.

[0115] With 1 hour as a cycle, 1 hour later, the first server executes the data corresponding to 15:00 on December 3, 2023, and sends 10:00 on December 1, 2023 to 15:00 on December 3, 2023 to the management node. The management node can determine that the compared data is between 10:00 on December 1, 2023 and 15:00 on December 3, 2023 through the received first time range; because the server executes the comparison task in the order of the generation time in the data file, the first server can also send only the generation time corresponding to the data currently being compared to the management node, that is, only 15:00 on December 3, 2023. In order to adapt to actual application scenarios, the embodiment of the present application uses the generation time to identify the data, and can also use other methods. The specific form of data identification is not limited here.

[0116] In some special scenarios, multiple data may have the same generation time. In this case, hash values ​​generated based on random numbers can be used to distinguish them. For example, if the generation time of data 1 and data 2 is both 10:00 on December 1, 2023, then data 1 is identified by the hash value generated by the random number "1" + the generation time, that is, data 1 is identified as hash (1) + December 1, 10:00; data 2 is identified by the hash value generated by the random number "2" + the generation time, that is, data 2 is identified as hash (2) + December 1, 10:00. Therefore, when performing a comparison task based on the generation time of the data, if it is found that the generation time of multiple data is the same, it can be further executed in the order of hash values ​​from large to small (or from small to large).

[0117] Furthermore, the data in the data file can be pre-sorted in order of generation time (if the generation time is the same, the data is sorted from small to large using the hash value) so that the server can perform the comparison task on the data in sequence, without the need to query the generation time during the data comparison process, thereby improving the execution efficiency of the comparison task.

[0118] In addition, the first server can also feedback the first time range to the management node and synchronize the comparison results to the management node to ensure that the comparison results will not be lost after the first server is abnormal, thereby avoiding repeated comparisons. The time when the management node receives the first time range and the comparison results can be the same or different, and the embodiments of the present application do not specifically limit this.

[0119] (3) The management node receives a second time range determined based on the generation time of the new data in the target data file.

[0120] The second time range is determined by the device to which the database storing the target data file belongs after determining that the target data file has new data. When the first server is operating normally, after receiving the second time range, the management node further notifies the first server of the second time range, so that the first server obtains the new data from the database and performs a comparison task on the new data.

[0121] For example, the time range of the generation time of the data in file A is 10:00 on December 1, 2023 to 10:00 on December 31, 2023. If the server corresponding to file A receives the second time range "10:00 on January 1, 2024 to 10:00 on January 10, 2024" during the process of executing the comparison task corresponding to the comparison instruction, the file A server will obtain the data corresponding to 10:00 on January 1, 2024 to 10:00 on January 10, 2024 from file A in the database. If the load capacity of the server corresponding to the current file A is small, the data corresponding to 10:00 on January 1, 2024 to 10:00 on January 10, 2024 in file A can also be used as a shard, and a server can be reselected to perform the comparison task for the newly added data. The embodiment of the present application does not specifically limit this.

[0122] S303: If an exception occurs during the target data file comparison process performed by the first server, a second target node is reselected from multiple second nodes of the hash ring based on the position of the first target node corresponding to the target data file, and the second server of the reselected second target node is triggered to execute the comparison task corresponding to the comparison instruction.

[0123] Since the first server periodically sends the first time range to the management node, if the management node does not receive the first time range for more than a preset time (such as 2 cycles), it is determined that the first server is abnormal; in addition, the management node can also monitor the first server in real time to promptly determine that the first server is abnormal. The embodiment of the present application does not specifically limit the method for determining the abnormality of the first server.

[0124] The data file comparison process provided in S302 is performed by the first server under normal circumstances. If an abnormality occurs in the first server during this process, it is necessary to reselect the second target node. The specific selection method is: select the second node closest to the first target node along the preset direction of the hash ring as the second target node. Fig. 9 As shown, the target data file 1 is originally stored in server 1. If server 1 is abnormal, based on the distribution of servers on the hash ring, server 2 should be selected to obtain the target data file 1 from the database, and trigger server 2 to perform a comparison task for the target data file 1.

[0125] Because the first server will send the first time range representing the matched data to the management node before the first server is abnormal, the management node will notify the second server of the first time range after the first server is abnormal, so that the second server will perform the matching task for the unmatched data based on the first time range. For example, the time range of the generation time of the data in any target data file is from 10:00 on December 1, 2023 to 10:00 on December 31, 2023. If the first time range received by the second service node is from 10:00 on December 1, 2023 to 10:00 on December 20, 2023, it is determined that the unmatched data is the data corresponding to 10:00 on December 20, 2023 to 10:00 on December 31, 2023. After the second service obtains the corresponding data from any target data file in the database, it continues to perform the matching task.

[0126] The management node also stores the comparison results of the target data file before the first server malfunctions. After the second server feeds back the comparison results of the incorrect data to the management node, the management node merges the comparison results of the compared data with the comparison results fed back by the second server to obtain the comparison results of the target data file.

[0127] In the embodiments provided in S301~S303, for operations involving changes in the number of servers in the server cluster (normal addition of servers, normal deletion of servers, or abnormal situations causing the server to be unable to start), the comparison task can be quickly restored by managing servers and data files through a hash ring, as well as rules for selecting servers, thereby reducing the number of affected servers and the duration of the impact, and improving the stability of the server cluster.

[0128] Based on the same inventive concept, the present application embodiment also provides a data comparison method applied to a server, such as Fig.10 As shown, the method includes:

[0129] S1001: Receive a comparison instruction sent by a management node.

[0130] The comparison instruction is sent after the management node determines the first target node in the hash ring corresponding to the target data file to be compared, and selects the second target node from the hash ring based on the position of the first target node. The comparison instruction contains the file identifier of the data file to be compared, such as file A; if the user does not need to compare all the data in file A, the data identifier can also be carried in the comparison instruction so that the server selects the data to be compared from file A. For example, the time range of the generation time of the data in file A is "10:00 on December 1, 2023 to 10:00 on December 31, 2023", and the data identifier contained in the comparison instruction is "10:00 on December 1, 2023 to 10:00 on December 20, 2023", indicating that the user wants to compare the data corresponding to 10:00 on December 1, 2023 to 10:00 on December 20, 2023 in file A.

[0131] S1002: Acquire a target data file corresponding to the comparison instruction from a database.

[0132] The comparison instruction sent by the user terminal 201 to the management node includes the file identifiers or data identifiers of multiple files that the user needs to compare, such as "file A and file B" comparison, or "data corresponding to 10:00 on December 1, 2023 to 10:00 on December 20, 2023 in file A and data corresponding to 10:00 on December 1, 2023 to 10:00 on December 20, 2023 in file B". The user terminal does not care whether file A and file B need to be divided, and which servers are used to perform the comparison task. After receiving the comparison instruction, the management node will determine the data files that need to be divided and the servers for each data file pair.

[0133] Take file A and file B as an example. Fig.11As shown, the management node determines to divide file A into file A1, file A2, file A3, and file B into file B1, file B2, file B3, as in the implementation in S301 above, which can be divided specifically according to the amount of data, which will not be repeated here. The management node maps file A1, file A2, file A3, file B1, file B2, and file B3 to the hash ring respectively, selects the corresponding server for each file, and then sends the comparison instruction to each server. Among them, the time range corresponding to the data in file A1 is 10:00 on December 1, 2023 to 10:00 on December 10, 2023, the time range corresponding to the data in file A2 is 10:00 on December 10, 2023 to 10:00 on December 20, 2023, and the time range corresponding to the data in file A3 is 10:00 on December 20, 2023 to 10:00 on December 31, 2023. Files B1, B2, and B3 also correspond to time ranges respectively.

[0134] A server in the server cluster may receive a comparison instruction containing "Compare the data from 10:00 on December 1, 2023 to 10:00 on December 10, 2023 (A1) in file A and the data from 10:00 on December 1, 2023 to 10:00 on December 10, 2023 (B1) in file B". Server 2 receives a comparison instruction containing "Compare the data from 10:00 on December 10, 2023 to 10:00 on December 20, 2023 (B2) in file A". Server 3 receives a comparison instruction including "compare the data (A2) in file A from 10:00 on December 10, 2023 to 10:00 on December 20, 2023 (B2)"; Server 3 receives a comparison instruction including "compare the data (A3) in file A from 10:00 on December 20, 2023 to 10:00 on December 31, 2023 and the data (B3) in file B from 10:00 on December 20, 2023 to 10:00 on December 31, 2023".

[0135] S1003: Based on the generation time sequence of the data in the target data file, executing the comparison task corresponding to the comparison instruction on the target data file, and obtaining the comparison result corresponding to the target data file.

[0136] For the scenario of large-scale data comparison, the embodiment of the present application divides the target data to be compared. Similarly, the comparison task corresponding to the comparison instruction is split into multiple subtasks and sent to multiple servers so that each server executes the corresponding subtask.

[0137] by Fig.12For example, server 1 may receive the comparison instructions of the files that it needs to compare, such as "Compare file A1 and file B1, compare file A1 and file B2, compare file A1 and file B3" sent by the management node; it may also receive all the instructions, such as "Compare file A1 and file B1, compare file A1 and file B2, compare file A1 and file B3, compare file A2 and file B1, compare file A2 and file B2, compare file A2 and file B3, compare file A3 and file B1, compare file A3 and file B2, compare file A3 and file B3", then server 1 needs to coordinate with other servers on how to perform the comparison task. For example, when server 1 compares file A1 and file B2, it finds that the current load is large, and informs server 2 to perform the task of comparing file A1 and file B2. It should be noted that the server will not perform tasks corresponding to data that has not been obtained, such as server 1 will not perform tasks unrelated to file A1 and file B1.

[0138] The comparison process of file A1 and file B1 is as follows: Fig.13 As shown, the data corresponding to 10:00 on December 1, 2023 to 10:00 on December 20, 2023 in file A1 is compared with the data corresponding to 10:00 on December 1, 2023 to 10:00 on December 20, 2023 in file B1. The data identifier of file A1 and the data identifier of file B1 can be the same or different, and the user can set it according to needs, which is not specifically limited in the embodiment of the present application.

[0139] Fig.12 Server 1 receives the tasks of "comparing file A1 and file B1, comparing file A1 and file B2, comparing file A1 and file B3". It can open up three threads to execute the three tasks simultaneously. When executing the comparison task of file A1 and file B2, file B2 needs to be copied from server 2 to the local computer. Considering the load problem of server 1, batch copying and batch execution of comparison tasks are possible.

[0140] like Fig.14 As shown, taking the example of 100 data in each of file A1 and file B1, with 20 as a batch, 20 data are copied from server 2 each time to perform the comparison task. In order to improve the efficiency of executing the comparison task, the next batch of data can be copied while executing the current batch of tasks to ensure the continuity of the task. The current batch of data is deleted after the comparison is completed to release memory.

[0141] Since the comparison instruction issued by the user terminal 201 is to compare file A and file B, after each server completes the comparison task, obtains the comparison results and aggregates the obtained comparison results to one server, the server merges the obtained comparison results to obtain the comparison results corresponding to file A and file B, and then reports them to the management node. Alternatively, each server may report the obtained comparison results to the management node, and the management node merges the comparison results. The merging process is as follows: Fig.15 As shown, the comparison results of file A1 and file B1, the comparison results of file A1 and file B2, and the comparison results of file A1 and file B3 are merged to obtain the comparison results of file A1 and file B. Similarly, the comparison results of file A2 and file B and the comparison results of file A3 and file B can be obtained. Then, the comparison results of A1 and file B, the comparison results of A2 and file B, and the comparison results of A3 and file B are merged to obtain the comparison results of file A and file B, as shown in FIG. Fig.16 In addition, a unique task ID can be set for each comparison task, and all the split subtasks will carry the unique subtask ID when they are executed, so that after the split subtasks are executed, the results of multiple servers can be merged based on the task ID.

[0142] When the server performs the comparison task, at least one of the following situations may occur:

[0143] (1) Periodically sending to the management node a first time range determined based on the generation time of the compared data.

[0144] When the server performs the comparison task normally, it will send the first time range to the management node so that the management node can determine the compared data in time, so that after the server is abnormal, other normal servers can continue to compare the data that may not be matched based on the first time range sent by the management node. The specific implementation method is as described in S302 above, which will not be repeated here.

[0145] (2) A comparison instruction including a first time range is received.

[0146] The first time range here is sent by the management node after determining that other servers are abnormal. If other servers that are performing the comparison task do not send the first time range to the management node within the preset time, it is determined that other servers are abnormal. At this time, the management node will reselect a normal server to continue to perform the comparison task performed by other servers, that is, transfer the tasks of other servers to this server.

[0147] For example, the time range corresponding to the data in file A is 10:00 on December 1, 2023 to 10:00 on December 31, 2023, and the first time range is 10:00 on December 1, 2023 to 10:00 on December 10, 2023, indicating that the data in file A that may not be correct is the data corresponding to 10:00 on December 10, 2023 to 10:00 on December 31, 2023. After receiving the first time range, the data corresponding to 10:00 on December 10, 2023 to 10:00 on December 31, 2023 in file A is obtained from the database, and the comparison task is performed on the obtained data.

[0148] (3) A second time range determined based on the generation time of the new data in the target data file is received.

[0149] The second time range is determined by the device to which the database storing the target data file belongs after determining that the target data file has new data. The scenario of the new data in the target data file includes at least one of the following situations:

[0150] 1. The server performs a comparison task on the current data in the target data file and receives the second time range before obtaining the comparison result corresponding to the current data.

[0151] If a second time range representing newly added data is received before the comparison result corresponding to the current data is obtained, the newly added data is obtained from the target data file in the database based on the second time range. After the comparison task is completed on the current data, the comparison task is continued to be performed on the newly added data to obtain the comparison result for the target data file.

[0152] 2. The server performs a comparison task on the current data in the target data file, obtains a comparison result corresponding to the current data, and receives a second time range.

[0153] If after obtaining the comparison result corresponding to the current data, a second time range representing the newly added data is received, the newly added data is obtained from the target data file in the database based on the second time range, and the comparison task is performed on the newly added data to obtain the comparison result of the newly added data, and then the comparison result corresponding to the current data is merged with the comparison result of the newly added data to obtain the comparison result corresponding to the target data file.

[0154] The above execution is for the comparison task of the newly added data. Fig.17 and Fig.18 shown.

[0155] Fig.17The time range corresponding to the current data in file A is 10:00 on December 1, 2023 to 10:00 on December 31, 2023, the time range corresponding to the current data in file B is 10:00 on December 1, 2023 to 10:00 on December 31, 2023, and the newly added data in file A is 10:00 on December 31, 2023 to 10:00 on January 10, 2024. If the comparison result corresponding to the current data has not been obtained, in the process of comparing the current data of file A with file B, the data corresponding to 10:00 on December 31, 2023 to 10:00 on January 10, 2024 is obtained from database file A. After comparing the current data of file A with file B, the newly added data of file A is compared with file B to obtain the comparison result of file A and file B. If the comparison result of the current data of file A and file B has been obtained, the data corresponding to 10:00 on December 31, 2023 to 10:00 on January 10, 2024 is obtained from the database file A, and the newly added data in file A is compared with file B to obtain the comparison result of the newly added data in file A and file B, and then the comparison result of the current data and the comparison result of the newly added data are merged to obtain the comparison result of file A and file B.

[0156] like Fig.18 As shown, if the newly added data in file A is compared with the current data in file B and the comparison result has not been obtained, the second time range representing the newly added data in file B is received: 10:00 on December 31, 2023 to 10:00 on January 10, 2024, then in the process of comparing the newly added data in file A with the current data in file B, the newly added data in file B is obtained from the database, and after the comparison of the newly added data in file A with the current data in file B is completed, the newly added data in file A (including the newly added data) is continued to be compared with the newly added data in file B. If the newly added data in file A is compared with the current data in file B and the comparison result is obtained, the second time range representing the newly added data in file B is received, the newly added data in file B is obtained from the database, and file A (including the newly added data) is compared with the newly added data in file B to obtain the comparison result of the newly added data in file A and file B, and then the comparison result of the newly added data in file A and file B is merged with the previous comparison result to obtain the comparison result of file A and file B.

[0157] After the user end 201 sends the comparison instruction, it does not know whether there is new data in file A or file B during the data file comparison process. The data comparison method in a new data scenario in an embodiment of the present application further merges the results of the new data with the results of the previous comparison, which can ensure the accuracy and completeness of the comparison results.

[0158] Below through Fig.19 The process of a data comparison method provided in an embodiment of the present application is described.

[0159] S1901: The management node responds to a comparison instruction sent by the user terminal and determines the first target node in the hash ring corresponding to the target data file, where the comparison instruction includes a file identifier of the data file that the user wants to compare;

[0160] S1902: The management node selects a server to perform the comparison task, specifically through a pre-built hash ring, which is specifically implemented as in S301 to S302 and will not be described in detail here;

[0161] S1903: The management node sends a comparison instruction to a corresponding server, where the server includes multiple servers;

[0162] S1904: After receiving the comparison instruction, the server executes the comparison task corresponding to the comparison instruction in a "multi-threaded, batched" manner; multi-threading means that a server will open up multiple threads to execute multiple comparison tasks at the same time, thereby improving the execution efficiency of the comparison tasks, such as Fig.12 When performing the comparison task of file A1 and file B2, server 1 obtains data from file B2 of server 2 in batches to perform the comparison task of file A1 and file B2, so as to ensure that server 1 will not be overloaded when performing the comparison task;

[0163] S1905: The server reports the obtained comparison results to the management node. If there is new data, the server merges the results of multiple comparisons and reports them to the management node.

[0164] S1906: The management node feeds back the comparison result to the user end.

[0165] During the overall execution of the comparison task for the target data file, the number of servers, threads, and thread batches that execute the task can be flexibly configured according to performance requirements and data file quantity requirements to meet the data file comparison requirements of complex scenarios. The task time consumption is optimized and the efficiency of data file comparison is improved by processing data file comparison tasks in batches using multiple servers, multiple threads, and each thread. The data files and servers in the server cluster determine the loading relationship through a consistent hash ring. When the number of servers changes due to manual addition, manual deletion, or abnormalities in the server cluster, the management node still loads the data files according to the preset rules to reduce the scope of impact and quickly restore availability.

[0166] Based on the same inventive concept, the embodiment of the present application also provides a data comparison device, which is applied to a management node, such as Fig. 20 As shown, the device comprises:

[0167] The corresponding relationship determination module 2001 is used to respond to the comparison instruction and determine the first target node in the hash ring corresponding to the target data file to be compared;

[0168] A first trigger module 2002 is used to select a second target node from the hash ring based on the position of the first target node, and trigger a first server corresponding to the second target node to execute a comparison task corresponding to the comparison instruction, wherein the hash ring includes a plurality of first nodes and a plurality of second nodes, a first node in the hash ring corresponds to a data file one-to-one, and a second node in the hash ring corresponds to a server one-to-one;

[0169] The second trigger module 2003 is used to reselect a second target node from multiple second nodes of the hash ring based on the position of the first target node corresponding to the target data file if an exception occurs during the target data file comparison process performed by the first server, and trigger the second server of the reselected second target node to execute the comparison task corresponding to the comparison instruction.

[0170] In a possible implementation, the corresponding relationship determination module 2001 is used to determine the first identification information of the first node in the hash ring corresponding to the target data file based on the hash value corresponding to the first identification of the target data file and the number of nodes in the hash ring;

[0171] A first target node is determined based on the first identification information.

[0172] In a possible implementation, the corresponding relationship determination module 2001 is used to determine, for any server, the second identification information of the second node in the hash ring corresponding to the any server based on the hash value obtained from the second identification of the any server and the number of nodes in the hash ring;

[0173] A corresponding relationship between any one of the servers and the second node in the hash ring is established based on the second identification information.

[0174] In a possible implementation, the corresponding relationship determination module 2001 is used to set virtual machines corresponding to the multiple servers on second nodes on the hash ring that are not less than a preset distance from the second nodes corresponding to the multiple servers when the distance between the second nodes corresponding to the multiple servers is less than a preset distance.

[0175] In a possible implementation, the corresponding relationship determination module 2001 is used to select a second node that is closest to the first target node along a preset direction of the hash ring as the second target node.

[0176] In a possible implementation, the first trigger module 2002 is used to:

[0177] Splitting the comparison task into multiple subtasks;

[0178] For any subtask and any first server, trigger the any first server to execute the any subtask;

[0179] After receiving the comparison results of each subtask, the comparison results are merged, and the number of the subtasks matches the number of the first servers.

[0180] In a possible implementation manner, the comparison task is performed by the server based on the generation time sequence of the data in the target data file;

[0181] The device further comprises a receiving module, which is used to periodically receive a first time range determined based on the generation time of the compared data and sent by the first server during the process of the first server executing the comparison task corresponding to the comparison instruction;

[0182] The second trigger module 2003 is used for:

[0183] After determining that an exception occurs during the target data file comparison process performed by the first server, the second server is notified of the first time range received most recently, so that the second server performs a comparison task for unmatched data based on the first time range.

[0184] In a possible implementation manner, the second trigger module 2003 is used to:

[0185] During the process of the first server executing the comparison task corresponding to the comparison instruction, if a second time range determined based on the generation time of the new data in the target data file is received, the second time range is notified to the second server so that the second server performs the comparison task for the new data based on the second time range, wherein the second time range is determined by the device to which the database storing the target data file belongs after determining that the target data file has new data.

[0186] Based on the same concept, the present application also provides a data comparison device, which is applied to a server, such as Fig.21 As shown, the device comprises:

[0187] The receiving module 2101 is used to receive a comparison instruction sent by the management node, wherein the comparison instruction is sent after the management node determines a first target node in a hash ring corresponding to a target data file to be compared, and selects a second target node from the hash ring based on the position of the first target node, the hash ring includes a plurality of first nodes and a plurality of second nodes, a first node in the hash ring corresponds one-to-one to a data file, and a second node in the hash ring corresponds one-to-one to a server;

[0188] The acquisition module 2102 is used to acquire the target data file corresponding to the comparison instruction from the database;

[0189] The comparison module 2103 is used to execute the comparison task corresponding to the comparison instruction on the target data file based on the generation time sequence of the data in the target data file, and obtain the comparison result corresponding to the target data file.

[0190] In a possible implementation, the comparison task is performed by multiple servers;

[0191] The comparison module 2103 is used to:

[0192] For any server, receiving a subtask sent by the management node, wherein the comparison instruction corresponds to a subtask obtained by splitting the comparison task;

[0193] After the subtasks are completed, the comparison results of the subtasks are fed back to the management node, so that the management node merges the comparison results of the subtasks.

[0194] In a possible implementation, the device further includes a sending module, configured to periodically send to the management node a first time range determined based on a generation time of the compared data during execution of the comparison task corresponding to the comparison instruction.

[0195] In a possible implementation, the comparison module 2103 is used to:

[0196] If the comparison instruction received includes a first time range, a comparison task is performed on the uncompared data based on the first time range, and the first time range is sent by the management node after determining that other servers are abnormal.

[0197] In a possible implementation, the comparison module 2103 is used to:

[0198] During the execution of the comparison task corresponding to the comparison instruction, if a second time range determined based on the generation time of the new data in the target data file is received, the comparison task is performed for the new data based on the second time range, wherein the second time range is determined by the device to which the database storing the target data file belongs after determining that the target data file has new data.

[0199] Based on the same inventive concept, an embodiment of the present application provides an electronic device, the device comprising:

[0200] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned data comparison method applied to the management node or server.

[0201] Refer to the following Fig. 22 220 according to this embodiment of the present application is described. Fig. 22 The displayed 220 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0202] like Fig. 22 As shown, 220 is in the form of a general electronic device. Components of 220 may include but are not limited to: at least one processor 221, at least one memory 222, and a bus 223 connecting different system components (including the memory 222 and the processor 221).

[0203] The processor 221 is used to read and execute instructions in the memory 222, so that the at least one processor can execute a data comparison method applied to a management node or a server provided in the above embodiment.

[0204] Bus 223 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0205] The memory 222 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 2221 and / or a cache memory 2222 , and may further include a read-only memory (ROM) 2223 .

[0206] The memory 222 may also include a program / utility 2225 having a set (at least one) of program modules 2224, such program modules 2224 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0207] The electronic device 220 may also communicate with one or more external devices 224 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 220, and / or communicate with any device (e.g., routers, modems, etc.) that enables the electronic device 220 to communicate with one or more other electronic devices. Such communication may be performed through an input / output (I / O) interface 225. In addition, the electronic device 220 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 226. As shown, the network adapter 226 communicates with other modules for the electronic device 220 through a bus 223. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 220, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0208] In some possible implementations, various aspects of a data comparison method applied to a management node or server provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a data comparison method applied to a management node or server according to various exemplary implementations of the present application described above in this specification.

[0209] In addition, the present application also provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program is used to enable a computer to execute any of the methods described in the above embodiments.

[0210] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0211] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0212] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0213] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data comparison method, characterized in that: The method comprises: In response to the comparison instruction, determine a first target node corresponding to the target data file in the hash ring based on a hash value corresponding to a first identifier of the target data file to be compared and the number of nodes in the hash ring, and when it is determined that the first target node is occupied, re-determine the first target node based on a position of the first target data node in the hash ring and the number of nodes in the hash ring; Selecting a second target node from the hash ring based on the position of the first target node, and triggering a first server corresponding to the second target node to execute a comparison task corresponding to the comparison instruction, wherein the hash ring includes a plurality of first nodes and a plurality of second nodes, a first node in the hash ring corresponds to a data file one-to-one, a second node in the hash ring corresponds to a server one-to-one, and the comparison task is executed by the server based on the generation time sequence of the data in the target data file, and when the generation time sequence is the same, it is executed based on the size sequence of the hash value of the random number; During the process of comparing the target data file by the first server, if the first time range determined based on the generation time corresponding to the compared data and the hash value of the random number is not received within a preset time, a second target node is reselected from multiple second nodes of the hash ring based on the position of the first target node corresponding to the target data file, and the second server of the reselected second target node is triggered to execute the comparison task corresponding to the comparison instruction; During the process of the first server executing the comparison task corresponding to the comparison instruction, if a second time range determined based on the generation time of the new data in the target data file and the hash value of the random number is received, the second time range is notified to the first server so that the first server performs the comparison task for the new data based on the second time range. The second time range is sent by the device to which the database storing the target data file belongs after determining that the target data file has new data.

2. The method according to claim 1, characterized in that The corresponding relationship between the server and the second node is determined in the following manner: For any one server, based on the hash value obtained from the second identifier of the any one server and the number of nodes in the hash ring, determine the second identifier information of the second node in the hash ring corresponding to the any one server; A corresponding relationship between any one of the servers and the second node in the hash ring is established based on the second identification information.

3. The method according to claim 2, characterized in that The method further comprises: If the distance between the second nodes corresponding to the multiple servers is less than the preset distance, the virtual machines corresponding to the multiple servers are set on the second nodes on the hash ring whose distance from the second nodes corresponding to the multiple servers is not less than the preset distance.

4. The method according to claim 1, characterized in that: The selecting a second target node from the hash ring based on the position of the first target node includes: A second node closest to the first target node is selected along a preset direction of the hash ring as the second target node.

5. The method according to claim 1, characterized in that: The first server executing the comparison task includes a plurality of servers; The triggering the first server corresponding to the second target node to execute the comparison task corresponding to the comparison instruction includes: Splitting the comparison task into multiple subtasks; For any subtask and any first server, trigger the any first server to execute the any subtask; After receiving the comparison results of each subtask, the comparison results are merged, and the number of the subtasks matches the number of the first servers.

6. The method according to claim 1, characterized in that The method further comprises: During the process in which the first server executes the comparison task corresponding to the comparison instruction, periodically receiving a first time range sent by the first server; The triggering of the reselected second server of the second target node to execute the comparison task corresponding to the comparison instruction also includes: After determining that an exception occurs during the target data file comparison process performed by the first server, the second server is notified of the first time range received most recently, so that the second server performs a comparison task for unmatched data based on the first time range.

7. A data comparison method, characterized in that: Applied to a server, the method comprises: Receiving a comparison instruction sent by a management node, wherein the comparison instruction is sent after the management node determines a first target node corresponding to a target data file to be compared in a hash ring, and selects a second target node from the hash ring based on a position of the first target node, wherein the first target node is determined based on a hash value corresponding to a first identifier of the target data file and the number of nodes in the hash ring; Acquire the target data file corresponding to the comparison instruction from the database; Based on the generation time sequence of the data in the target data file and the size sequence of the hash value of the random number, the comparison task corresponding to the comparison instruction is executed on the target data file to obtain the comparison result corresponding to the target data file; If a first time range determined based on the generation time corresponding to the compared data and the hash value of the random number included in the comparison instruction is received, a comparison task is performed on the uncompared data based on the first time range, and the first time range is sent by the management node after determining that other servers are abnormal; In the process of executing the comparison task corresponding to the comparison instruction, if a second time range determined based on the generation time of the newly added data in the target data file and the hash value of the random number is received, the comparison task is performed on the newly added data based on the second time range, wherein the second time range is determined by the device to which the database storing the target data file belongs after determining that the target data file has newly added data; The hash ring includes a plurality of first nodes and a plurality of second nodes, the first nodes in the hash ring correspond one-to-one to the data files, and the second nodes in the hash ring correspond one-to-one to the servers.

8. The method according to claim 7, characterized in that Execute the comparison task through multiple servers; The performing the comparison task corresponding to the comparison instruction on the target data file includes: For any server, receiving a comparison instruction sent by the management node, wherein the comparison instruction corresponds to a subtask obtained by splitting the comparison task; After the subtasks are completed, the comparison results of the subtasks are fed back to the management node, so that the management node merges the comparison results of the subtasks.

9. The method according to claim 7, characterized in that: The performing the comparison task corresponding to the comparison instruction on the target data file includes: During the execution of the comparison task corresponding to the comparison instruction, a first time range determined based on the generation time of the compared data is periodically sent to the management node.

10. An electronic device, characterized in that: The device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor executes a method as described in any one of claims 1 to 6, or executes a method as described in any one of claims 7 to 9.

11. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 6, or to execute the method according to any one of claims 7 to 9.

Citation Information

Patent Citations

  • Data processing method and device and storage medium

    CN116028196A