Data deduplication method and device based on rail transit system

Abstract

The application provides a data deduplication method and device based on a rail transit system, and the method comprises the following steps: confirming a master node and a plurality of slave nodes of the rail transit system based on a distributed lock mechanism; deploying a plurality of filters on the master node and initializing the filters, generating initialization parameters and performing cluster configuration based on a Redis database, and initializing the filters based on the initialization parameters on the plurality of slave nodes; and performing data deduplication processing based on the plurality of filters to obtain a data deduplication result. The application realizes multi-node distributed deduplication, and improves deduplication accuracy and efficiency.

Description

Technical Field

[0001] This invention relates to the field of rail transit information technology, and in particular to a data deduplication method and apparatus based on rail transit systems. Background Technology

[0002] With the rapid advancement of urbanization, rail transit technology is developing rapidly. In response to the national policy of advocating green travel, the construction of public rail transit is increasing.

[0003] As rail transit becomes increasingly sophisticated and the number of vehicles and trackside equipment grows, more and more status data needs to be reported. However, the sheer volume of data increases the difficulty of analysis and computation. To mitigate this difficulty, we typically add a deduplication step to remove highly similar data from similar time periods. The method and accuracy of deduplication significantly impact the data quality.

[0004] Therefore, improving the accuracy and efficiency of data deduplication in rail transit systems is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] This invention provides a data deduplication method and apparatus based on rail transit systems, which addresses the shortcomings of existing technologies where the deduplication method and accuracy have a significant impact on the data. It achieves deduplication through multi-node distributed deduplication, thereby improving the accuracy and efficiency of deduplication.

[0006] This invention provides a data deduplication method based on a rail transit system, comprising:

[0007] The master node and multiple slave nodes of the rail transit system are confirmed based on a distributed lock mechanism.

[0008] Multiple filters are deployed and initialized on the master node, initialization parameters are generated, and cluster configuration is performed based on the Redis database. The filters are then initialized on the multiple slave nodes based on the initialization parameters.

[0009] Data deduplication is performed based on the multiple filters to obtain the data deduplication result.

[0010] According to a data deduplication method for a rail transit system provided by the present invention, after the plurality of slave nodes initialize the filter based on the initialization parameters, the method further includes:

[0011] Based on a preset first time threshold, the node permissions are contested periodically. A target master node is identified among the multiple slave nodes using a distributed lock mechanism, and the target master node replaces the master node.

[0012] Based on the operating configuration of the target master node, all performance points are coordinated within the cluster.

[0013] According to a data deduplication method for rail transit systems provided by the present invention, the filter includes a main filter and at least two backup filters;

[0014] The data deduplication process based on the multiple filters to obtain the data deduplication result includes:

[0015] Based on the presence of verification data by the main filter and any backup filter, a verification result is obtained, and based on the verification result, it is determined whether the data is duplicated;

[0016] Based on a preset second time threshold, the main filter and at least two backup filters are repeatedly switched and recycled to repeatedly deduplicate the data and obtain the deduplicated data result.

[0017] According to the present invention, a data deduplication method based on a rail transit system includes repeatedly switching and recycling the main filter and at least two backup filters based on a preset second time threshold to repeatedly perform deduplication processing on the data and obtain a data deduplication result, comprising:

[0018] At each preset second time threshold interval, the main filter is switched and the backup filter is recycled, and the new backup filter is set as the target main filter, and a switching identifier, a recycling identifier, and a new target identifier are generated accordingly;

[0019] Based on the switching identifier, the recycling identifier, and the newly set target identifier, the data is deduplicated multiple times to obtain the data deduplication result.

[0020] According to the present invention, a data deduplication method based on a rail transit system includes deploying and initializing multiple filters on the master node, generating initialization parameters, and configuring the cluster based on a Redis database, comprising:

[0021] Multiple filters are deployed and initialized on the master node, the initialization parameters are generated and stored in the Redis database, and cluster configuration is performed based on the target key specified in the Redis database.

[0022] The initialization of the filter by the plurality of slave nodes based on the initialization parameters includes:

[0023] The initialization parameters are retrieved from the Redis database, and multiple filters on the slave node are initialized based on the initialization parameters.

[0024] According to a data deduplication method for a rail transit system provided by the present invention, the initialization of the filter by the plurality of slave nodes based on the initialization parameters includes:

[0025] Message notification is performed using open-source message broker software, sending the initialization parameters of the master node to multiple slave nodes;

[0026] The multiple slave nodes listen to the open-source message broker software, receive initialization parameters sent by the master node, and initialize the filter based on the initialization parameters.

[0027] The present invention also provides a data deduplication device based on a rail transit system, comprising:

[0028] The node confirmation module is used to confirm the master node and multiple slave nodes of the rail transit system based on a distributed lock mechanism.

[0029] An initialization module is used to deploy and initialize multiple filters on the master node, generate initialization parameters and configure the cluster based on the Redis database, and initialize the filters on the multiple slave nodes based on the initialization parameters;

[0030] The deduplication module is used to perform data deduplication processing based on the multiple filters to obtain the data deduplication result.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data deduplication method based on the rail transit system described above.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data deduplication method based on a rail transit system as described above.

[0033] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data deduplication method based on a rail transit system as described above.

[0034] The present invention provides a data deduplication method and apparatus based on a rail transit system. It identifies the master node and multiple slave nodes of the rail transit system based on a distributed lock mechanism, deploys and initializes multiple filters on the master node, generates initialization parameters and configures the cluster based on a Redis database, initializes the filters on multiple slave nodes based on the initialization parameters, performs data deduplication processing based on multiple filters, and obtains the data deduplication result. It realizes deduplication through multi-node distributed deduplication, thereby improving the accuracy and efficiency of deduplication. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0036] Figure 1 This is one of the flowcharts illustrating the data deduplication method based on a rail transit system provided by the present invention;

[0037] Figure 2 This is a schematic diagram of the master node sending initialization parameters to the slave node in this invention;

[0038] Figure 3 This is the second flowchart of the data deduplication method based on rail transit systems provided by the present invention;

[0039] Figure 4 This is a schematic diagram of the data deduplication device based on a rail transit system provided by the present invention;

[0040] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0042] Reference Figure 1 , Figure 1 This is a flowchart illustrating the data deduplication method for rail transit systems provided by the present invention, which includes the following steps:

[0043] Step 110: Based on the distributed lock mechanism, confirm the master node and multiple slave nodes of the rail transit system;

[0044] Step 120: Deploy and initialize multiple filters on the master node, generate initialization parameters and configure the cluster based on the Redis database, and initialize the filters on the multiple slave nodes based on the initialization parameters;

[0045] Step 130: Perform data deduplication based on the multiple filters to obtain the data deduplication result.

[0046] The data deduplication method for rail transit systems provided by this invention can be implemented by an electronic device, a component within the electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), or personal computer (PC), etc. This invention does not impose specific limitations.

[0047] The following example, using a computer executing the data deduplication method based on a rail transit system provided by this invention, illustrates the technical solution of this invention in detail.

[0048] First of all, it should be noted that the data in this embodiment all come from the rail transit system, including vehicle data, track data, platform data, etc.

[0049] In step 110, when multiple nodes of the rail transit system start up, they will use Redis's SETNX mechanism (i.e., distributed lock mechanism) to compete for the lock. The node that obtains the distributed lock becomes the master node.

[0050] It should be noted that Redis (Remote Dictionary Server) is an open-source, ANSI C-written, network-enabled, in-memory or persistent log-structured key-value database that provides APIs for multiple languages.

[0051] The SETNX locking mechanism works as follows: If the specified key does not exist, it is created, its value is set, and a status code 1 is returned; if the specified key exists, a status code 0 is returned directly. A return value of 1 indicates that the lock has been acquired; when other processes attempt to create the lock again, since the key already exists, a status code 0 will be returned, indicating that the lock is already held.

[0052] This step divides multiple nodes into master nodes and slave nodes, enabling distributed deduplication of data across multiple nodes. By leveraging the master-slave relationship between different nodes, internal cluster coordination is achieved, thereby improving the efficiency and accuracy of data deduplication.

[0053] In step 120, after the master node is selected, multiple filters are prepared, such as filter A, filter B, and filter C. It should be noted that all filters in this step are Bloom filters.

[0054] The principle of a Bloom filter is that when an element is added to a set, it is mapped to K points in a bit array using K hash functions, and these points are set to 1. During retrieval, we only need to check if all these points are 1 to (approximately) know whether the element is in the set: if any of these points is 0, the element being searched is definitely not present; if all of them are 1, the element being searched is very likely present.

[0055] Initialize any two of the three filters, for example, initialize filters A and B, while keeping filter C with its current data. Generate the initialization configuration parameters, which are then configured for clustering using the specified key from the Redis database.

[0056] After the master node is initialized, multiple slave nodes can be initialized with slave node filters based on the master node's initialization parameters. These initialization parameters can be sent from the master node to the slave nodes, or retrieved from the Redis database by the slave nodes.

[0057] In step 130, after the master node and multiple slave nodes initialize the filter, multiple filters are used to repeatedly deduplicate the data. There is no need to pursue unlimited deduplication. Based on high availability and low maintenance, data deduplication within a time period is achieved, which improves deduplication efficiency and accuracy, and saves operation and maintenance costs.

[0058] The data deduplication method based on rail transit systems provided in this invention uses a distributed lock mechanism to identify the master node and multiple slave nodes of the rail transit system. Multiple filters are deployed and initialized on the master node, initialization parameters are generated and cluster configuration is performed based on a Redis database, and the filters are initialized on multiple slave nodes based on the initialization parameters. Data deduplication is performed based on multiple filters to obtain the data deduplication result. This method improves the accuracy and efficiency of deduplication through multi-node distributed deduplication.

[0059] Based on the above embodiments, after the plurality of slave nodes initialize the filter based on the initialization parameters, the method further includes:

[0060] Based on a preset first time threshold, the node permissions are contested periodically. A target master node is identified among the multiple slave nodes using a distributed lock mechanism, and the target master node replaces the master node.

[0061] Based on the operating configuration of the target master node, all performance points are coordinated within the cluster.

[0062] Specifically, this embodiment provides an implementation process for switching the master node.

[0063] To prevent the master node from going offline, all nodes periodically compete for master node privileges. During this competition, each slave node also competes for the new master node through the SETNX locking mechanism.

[0064] After multiple slave nodes compete for control, a new master node is selected as the target master node, replacing the previous master node. Then, the cluster's internal coordination and filter initialization configurations are based on the current target master node.

[0065] The data deduplication method based on rail transit systems provided in this invention ensures that nodes do not go offline by periodically competing for the master node and using the new master node for cluster configuration and filter initialization. This guarantees the overall framework of multi-node distributed deduplication, thereby improving the accuracy and efficiency of deduplication.

[0066] Based on the above embodiments, the filter includes a main filter and at least two backup filters;

[0067] The data deduplication process based on the multiple filters to obtain the data deduplication result includes:

[0068] Based on the presence of verification data by the main filter and any backup filter, a verification result is obtained, and based on the verification result, it is determined whether the data is duplicated;

[0069] Based on a preset second time threshold, the main filter and at least two backup filters are repeatedly switched and recycled to repeatedly deduplicate the data and obtain the deduplicated data result.

[0070] Specifically, in this embodiment, both the main filter and the backup filter are Bloom filters.

[0071] Set filter B as the primary filter and filter A as the backup filter.

[0072] When data needs to be validated, based on the principle of Bloom filters, it first checks whether the data is in filter A or filter B, and then outputs the validation result. The validation result indicates whether the data exists or not.

[0073] If the data exists, inform the user that it is a duplicate. If the data does not exist, add it to the main filter and inform the user that the data is not a duplicate.

[0074] Then, after the set unit time expires, the main filter B and the backup filter A are switched and recycled, and another backup filter is then used as the main filter. The filter switching and recycling are performed at fixed intervals to repeatedly deduplicate the data. Data deletion is achieved through multiple filters, solving the problem that a single filter cannot delete data.

[0075] The data deduplication method based on rail transit systems provided in this invention uses timed switching of recycling filters and the coordinated use of multiple filters to delete data in the filters in a timely manner. This achieves repeated data deduplication through multiple filters, improving the accuracy and efficiency of deduplication. Furthermore, it does not require large computing resources, saving computing and maintenance costs and simplifying the calculation steps.

[0076] Based on the above embodiments, the step of repeatedly switching and recycling the main filter and at least two backup filters based on a preset second time threshold to repeatedly perform deduplication on the data and obtain the data deduplication result includes:

[0077] At each preset second time threshold interval, the main filter is switched and the backup filter is recycled, and the new backup filter is set as the target main filter, and a switching identifier, a recycling identifier, and a new target identifier are generated accordingly;

[0078] Based on the switching identifier, the recycling identifier, and the newly set target identifier, the data is deduplicated multiple times to obtain the data deduplication result.

[0079] Specifically, this embodiment provides an implementation method for switching, recycling, and setting a new main filter.

[0080] If filter B is set as the primary filter, then filter A is the backup filter. At fixed intervals, filter B is switched to the backup filter; a signal (switch flag) is generated upon successful switching. Filter A is then recycled; a signal (recycle flag) is generated upon recycling. Filter C is set as the new primary filter (target primary filter); a signal (new target flag) is generated upon setting. Then, using the switch flag, recycling flag, and new target flag, the data is deduplicated multiple times to obtain the deduplicated result.

[0081] The data deduplication method based on rail transit systems provided in this invention involves periodically switching filter B, recycling filter A, setting a new main filter C, and repeating this process multiple times. By using multiple filters in coordination, data in the filters is deleted in a timely manner, thereby achieving data deduplication through repeated use of multiple filters. This improves the accuracy and efficiency of deduplication, and does not require a large amount of computing resources, saving computing and maintenance costs and simplifying the calculation steps.

[0082] Based on the above embodiments, the step of deploying and initializing multiple filters on the master node, generating initialization parameters, and configuring the cluster based on the Redis database includes:

[0083] Multiple filters are deployed and initialized on the master node, the initialization parameters are generated and stored in the Redis database, and cluster configuration is performed based on the target key specified in the Redis database.

[0084] The initialization of the filter by the plurality of slave nodes based on the initialization parameters includes:

[0085] The initialization parameters are retrieved from the Redis database, and multiple filters on the slave node are initialized based on the initialization parameters.

[0086] Specifically, this embodiment provides methods for initializing master nodes and slave nodes.

[0087] The master node deploys and initializes multiple filters. After master node initialization is complete, it generates the relevant initialization parameters and stores them in the Redis database. Then, cluster configuration is performed based on the target key specified in the Redis database.

[0088] In one implementation of initialization on a slave node, multiple filters on the slave node are initialized by calling initialization parameters stored in the Redis database.

[0089] The data deduplication method for rail transit systems provided in this invention initializes filters through the master node and saves the initialization parameters to a Redis database. The cluster configuration is then stored using a target key specified in the Redis database. During slave node initialization, the initialization parameters in the Redis database can be actively invoked to initialize multiple filters on the slave node, thereby achieving distributed deduplication across multiple nodes and improving deduplication accuracy and efficiency.

[0090] Based on the above embodiments, the initialization of the filter by the plurality of slave nodes based on the initialization parameters includes:

[0091] Message notification is performed using open-source message broker software, sending the initialization parameters of the master node to multiple slave nodes;

[0092] The multiple slave nodes listen to the open-source message broker software, receive initialization parameters sent by the master node, and initialize the filter based on the initialization parameters.

[0093] Reference Figure 2 In another implementation where the slave nodes are initialized, the master node sends a message notification through the open-source message broker software (RabbitMQ) to notify all slave nodes of the configuration information.

[0094] The slave node listens to RabbitMQ to obtain configuration and initializes system filter parameters. For slave nodes that miss RabbitMQ messages during startup, they can still read cluster configuration information from the Redis database to perform system initialization.

[0095] The data deduplication method based on rail transit systems provided in this invention involves the master node notifying all slave nodes of configuration information through open-source message broker software. During slave node initialization, the master node can actively call the initialization parameters in the Redis database to initialize multiple filters on the slave node, thereby achieving distributed deduplication through multiple nodes and improving the accuracy and efficiency of deduplication.

[0096] Reference Figure 3 , Figure 3 The flowchart of the data deduplication method based on rail transit systems provided by this invention includes the following steps:

[0097] Step 310: Competitive election of master node;

[0098] Step 320: The master node initializes the Bloom filter and sends a cluster notification;

[0099] Step 330: Initialize the configuration from the node;

[0100] Step 340: Subsequently, compete for master node privileges periodically, change configuration information, and coordinate the cluster.

[0101] The data deduplication device based on rail transit system provided by the present invention is described below. The data deduplication device based on rail transit system described below and the data deduplication method based on rail transit system described above can be referred to and correspond to each other.

[0102] Reference Figure 4 The present invention provides a data deduplication device based on a rail transit system, comprising:

[0103] The node confirmation module 410 is used to confirm the master node and multiple slave nodes of the rail transit system based on a distributed lock mechanism.

[0104] The initialization module 420 is used to deploy and initialize multiple filters on the master node, generate initialization parameters and configure the cluster based on the Redis database, and initialize the filters on the multiple slave nodes based on the initialization parameters;

[0105] The deduplication module 430 is used to perform data deduplication processing based on the multiple filters to obtain the data deduplication result.

[0106] The data deduplication device based on a rail transit system provided in this embodiment of the invention identifies the master node and multiple slave nodes of the rail transit system based on a distributed lock mechanism. Multiple filters are deployed and initialized on the master node, initialization parameters are generated and cluster configuration is performed based on a Redis database, and the filters are initialized on multiple slave nodes based on the initialization parameters. Data deduplication is performed based on multiple filters to obtain the data deduplication result. This achieves deduplication through multi-node distributed deduplication, thereby improving the accuracy and efficiency of deduplication.

[0107] Based on the above embodiments, the device further includes a competition module:

[0108] The competition module is specifically used to: compete for node permissions at preset first time thresholds, identify a target master node among the multiple slave nodes based on a distributed lock mechanism, and replace the master node with the target master node;

[0109] Based on the operating configuration of the target master node, all performance points are coordinated within the cluster.

[0110] Based on the above embodiments, the filter includes a main filter and at least two backup filters;

[0111] The deduplication module is specifically used for:

[0112] Based on the presence of verification data by the main filter and any backup filter, a verification result is obtained, and based on the verification result, it is determined whether the data is duplicated;

[0113] Based on a preset second time threshold, the main filter and at least two backup filters are repeatedly switched and recycled to repeatedly deduplicate the data and obtain the deduplicated data result.

[0114] Based on the above embodiments, the deduplication module is specifically used for:

[0115] At each preset second time threshold interval, the main filter is switched and the backup filter is recycled, and the new backup filter is set as the target main filter, and a switching identifier, a recycling identifier, and a new target identifier are generated accordingly;

[0116] Based on the switching identifier, the recycling identifier, and the newly set target identifier, the data is deduplicated multiple times to obtain the data deduplication result.

[0117] Based on the above embodiments, the node confirmation module is specifically used for:

[0118] Multiple filters are deployed and initialized on the master node, the initialization parameters are generated and stored in the Redis database, and cluster configuration is performed based on the target key specified in the Redis database.

[0119] The initialization module is specifically used for:

[0120] The initialization parameters are retrieved from the Redis database, and multiple filters on the slave node are initialized based on the initialization parameters.

[0121] Based on the above embodiments, the initialization module is specifically used for:

[0122] Message notification is performed using open-source message broker software, sending the initialization parameters of the master node to multiple slave nodes;

[0123] The multiple slave nodes listen to the open-source message broker software, receive initialization parameters sent by the master node, and initialize the filter based on the initialization parameters.

[0124] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a data deduplication method based on the rail transit system, the method including:

[0125] The master node and multiple slave nodes of the rail transit system are confirmed based on a distributed lock mechanism.

[0126] Multiple filters are deployed and initialized on the master node, initialization parameters are generated, and cluster configuration is performed based on the Redis database. The filters are then initialized on the multiple slave nodes based on the initialization parameters.

[0127] Data deduplication is performed based on the multiple filters to obtain the data deduplication result.

[0128] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the data deduplication method based on the rail transit system provided by the above methods, the method comprising:

[0130] The master node and multiple slave nodes of the rail transit system are confirmed based on a distributed lock mechanism.

[0131] Multiple filters are deployed and initialized on the master node, initialization parameters are generated, and cluster configuration is performed based on the Redis database. The filters are then initialized on the multiple slave nodes based on the initialization parameters.

[0132] Data deduplication is performed based on the multiple filters to obtain the data deduplication result.

[0133] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data deduplication method based on a rail transit system provided by the above methods, the method comprising:

[0134] The master node and multiple slave nodes of the rail transit system are confirmed based on a distributed lock mechanism.

[0135] Multiple filters are deployed and initialized on the master node, initialization parameters are generated, and cluster configuration is performed based on the Redis database. The filters are then initialized on the multiple slave nodes based on the initialization parameters.

[0136] Data deduplication is performed based on the multiple filters to obtain the data deduplication result.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data deduplication method based on a rail transit system, characterized in that, The method comprises the steps of: confirming a master node and a plurality of slave nodes of a rail transit system based on a distributed lock mechanism; deploying a plurality of filters on the master node and initializing the filters, generating initialization parameters and performing cluster configuration based on a Redis database, and initializing the filters based on the initialization parameters on the plurality of slave nodes; performing data deduplication processing based on the plurality of filters to obtain a data deduplication result; the filters comprise a master filter and at least two backup filters; the data deduplication processing based on the plurality of filters to obtain a data deduplication result comprises: checking whether data exists based on the master filter and any backup filter to obtain a checking result, and determining whether the data is duplicated based on the checking result; repeatedly switching and recycling the master filter and the at least two backup filters based on a preset second time threshold to repeatedly perform data deduplication processing to obtain a data deduplication result; after the initialization of the filters based on the initialization parameters on the plurality of slave nodes, the method further comprises: timing a node authority based on a preset first time threshold, confirming a target master node based on a distributed lock mechanism in the plurality of slave nodes, and replacing the master node with the target master node; performing internal cluster coordination of all nodes based on the running configuration of the target master node; the repeatedly switching and recycling the master filter and the at least two backup filters based on a preset second time threshold to repeatedly perform data deduplication processing to obtain a data deduplication result comprises: switching the master filter and recycling the backup filter every interval of the preset second time threshold, setting a new backup filter as a target master filter, and correspondingly generating a switching identifier, a recycling identifier, and a new target identifier; performing multiple times of data deduplication processing based on the switching identifier, the recycling identifier, and the new target identifier to obtain the data deduplication result.

2. The method of claim 1, wherein, the deploying a plurality of filters on the master node and initializing the filters, generating initialization parameters and performing cluster configuration based on a Redis database comprises: deploying a plurality of filters on the master node and initializing the filters, generating the initialization parameters and storing the initialization parameters in a Redis database, and performing cluster configuration based on a target key specified by the Redis database; the initializing the filters based on the initialization parameters on the plurality of slave nodes comprises: calling the initialization parameters from the Redis database, and initializing a plurality of filters of the slave nodes based on the initialization parameters.

3. The method of claim 1, wherein, the initializing the filters based on the initialization parameters on the plurality of slave nodes comprises: performing message notification based on open source message broker software, and sending the initialization parameters of the master node to a plurality of slave nodes; listening to the open source message broker software by the plurality of slave nodes, receiving the initialization parameters sent by the master node, and initializing the filters based on the initialization parameters.

4. A data deduplication device based on a rail transit system, characterized in that, The method comprises the steps of: A node confirmation module is configured to confirm a master node of a rail transit system and a plurality of slave nodes based on a distributed lock mechanism. An initialization module is configured to deploy a plurality of filters and perform initialization on the master node, generate initialization parameters, and perform cluster configuration based on a Redis database, and initialize the filters based on the initialization parameters on the plurality of slave nodes. A deduplication module is configured to perform data deduplication processing based on the plurality of filters to obtain a data deduplication result. The filters include a master filter and at least two backup filters. The data deduplication processing based on the plurality of filters to obtain a data deduplication result includes: checking whether data exists based on the master filter and any backup filter to obtain a checking result, and determining whether the data is duplicated based on the checking result; and repeatedly switching and recycling the master filter and the at least two backup filters based on a preset second time threshold to repeatedly perform data deduplication processing to obtain a data deduplication result. After the initialization of the filters based on the initialization parameters on the plurality of slave nodes, the method further includes: timing node authority based on a preset first time threshold, confirming a target master node based on a distributed lock mechanism among the plurality of slave nodes, replacing the master node with the target master node, and performing cluster internal coordination on all nodes based on a running configuration of the target master node. The repeatedly switching and recycling of the master filter and the at least two backup filters based on the preset second time threshold to repeatedly perform data deduplication processing to obtain a data deduplication result includes: switching the master filter and recycling the backup filters every interval of the preset second time threshold, setting a new backup filter as a target master filter, and correspondingly generating a switching identifier, a recycling identifier, and a newly set target identifier; and performing multiple times of data deduplication processing based on the switching identifier, the recycling identifier, and the newly set target identifier to obtain the data deduplication result.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the data deduplication method based on a rail transit system according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the data deduplication method based on a rail transit system according to any one of claims 1 to 3.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the data deduplication method based on a rail transit system according to any one of claims 1 to 3.

Images