Batch data updating method based on Redis and md5

Abstract

The invention particularly relates to a batch data updating method based on Redis and md5. According to the batch data updating method based on Redis and md5, data of all departments are collected into a temporary table, md5 values of the data of all the departments are generated, and the md5 values and UUIDs are stored in a Redis cache together; reading newly collected temp table data in the Redis cache in batches, comparing the temp table data with current formal table data according to departments, and finding out deleted data, newly added data and modified data according to md5 values of the data; if the data change rate is not lower than the self-defined threshold value, batch updating is conducted on the current formal table according to newly-collected temp table data in the Redis cache. According to the batch data updating method based on the Redis and the md5, rapid reading and caching of the data are achieved, the number of access times of a back-end database is remarkably reduced, the burden of a database server is effectively relieved, the response speed and the data processing efficiency of the whole system are improved, the safety and the integrity of the data are ensured, and the data updating efficiency is improved. The problem of system data loss during total collection of the database is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a batch data updating method based on Redis and md5. Background Art

[0002] Database management systems have become an important part of software today. Open source databases such as MySQL and PostgreSQL as well as commercial databases such as Oracle are now ubiquitous, and almost all services rely on database management systems to store data.

[0003] Persistence is the most basic guarantee of a database, but in reality it is very difficult to ensure that data is not lost. For example, MongoDB database has not been able to guarantee persistence for a long time in the past, and data is easily lost; RocksDB DeleteRange function causes data loss problems, etc.

[0004] Whether it is an open source database or a service provided by a cloud service provider, data loss may occur. Operation and maintenance and configuration errors caused by human factors are the primary reasons for database data loss. Damage to the disk used to store database data can also cause data loss. In addition, the functions and implementation of the database are complex, and there is a risk of data loss if it is not flushed to the disk in time.

[0005] Based on the above situation, the present invention proposes a batch data update method based on Redis and md5. Summary of the invention

[0006] In order to make up for the defects of the prior art, the present invention provides a simple and efficient batch data update method based on Redis and md5.

[0007] The present invention is achieved through the following technical solutions:

[0008] A batch data update method based on Redis and md5, characterized in that it includes the following steps:

[0009] Step S1, first collect the data of each department into a temporary table, use the MD5 encryption algorithm to perform hash operation on the data of each department, generate a hash value of fixed length, that is, the md5 value of the data of each department, and store it in the Redis cache together with the universally unique identifier UUID (Universally Unique Identifier) ​​in the form of a key-value pair;

[0010] Step S2, using Redis batch operation commands or APIs to batch read the newly collected temp table data (temporary files or data) in the Redis cache, and then compare them with the current formal table data by department, and find out the deleted data, newly added data, and modified data according to the md5 value of the data;

[0011] If the data comparison is successful, the comparison task result data is generated and the modification status is successful;

[0012] If the comparison fails, the reason for the failure is recorded and the status is recorded as failure;

[0013] Step S3, calculating the data change rate according to the comparison result, if the data change rate is lower than the custom threshold, it is determined that the data has been synchronized, otherwise it is considered that the data has not been synchronized;

[0014] If the data is not synchronized, the deleted data and the newly added data found in step S2 are deleted from the current formal table, and the newly added data and the modified data are batch read from the newly collected temp table in the Redis cache using Redis batch operation commands or APIs, and written into the current formal table using database batch insert or update statements.

[0015] In step S2, first determine whether the md5 data in the current formal database can be obtained from the Redis cache:

[0016] If the Redis cache does not contain the md5 data in the current official database, determine whether there is data in the current official table:

[0017] If there is no data in the current formal table, all data in the temp table will be synchronized to the current formal table, and an md5 value will be generated for the current formal table and stored in the Redis cache together with the universal unique identifier UUID;

[0018] Then determine whether the md5 value of the newly collected temp table data can be obtained from the Redis cache:

[0019] If there is no md5 data of the newly collected temp table data in the Redis cache, the md5 value generated by the temp table is stored in the Redis cache, and then compared by department to find out the deleted data, newly added data, and modified data.

[0020] In step S2, the current data of a department is taken out and compared with the newly collected data, and the difference set is calculated using the current data and the newly collected data to find out the deleted data and the newly added data;

[0021] Use the newly collected data and the current data to find the intersection, and determine whether the md5 value corresponding to the universal unique identifier UUID of the data in the intersection is the same. If they are the same, no processing will be done. If they are different, the data will be regarded as modified.

[0022] A batch data update system based on Redis and md5, including a Redis cache module, an MD5 encryption module, a batch data reading module and a batch data update module;

[0023] The Redis cache module is responsible for storing the md5 value of each department's data and the universal unique identifier UUID in the form of a key-value pair and providing a fast access mechanism;

[0024] The MD5 encryption module is responsible for performing hash operations on the data of each department using the MD5 encryption algorithm to generate a hash value of a fixed length, i.e., the md5 value of the data of each department;

[0025] The batch data reading module is responsible for batch reading encrypted data from the Redis cache using Redis batch operation commands or APIs;

[0026] The data update module is responsible for comparing the newly collected temp table data in the Redis cache with the current formal table data by department, finding out the deleted data, newly added data and modified data according to the md5 value of the data; and when the data change rate reaches or exceeds the custom threshold, deleting the deleted data and newly added data from the current formal table, using Redis batch operation commands or APIs to batch read the newly added data and modified data from the newly collected temp table in the Redis cache, and using database batch insert or update statements to write them into the current formal table.

[0027] When the data update module compares data by department, it first determines whether the md5 data in the current official database can be obtained from the Redis cache:

[0028] If the Redis cache does not contain the md5 data in the current official database, determine whether there is data in the current official table:

[0029] If there is no data in the current formal table, all data in the temp table will be synchronized to the current formal table, and an md5 value will be generated for the current formal table and stored in the Redis cache together with the universal unique identifier UUID;

[0030] Then determine whether the md5 value of the newly collected temp table data can be obtained from the Redis cache:

[0031] If there is no md5 data of the newly collected temp table data in the Redis cache, the md5 value generated by the temp table is stored in the Redis cache, and then compared by department to find out the deleted data, newly added data, and modified data.

[0032] The data update module takes out the current data of a department and compares it with the newly collected data, uses the current data and the newly collected data to find the difference set, and finds the deleted data and the newly added data;

[0033] Use the newly collected data and the current data to find the intersection, and determine whether the md5 value corresponding to the universal unique identifier UUID of the data in the intersection is the same. If they are the same, no processing will be done. If they are different, the data will be regarded as modified.

[0034] A batch data update device based on Redis and md5, characterized by comprising a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0035] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0036] The beneficial effects of the present invention are as follows: the batch data update method based on Redis and md5 utilizes the memory storage advantage of Redis to realize fast reading and caching of data, thereby significantly reducing the number of accesses to the back-end database, effectively reducing the burden on the database server, improving the response speed and data processing efficiency of the entire system, effectively preventing data from being illegally obtained or tampered with during transmission or storage, ensuring the security and integrity of the data, and solving the problem of system data loss when the database is fully collected. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Attached Figure 1 This is a schematic diagram of a batch data update method based on Redis and md5 in the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] The batch data update method based on Redis and md5 includes the following steps:

[0041] Step S1, first collect the data of each department into a temporary table, use the MD5 encryption algorithm to perform hash operation on the data of each department, generate a hash value of fixed length, that is, the md5 value of the data of each department, and store it in the Redis cache together with the universally unique identifier UUID (Universally Unique Identifier) ​​in the form of a key-value pair, and use the fast reading and writing capabilities of the Redis cache to achieve fast access to data;

[0042] Step S2, using Redis batch operation commands or APIs to batch read the newly collected temp table data (temporary files or data) in the Redis cache, and then compare them with the current formal table data by department, and find out the deleted data, newly added data, and modified data according to the md5 value of the data;

[0043] If the data comparison is successful, the comparison task result data is generated and the modification status is successful;

[0044] If the comparison fails, the reason for the failure is recorded and the status is recorded as failure;

[0045] Step S3, calculating the data change rate according to the comparison result, if the data change rate is lower than the custom threshold, it is determined that the data has been synchronized, otherwise it is considered that the data has not been synchronized;

[0046] If the data is not synchronized, the deleted data and the newly added data found in step S2 are deleted from the current formal table, and the newly added data and the modified data are batch read from the newly collected temp table in the Redis cache using Redis batch operation commands or APIs, and written into the current formal table using database batch insert or update statements.

[0047] In step S2, first determine whether the md5 data in the current formal database can be obtained from the Redis cache:

[0048] If the Redis cache does not contain the md5 data in the current official database, determine whether there is data in the current official table:

[0049] If there is no data in the current formal table, all data in the temp table will be synchronized to the current formal table, and an md5 value will be generated for the current formal table and stored in the Redis cache together with the universal unique identifier UUID;

[0050] Then determine whether the md5 value of the newly collected temp table data can be obtained from the Redis cache:

[0051] If there is no md5 data of the newly collected temp table data in the Redis cache, the md5 value generated by the temp table is stored in the Redis cache, and then compared by department to find out the deleted data, newly added data, and modified data.

[0052] In step S2, the current data of a department is taken out and compared with the newly collected data, and the difference set is calculated using the current data and the newly collected data to find out the deleted data and the newly added data;

[0053] Use the newly collected data and the current data to find the intersection, and determine whether the md5 value corresponding to the universal unique identifier UUID of the data in the intersection is the same. If they are the same, no processing will be done. If they are different, the data will be regarded as modified.

[0054] The batch data update system based on Redis and md5 includes a Redis cache module, an MD5 encryption module, a batch data reading module and a batch data update module;

[0055] The Redis cache module is responsible for storing the md5 value of each department's data and the universally unique identifier UUID (Universally Unique Identifier) ​​in the form of a key-value pair and providing a fast access mechanism;

[0056] The MD5 encryption module is responsible for performing hash operations on the data of each department using the MD5 encryption algorithm to generate a hash value of a fixed length, i.e., the md5 value of the data of each department;

[0057] The batch data reading module is responsible for batch reading encrypted data from the Redis cache using Redis batch operation commands or APIs;

[0058] The data update module is responsible for comparing the newly collected temp table data in the Redis cache with the current formal table data by department, finding out the deleted data, newly added data and modified data according to the md5 value of the data; and when the data change rate reaches or exceeds the custom threshold, deleting the deleted data and newly added data from the current formal table, using Redis batch operation commands or APIs to batch read the newly added data and modified data from the newly collected temp table in the Redis cache, and using database batch insert or update statements to write them into the current formal table.

[0059] When the data update module compares data by department, it first determines whether the md5 data in the current official database can be obtained from the Redis cache:

[0060] If the Redis cache does not contain the md5 data in the current official database, determine whether there is data in the current official table:

[0061] If there is no data in the current formal table, all data in the temp table will be synchronized to the current formal table, and an md5 value will be generated for the current formal table and stored in the Redis cache together with the universal unique identifier UUID;

[0062] Then determine whether the md5 value of the newly collected temp table data can be obtained from the Redis cache:

[0063] If there is no md5 data of the newly collected temp table data in the Redis cache, the md5 value generated by the temp table is stored in the Redis cache, and then compared by department to find out the deleted data, newly added data, and modified data.

[0064] The data update module takes out the current data of a department and compares it with the newly collected data, uses the current data and the newly collected data to find the difference set, and finds the deleted data and the newly added data;

[0065] Use the newly collected data and the current data to find the intersection, and determine whether the md5 value corresponding to the universal unique identifier UUID of the data in the intersection is the same. If they are the same, no processing will be done. If they are different, the data will be regarded as modified.

[0066] The batch data update device based on Redis and md5 includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0067] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0068] This batch data update method based on Redis and md5 uses the memory storage advantage of Redis to achieve fast reading and caching of data, thereby significantly reducing the number of accesses to the back-end database, effectively reducing the burden on the database server, and improving the response speed and data processing efficiency of the entire system. It effectively prevents data from being illegally obtained or tampered with during transmission or storage, ensures the security and integrity of the data, and solves the problem of system data loss when the database is fully collected.

[0069] This batch data update method based on Redis and md5 can avoid the full clear and full collection operation of the database, reduce the system's dependence on data, and improve the flexibility of resource data update. At the same time, this method is also applicable to a variety of data collection scenarios, which helps to improve the efficiency of production data update and is a technical solution with important application value and prospects.

[0070] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A batch data update method based on Redis and md5, characterized in that: The following steps are involved: Step S1, first collect the data of each department into a temporary table, use the MD5 encryption algorithm to perform a hash operation on the data of each department, generate a hash value of a fixed length, that is, the md5 value of the data of each department, and store it in the Redis cache together with the universal unique identification code UUID in the form of a key-value pair; Step S2, using Redis batch operation commands or APIs to batch read the newly collected temp table data in the Redis cache, and then compare it with the current formal table data by department, and find out the deleted data, newly added data, and modified data according to the md5 value of the data; If the data comparison is successful, the comparison task result data is generated and the modification status is successful; If the comparison fails, the reason for the failure is recorded and the status is recorded as failure; Step S3, calculating the data change rate according to the comparison result, if the data change rate is lower than the custom threshold, it is determined that the data has been synchronized, otherwise it is considered that the data has not been synchronized; If the data is not synchronized, the deleted data and the newly added data found in step S2 are deleted from the current formal table, and the newly added data and the modified data are batch read from the newly collected temp table in the Redis cache using Redis batch operation commands or APIs, and written into the current formal table using database batch insert or update statements.

2. The batch data update method based on Redis and md5 according to claim 1 is characterized in that: In step S2, first determine whether the md5 data in the current formal database can be obtained from the Redis cache: If the Redis cache does not contain the md5 data in the current official database, determine whether there is data in the current official table: If there is no data in the current formal table, all data in the temp table will be synchronized to the current formal table, and an md5 value will be generated for the current formal table and stored in the Redis cache together with the universal unique identifier UUID; Then determine whether the md5 value of the newly collected temp table data can be obtained from the Redis cache: If there is no md5 data of the newly collected temp table data in the Redis cache, the md5 value generated by the temp table is stored in the Redis cache, and then compared by department to find out the deleted data, newly added data, and modified data.

3. The batch data update method based on Redis and md5 according to claim 2 is characterized in that: In step S2, the current data of a department is taken out and compared with the newly collected data, and the difference set is calculated using the current data and the newly collected data to find out the deleted data and the newly added data; Use the newly collected data and the current data to find the intersection, and determine whether the md5 value corresponding to the universal unique identifier UUID of the data in the intersection is the same. If they are the same, no processing will be done. If they are different, the data will be regarded as modified.

4. A batch data update system based on Redis and md5, characterized by: Including Redis cache module, MD5 encryption module, batch data reading module and batch data update module; The Redis cache module is responsible for storing the md5 value of each department's data and the universal unique identifier UUID in the form of a key-value pair and providing a fast access mechanism; The MD5 encryption module is responsible for performing hash operations on the data of each department using the MD5 encryption algorithm to generate a hash value of a fixed length, i.e., the md5 value of the data of each department; The batch data reading module is responsible for batch reading encrypted data from the Redis cache using Redis batch operation commands or APIs; The data update module is responsible for comparing the newly collected temp table data in the Redis cache with the current formal table data by department, and finding out the deleted data, newly added data and modified data according to the md5 value of the data; When the data change rate reaches or exceeds the custom threshold, the deleted data and the newly added data are deleted from the current formal table, and the newly added data and the modified data are read in batches from the newly collected temp table in the Redis cache using Redis batch operation commands or APIs, and written to the current formal table using database batch insert or update statements.

5. The batch data update system based on Redis and md5 according to claim 4 is characterized in that: When the data update module compares data by department, it first determines whether the md5 data in the current official database can be obtained from the Redis cache: If the Redis cache does not contain the md5 data in the current official database, determine whether there is data in the current official table: If there is no data in the current formal table, all data in the temp table will be synchronized to the current formal table, and an md5 value will be generated for the current formal table and stored in the Redis cache together with the universal unique identifier UUID; Then determine whether the md5 value of the newly collected temp table data can be obtained from the Redis cache: If there is no md5 data of the newly collected temp table data in the Redis cache, the md5 value generated by the temp table is stored in the Redis cache, and then compared by department to find out the deleted data, newly added data, and modified data.

6. The batch data update system based on Redis and md5 according to claim 5 is characterized in that: The data update module takes out the current data of a department and compares it with the newly collected data, uses the current data and the newly collected data to find the difference set, and finds the deleted data and the newly added data; Use the newly collected data and the current data to find the intersection, and determine whether the md5 value corresponding to the universal unique identifier UUID of the data in the intersection is the same. If they are the same, no processing will be done. If they are different, the data will be regarded as modified.

7. A batch data update device based on Redis and md5, characterized by: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method steps as claimed in any one of claims 1 to 3 when executing the computer program.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 3 are implemented.

Images