Data processing method, device, equipment, storage medium and program product

Abstract

The present application discloses a data processing method, a data processing device, a data processing equipment, a storage medium and a program product, and relates to the field of data storage technology. In the present application, when a client initiates a data change request to start a database transaction, adaptive data synchronization is performed according to the data type of the change data in response to the data change request, and the configuration data is transferred to the Redis cache to improve data access efficiency, while maintaining the persistent storage of the configuration data in the MySQL database, and synchronizing the business data to the MySQL database, so as to perform adaptive data synchronization according to the data type and improve data synchronization efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data processing method, a data processing apparatus, a data processing device, a storage medium and a program product. Background Art

[0002] At present, the combination of MySQL and Redis commonly used in database systems has a relatively common data consistency problem. Many systems use various solutions to maintain data consistency and real-time synchronization between MySQL and Redis, such as write-through cache and cache bypass mode. However, existing technical solutions are usually only one-way synchronization or simple data overwrite, or use scheduled tasks to perform regular full comparisons. The above solutions are not very targeted to data types, resulting in low data synchronization efficiency.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a data processing method, a data processing device, a data processing equipment, a storage medium and a program product, aiming to solve the technical problem of low data synchronization efficiency.

[0005] To achieve the above purpose, the present application proposes a data processing method, which includes:

[0006] When the client initiates a data change request, determining that the change data in response to the data change request is configuration data or business data;

[0007] If the changed data is configuration data, after successfully synchronizing the changed data to the Redis cache, the changed data is synchronized to the MySQL database;

[0008] If the changed data is business data, the business data is synchronized to the MySQL database.

[0009] In one embodiment, the data processing method further includes:

[0010] When starting the client and loading a series of startup data tables, it is determined that the current data table to be loaded stores configuration data or business data;

[0011] If the configuration data is saved, determine whether the data to be loaded exists in the Redis cache; if the data to be loaded exists in the Redis cache, load the data to be loaded from the Redis cache; if the data to be loaded does not exist in the Redis cache, load the data to be loaded from the MySQL database;

[0012] If the business data is saved, the data to be loaded is loaded from the MySQL database.

[0013] In one embodiment, the data processing method further includes:

[0014] When performing data consistency detection, the configuration data table to be verified is read from the Redis cache and the MySQL database respectively;

[0015] Based on the data records in the configuration data table, data is repaired for the Redis cache and MySQL database.

[0016] In one embodiment, the step of repairing data on the Redis cache and the MySQL database based on the data records of the configuration data table includes:

[0017] Compare whether the field information contained in the data records of the configuration data table is the same;

[0018] When the field information is not the same, determine whether the version numbers of the data records in the configuration data table are the same;

[0019] According to the version number comparison result, data repair of the data records of the configuration data table is performed between the Redis cache and the MySQL database.

[0020] In one embodiment, the step of repairing data records of the configuration data table between the Redis cache and the MySQL database according to the version number comparison result includes:

[0021] When the version numbers are the same, synchronize the data records in the Redis cache with the data records in the MySQL database;

[0022] When the version numbers are different, the data records with smaller version numbers between the Redis cache and the MySQL database are synchronized with the data records with larger version numbers.

[0023] In one embodiment, the step of comparing whether the field information contained in the data records of the configuration data table is the same includes:

[0024] Compare whether the data modification version numbers in the data records of the configuration data table are the same;

[0025] When the data modification version numbers are the same, the data in the configuration data table is standardized, and the character hash values ​​after the standardization are compared to see whether they are the same.

[0026] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:

[0027] A determination module, used to determine whether the changed data in response to the data change request is configuration data or business data when the client initiates a data change request;

[0028] A first synchronization module, configured to synchronize the changed data to a MySQL database after successfully synchronizing the changed data to a Redis cache if the changed data is configuration data;

[0029] The second synchronization module is used to synchronize the business data to the MySQL database if the changed data is business data.

[0030] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0031] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0033] One or more technical solutions proposed in this application have at least the following technical effects:

[0034] In this application, when the client initiates a data change request to start a database transaction, adaptive data synchronization is performed according to the data type of the change data in response to the data change request, and the configuration data is transferred to the Redis cache to improve data access efficiency. At the same time, the configuration data is maintained in the MySQL database for persistent storage, and the business data is synchronized to the MySQL database, so as to perform adaptive data synchronization according to the data type and improve data synchronization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0038] Figure 2 A schematic diagram of a data change process provided in Example 1 of the data processing method of this application;

[0039] Figure 3 A schematic diagram of a data synchronization process provided for the second embodiment of the data processing method of the present application;

[0040] Figure 4 A schematic diagram of a data detection and repair process provided in Embodiment 3 of the data processing method of the present application;

[0041] Figure 5 A schematic diagram of the system architecture of the data processing method of this application;

[0042] Figure 6 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;

[0043] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0044] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0046] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0047] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data processing device, etc. The following takes a data processing device as an example to illustrate this embodiment and the following embodiments.

[0048] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1This is a flowchart of the first embodiment of the data processing method of the present application.

[0049] In this embodiment, the data processing method includes steps S10 to S30:

[0050] Step S10, when the client initiates a data change request, determining whether the changed data in response to the data change request is configuration data or business data;

[0051] It should be noted that the data change request includes modification operations such as adding, deleting, and modifying data, and the changed data in response to the data change request is the configuration data or business data after the modification operations such as adding, deleting, and modifying are performed.

[0052] Step S20: if the changed data is configuration data, after successfully synchronizing the changed data to the Redis cache, synchronizing the changed data to the MySQL database;

[0053] In one embodiment, after the configuration data is successfully synchronized to the Redis cache, the configuration data is synchronized to the MySQL database.

[0054] In another embodiment, after the configuration data is successfully synchronized to the MySQL database, the configuration data is synchronized to the Redis cache.

[0055] Compared with many current systems that only support one-way synchronization from MySQL database to Redis cache and cannot meet the demand of synchronizing changed data in Redis cache back to MySQL database, this embodiment supports two-way synchronization of configuration data between the two, not only supporting synchronization from MySQL database to Redis cache, but also supporting synchronization from Redis cache to MySQL database, ensuring two-way consistency of data.

[0056] Step S30: If the changed data is business data, the business data is synchronized to the MySQL database.

[0057] It should be noted that as the amount of data increases, frequent I / O access will cause operation delays. Therefore, the configuration data is transferred to the Redis cache to improve data access efficiency while maintaining persistent storage of the configuration data in the MySQL database.

[0058] In this embodiment, when the client initiates a data change request to start a database transaction, adaptive data synchronization is performed according to the data type of the change data in response to the data change request, and the configuration data is transferred to the Redis cache to improve data access efficiency. At the same time, the configuration data is maintained in a persistent storage in the MySQL database, and the business data is synchronized to the MySQL database, so as to perform adaptive data synchronization according to the data type and improve data synchronization efficiency.

[0059] In one embodiment, referring to Figure 2 ,Step 101: The client initiates a data change request;

[0060] Step 102: Start a database transaction;

[0061] Step 103: Determine whether the change operation is a modification operation;

[0062] Step 104: If it is a modification operation, increase the version number of the modified version in the data record by 1;

[0063] Step 105: If it is not a modification operation, determine whether the change operation is an operation of adding a data record;

[0064] Step 106: If it is an operation of adding a data record, the version number of the data record is initialized to 1;

[0065] Step 107: If it is not an operation to add a data record, determine whether the changed data is configuration data, and the changed data is the data to be changed corresponding to the data change request;

[0066] Step 108: If it is configuration data, synchronize the data to the Redis cache. If not, execute step 111;

[0067] Step 109: Determine whether the data record is successfully synchronized to the Redis cache. If yes, execute step 111, otherwise execute step 110. A solution for determining whether the data record is successfully synchronized to the Redis cache is to determine whether a series of commands for executing Redis data synchronization all return success. If there is one command that returns failure, it means that the synchronization is failed, and then the command that returns failure is manually rolled back to the state before synchronization.

[0068] Step 110: Determine whether the number of retries to synchronize to the Redis cache has reached n times. If yes, execute step 114. If not, return to step 108 and make n=n+1.

[0069] Step 111: The MySQL database performs a synchronous change operation, that is, writes and saves the business data into the MySQL database, or synchronously writes the configuration data successfully written into the Redis cache into the MySQL database;

[0070] Step 112: Determine whether the MySQL database successfully executes the data record synchronization operation. If yes, end the process. If no, execute step 113.

[0071] Step 113: The MySQL database performs a rollback operation to restore the data record status before the modification operation;

[0072] Step 114: The client throws an exception and records the exception information to the producer message queue;

[0073] Step 115: The abnormal consumer thread retrieves the abnormal information from the message queue, displays a pop-up window to prompt the abnormal information, and records it in the system alarm.

[0074] In the above data change operation, the client initiates a data change request, starts a database transaction, adjusts the version number of the data record according to the change type, and synchronizes it to the Redis cache, and proposes a retry logic for handling synchronization failures to ensure data consistency. At the same time, fine-grained control is performed, that is, through the version number and transaction mechanism, to ensure that each data change can be accurately synchronized and detected to reduce errors.

[0075] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. On this basis, the data processing method also includes:

[0076] When starting the client and loading a series of startup data tables, it is determined that the current data table to be loaded stores configuration data or business data;

[0077] If the configuration data is saved, determine whether the data to be loaded exists in the Redis cache; if the data to be loaded exists in the Redis cache, load the data to be loaded from the Redis cache; if the data to be loaded does not exist in the Redis cache, load the data to be loaded from the MySQL database;

[0078] If the business data is saved, the data to be loaded is loaded from the MySQL database.

[0079] It should be noted that the startup data table loaded when the client is started is an initialization data table required for the client, such as the host computer software, to start.

[0080] In the first embodiment, after the configuration data is transferred to the Redis cache, when the client is started and the startup data table is loaded, it is first determined whether the data to be loaded in the current data table is configuration data or business data. If it is configuration data, it is directly determined in the Redis cache whether there is data to be loaded, that is, the configuration data; when there is data to be loaded in the Redis cache, the data to be loaded is directly loaded from the Redis cache; when there is no data to be loaded in the Redis cache, the data to be loaded is loaded from the MySQL database. If it is business data, the data to be loaded, that is, the business data, is directly loaded from the MySQL database. In this way, data access efficiency is improved.

[0081] In one embodiment, referring to Figure 3 , Step 201: The client starts and loads the startup data table;

[0082] Step 202: Check whether the current data table belongs to configuration data, that is, check whether the data type stored in the data table to be loaded is configuration data; if yes, go to step 203, if not, go to step 207;

[0083] Step 203: Check whether the key corresponding to data table i exists in the Redis cache. If so, go to step 206; if not, go to step 204;

[0084] Step 204: passing the miss configuration table i into the cache synchronization message queue;

[0085] Step 205: Load configuration data from the MySQL database, and then store the data information in the memory; that is, record the table information, and load and synchronize it from the MySQL database to the Redis cache through steps 209-211; this is also equivalent to data synchronization from the MySQL database to the Redis cache;

[0086] Step 206: Load the configuration table i cache information from the Redis cache;

[0087] Step 207: Load data i from the MySQL database;

[0088] Step 208: Determine whether the client has completed loading all startup data tables. If yes, go to step 209; if no, go to step 202 and make n=n+1;

[0089] Step 209: Start the cache synchronization task thread;

[0090] Step 210: fetching the identifier of the to-be-cached synchronization configuration table from the cache synchronization message queue;

[0091] Step 211: Synchronize the configuration data information stored in the memory to the Redis cache.

[0092] In the above, when the client starts, it loads the startup data table, checks the data table type, decides to load data from the Redis cache or MySQL database, starts the cache synchronization task, and synchronizes the configuration information in the memory to the Redis cache.

[0093] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. On this basis, the data processing method also includes:

[0094] When performing data consistency detection, the configuration data table to be verified is read from the Redis cache and the MySQL database respectively;

[0095] Based on the data records in the configuration data table, data is repaired for the Redis cache and MySQL database.

[0096] Currently, the data consistency check of Redis cache and MySQL database is not comprehensive. Existing solutions mostly use regular full comparison, which is inefficient and unable to detect and repair data inconsistencies in a timely manner.

[0097] In this embodiment, a repair technology for data management is proposed, particularly a method for data consistency check and repair between business data and configuration data stored in a MySQL relational database and a Redis cache database in a large client. According to the data records of the configuration data table to be verified read from the Redis cache and the MySQL database respectively, data repair is performed on the Redis cache and the MySQL database to ensure the consistency of the data in the two databases and improve data access performance.

[0098] In a feasible implementation manner, the step of repairing data in the Redis cache and the MySQL database based on the data records in the configuration data table includes:

[0099] Compare whether the field information contained in the data records of the configuration data table is the same;

[0100] When the field information is not the same, determine whether the version numbers of the data records in the configuration data table are the same;

[0101] According to the version number comparison result, data repair of the data records of the configuration data table is performed between the Redis cache and the MySQL database.

[0102] In a feasible implementation manner, the step of comparing whether the field information contained in the data records of the configuration data table is the same includes:

[0103] Compare whether the data modification version numbers in the data records of the configuration data table are the same;

[0104] When the data modification version numbers are the same, the data in the configuration data table is standardized, and the character hash values ​​after the standardization are compared to see whether they are the same.

[0105] In one embodiment, the field information contained in n data records of two data sources is compared one by one to see if they are the same. A preferred implementation scheme for comparing data records is: first compare whether the version numbers of the version number field in the data record are the same, and if they are not the same, directly return different. If they are the same, continue to standardize the data, such as removing spaces from the string field, converting the numeric field into a string with a fixed number of decimal places, and converting the time and date field into a unified time format string. Secondly, the values ​​of all converted fields are spliced ​​into a string in a fixed order, and finally a hash algorithm (such as SHA-256) is used to calculate the hash value of the spliced ​​string to compare whether the hash values ​​of the data records of the two data sources are the same. The advantage of this scheme of calculating the data hash value and then comparing it is that the comparison of the hash value only requires one calculation and one comparison, while the comparison of the fields requires multiple comparison operations. When the amount of data is large, the overall efficiency of calculating the hash value is higher. Moreover, the hash value method integrates the comparison of all fields into a single operation, simplifies the logical process, and the comparison of fields one by one needs to process the differences of each field type.

[0106] In a feasible implementation, the step of repairing data records of the configuration data table between the Redis cache and the MySQL database according to the version number comparison result includes:

[0107] When the version numbers are the same, synchronize the data records in the Redis cache with the data records in the MySQL database;

[0108] When the version numbers are different, the data records with smaller version numbers between the Redis cache and the MySQL database are synchronized with the data records with larger version numbers.

[0109] In one embodiment, if the version numbers are the same, the data records in the Redis cache are deleted and the data records in the MySQL cache are synchronized to the Redis cache according to the data source priority, that is, the MySQL data source data is the main one and the Redis cache data source data is the auxiliary one. If the version numbers are different, the data record on the side with the larger version number is taken and synchronized to the data source on the other side, that is, the data record on the side with the smaller version number of the Redis cache and the MySQL database is synchronized to the data record on the side with the larger version number.

[0110] In one embodiment, referring to Figure 4 ,Step 301: Add optimistic locking to the database processing to ensure that the data will not be modified during the consistency detection process, and to ensure that the data consistency detection and repair process will not be disturbed. Optimistic locking is a concurrency control mechanism commonly used in data change operations. It assumes that the probability of data conflict is very low when multiple transactions operate concurrently. When updating data, first check whether the data has been modified by other transactions. If not, perform the update operation. The advantage of optimistic locking is that it reduces the lock overhead and improves concurrency performance.

[0111] Step 302: Design a data consistency verification script according to specific business requirements, and write a corresponding verification program to facilitate subsequent automated verification.

[0112] Step 303: Read the configuration data tables that need to be verified from the MySQL and Redis databases respectively to prepare for subsequent data consistency detection.

[0113] Step 304: Compare the number of records in the configuration data table in MySQL and Redis, that is, the number of data records, to determine whether the number of records in the two is consistent. If the number of records is different, proceed to step 314; if the number of records is the same, proceed to step 305.

[0114] Step 305 : Determine whether the configuration data table i is a key data table. If yes, go to step 306 ; if no, go to step 307 .

[0115] Step 306: Perform a full comparison on all data records in the table.

[0116] Step 307: Use a simple random sampling strategy to select several segments of data records in the configuration data table i. An equivalent solution: For non-critical data tables with a small amount of data, full sampling can be used; an alternative solution: randomly select several segments of data records in the configuration data table i.

[0117] Step 308: Format the data records of the extracted configuration data table i of the two data sources into a unified structure type.

[0118] Step 309: Compare the field information of the n data records of the two data sources one by one to see if they are the same. If yes, execute step 315; if no, mark the different data records and execute step 310. A preferred implementation scheme for comparing data records is: first compare whether the version numbers of the version number fields in the data records are the same, and if they are not the same, directly return different. If they are the same, continue to standardize the data, such as pre-processing the string field by removing spaces, converting the numeric field into a string with a fixed number of decimal places, and converting the time and date field into a unified time format string. Secondly, concatenate the values ​​of all converted fields into a string in a fixed order, and finally use a hash algorithm (such as SHA-256) to calculate the hash value of the concatenated string, and compare whether the hash values ​​of the data records of the two data sources are the same. The advantage of this scheme of calculating the data hash value and then comparing it is that the comparison of the hash value only requires one calculation and one comparison, while the comparison of the fields requires multiple comparison operations. When the amount of data is large, the overall efficiency of calculating the hash value is higher. Moreover, the hash value method integrates the comparison of all fields into a single operation, which simplifies the logical process. Comparison of each field needs to deal with the differences in each field type.

[0119] Step 310: Determine whether the version numbers of the two marked data records are the same. If yes, execute step 311; if no, execute step 312.

[0120] Step 311: If the version numbers are the same, based on the data source priority, that is, the MySQL data source data is primary and the Redis cache data source data is secondary, the data records in Redis are deleted, and the data records in MySQL are synchronized to Redis.

[0121] Step 312: If the version numbers are different, the data record with the larger version number is taken and synchronized to another data source.

[0122] Step 313: Record the verification differences of the configuration data table i to the log system and the alarm system for subsequent analysis and processing. The log system includes inconsistent field names, detailed error description information, time of occurrence, etc., for subsequent analysis and processing. Recording to the alarm system is to display the alarm information in the alarm bar on the one hand, and to send an alarm notification on the other hand to remind relevant personnel to handle it.

[0123] Step 314: Delete the data records of the configuration data table i in Redis, and synchronize the MySQL data records to Redis in batches.

[0124] Step 315: Determine whether the data records in all configuration tables have completed the comparison and verification process. If yes, go to step 316; if no, go to step 303.

[0125] Step 316: After the verification process of all configuration tables is completed, the database processing read-write lock is released and normal database operation is resumed.

[0126] Step 317: Generate a report on the difference results of the configuration data table check, providing detailed verification results and records of problems found.

[0127] At present, in high-concurrency scenarios, simple data coverage methods will lead to performance bottlenecks and cannot meet real-time requirements. In this embodiment, data hash value comparison is an efficient data consistency detection method. By calculating the hash value of the data, it is quickly determined whether the data is consistent. Compared with comparing fields one by one, the hash value comparison is more efficient and simplifies the logical process. Optimistic locks are used to ensure that the data is not modified during the detection process, and a data consistency verification script is designed to compare the data records in MySQL and Redis. Random sampling and full comparison strategies are used to compare the consistency of field information through data hash values, and data repair is performed according to the version number. In addition, during the comprehensive data consistency check, a data consistency verification script is pre-designed, and random sampling and full comparison strategies are used. The consistency of field information is compared by data hash values, and data repair is performed according to the version number to ensure that the data consistency check is more comprehensive, timely and efficient. In other words, by using optimistic locks and data hash value comparison methods, the efficiency of data synchronization and consistency detection is greatly improved, which is suitable for high-concurrency scenarios and avoids performance bottleneck problems. Finally, high-performance processing is achieved: improving concurrent performance, accelerating data consistency detection, and comprehensive data consistency checks: timely discovery and repair of data inconsistencies, ensuring data accuracy.

[0128] In addition, there is currently a lack of intelligent automatic repair mechanisms: manual intervention is required when data is inconsistent, increasing maintenance costs and complexity. In this embodiment, an intelligent data repair mechanism is provided that can automatically detect and repair data inconsistencies between MySQL and Redis, improving the reliability and automation of the system.

[0129] Above, in this application, a method for synchronizing and repairing data consistency between MySQL and Redis is proposed. A data change request is initiated through the client, a database transaction is started, and the version number of the data record is adjusted according to the change operation type. The changed data is synchronized to Redis, and the retry logic of synchronization failure is processed to ensure data consistency. When the client starts, the startup data table is loaded, the data table type is checked, and it is decided to load data from Redis or MySQL. The cache synchronization task is started to synchronize the configuration information in the memory to the Redis cache. During the data detection and repair process, an optimistic lock is used to ensure that the data is not modified during the consistency detection process, a data consistency verification script is designed, the data records in MySQL and Redis are compared, random sampling and full comparison strategies are used, the consistency of the field information is compared by the data hash value, and the data is repaired according to the version number to ensure data consistency. The present invention ensures efficient and stable operation of the system in high concurrency scenarios through intelligent data repair mechanism, two-way synchronization support and high-performance processing.

[0130] In an application scenario of a data processing method of the present application, refer to Figure 7 , including data change processing module, data synchronization module and data detection and repair module. Through database transactions, version number management, optimistic locking mechanism, full and sampling comparison strategy and hash value calculation and other technical means, the data consistency between MySQL and Redis is ensured. Through the client's automatic task thread, real-time synchronization and automatic repair of data are achieved, which effectively improves data access efficiency and ensures data consistency.

[0131] The client sends a data change request to the MySQL transaction management module and the Redis cache. It should be noted that when making data changes, data change requests will be sent to both at the same time. However, due to high concurrency and other reasons, the changes may not be synchronized when the two are making data changes, and the change results may be inconsistent. Therefore, consistency check and repair are required. In case of inconsistency, further data synchronization is performed; then, the MySQL transaction management module receives the change request, executes the corresponding transaction operation, and synchronizes the change to the MySQL database. The MySQL database receives the change operation and updates the corresponding data. The changed data is synchronized to the Redis cache to ensure that the data in the cache is consistent with the MySQL database. The data consistency check module reads data from the MySQL database and the Redis cache regularly or in real time for comparison and inspection. Check whether the data is consistent. If inconsistency is found, a report is generated and a repair operation is triggered. The automatic repair system module performs data repair operations based on the data consistency check results. According to the repair strategy, the data in the MySQL database and the Redis cache are corrected to ensure that the two are consistent. The log and notification system module records the exceptions and operation logs during the data consistency check and repair process. Notify relevant personnel to handle the exceptions through email or real-time notification system.

[0132] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0133] This application also provides a data processing device, please refer to Figure 6 , the data processing device comprises:

[0134] A determination module 10, configured to determine, when a client initiates a data change request, whether the change data in response to the data change request is configuration data or business data;

[0135] A first synchronization module 20 is used for synchronizing the changed data to a MySQL database after successfully synchronizing the changed data to a Redis cache if the changed data is configuration data;

[0136] The second synchronization module 30 is used to synchronize the business data to the MySQL database if the changed data is business data.

[0137] In one embodiment, the data processing device further includes a loading module, which is used to:

[0138] When starting the client and loading a series of startup data tables, it is determined that the current data table to be loaded stores configuration data or business data;

[0139] If the configuration data is saved, determine whether the data to be loaded exists in the Redis cache; if the data to be loaded exists in the Redis cache, load the data to be loaded from the Redis cache; if the data to be loaded does not exist in the Redis cache, load the data to be loaded from the MySQL database;

[0140] If the business data is saved, the data to be loaded is loaded from the MySQL database.

[0141] In one embodiment, the data processing device further includes a repair module, which is used to:

[0142] When performing data consistency detection, the configuration data table to be verified is read from the Redis cache and the MySQL database respectively;

[0143] Based on the data records in the configuration data table, data is repaired for the Redis cache and MySQL database.

[0144] In one embodiment, the repair module is further used for:

[0145] Compare whether the field information contained in the data records of the configuration data table is the same;

[0146] When the field information is not the same, determine whether the version numbers of the data records in the configuration data table are the same;

[0147] According to the version number comparison result, data repair of the data records of the configuration data table is performed between the Redis cache and the MySQL database.

[0148] In one embodiment, the repair module is further used for:

[0149] When the version numbers are the same, synchronize the data records in the Redis cache with the data records in the MySQL database;

[0150] When the version numbers are different, the data records with smaller version numbers between the Redis cache and the MySQL database are synchronized with the data records with larger version numbers.

[0151] In one embodiment, the repair module is further used for:

[0152] Compare whether the data modification version numbers in the data records of the configuration data table are the same;

[0153] When the data modification version numbers are the same, the data in the configuration data table is standardized, and the character hash values ​​after the standardization are compared to see whether they are the same.

[0154] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem of low data synchronization efficiency. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0155] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0156] Reference below Figure 7 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0157] like Figure 7As shown, the data processing device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0158] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0159] The data processing device provided by the present application adopts the data processing method in the above embodiment to solve the technical problem of low data synchronization efficiency. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0160] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0161] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0162] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0163] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0164] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0165] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data processing device, the data processing device: when a client initiates a data change request, determines whether the change data in response to the data change request is configuration data or business data; if the change data is configuration data, then after successfully synchronizing the change data to the Redis cache, synchronize the change data to a MySQL database; if the change data is business data, then synchronize the business data to the MySQL database.

[0166] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0167] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0169] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem of low data synchronization efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be described in detail here.

[0170] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0171] The computer program product provided by the present application can solve the technical problem of low data synchronization efficiency. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, which will not be repeated here.

[0172] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises: When the client initiates a data change request, determining that the change data in response to the data change request is configuration data or business data; If the changed data is configuration data, after successfully synchronizing the changed data to the Redis cache, the changed data is synchronized to the MySQL database; If the changed data is business data, the business data is synchronized to the MySQL database; The data processing method further includes: After the client initiates data change requests to the Redis cache and the MySQL database at the same time, and the data in the Redis cache and the MySQL database are changed, the data consistency check is performed on the MySQL database and the Redis cache; When performing data consistency detection, the configuration data table to be verified is read from the Redis cache and the MySQL database respectively; Compare whether the field information contained in the data records of the configuration data table is the same; When the field information is not the same, determine whether the version numbers of the data records in the configuration data table are the same; When the version numbers are the same, synchronize the data records in the Redis cache with the data records in the MySQL database; When the version numbers are different, the data records with smaller version numbers between the Redis cache and the MySQL database are synchronized with the data records with larger version numbers.

2. The data processing method according to claim 1, characterized in that: The data processing method further includes: When starting the client and loading a series of startup data tables, it is determined that the current data table to be loaded stores configuration data or business data; If the configuration data is saved, determine whether the data to be loaded exists in the Redis cache; if the data to be loaded exists in the Redis cache, load the data to be loaded from the Redis cache; if the data to be loaded does not exist in the Redis cache, load the data to be loaded from the MySQL database; If the business data is saved, the data to be loaded is loaded from the MySQL database.

3. The data processing method according to claim 1, characterized in that: The step of comparing whether the field information contained in the data records of the configuration data table is the same includes: Compare whether the data modification version numbers in the data records of the configuration data table are the same; When the data modification version numbers are the same, the data in the configuration data table is standardized, and the character hash values ​​after the standardization are compared to see whether they are the same.

4. A data processing device, characterized in that: The data processing device comprises: A determination module, used to determine whether the changed data in response to the data change request is configuration data or business data when the client initiates a data change request; A first synchronization module, configured to synchronize the changed data to a MySQL database after successfully synchronizing the changed data to a Redis cache if the changed data is configuration data; A second synchronization module is used to synchronize the business data to the MySQL database if the changed data is business data; The data processing device further comprises a repair module, which is used for: After the client initiates data change requests to the Redis cache and the MySQL database at the same time, and the data in the Redis cache and the MySQL database are changed, the data consistency check is performed on the MySQL database and the Redis cache; When performing data consistency detection, the configuration data table to be verified is read from the Redis cache and the MySQL database respectively; Compare whether the field information contained in the data records of the configuration data table is the same; When the field information is not the same, determine whether the version numbers of the data records in the configuration data table are the same; When the version numbers are the same, synchronize the data records in the Redis cache with the data records in the MySQL database; When the version numbers are different, the data records with smaller version numbers between the Redis cache and the MySQL database are synchronized with the data records with larger version numbers.

5. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 3.

6. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 3 are implemented.

7. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 3 are implemented.

Images