A data processing method, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides a data processing method, apparatus, electronic device, and storage medium. The method includes: invoking the master-slave synchronization interface of the Redis storage system to obtain an operation instruction for the key-value pair data; counting the data size of the data that has changed according to the operation instruction; storing the data identifier and the data size as target key-value pair data; determining, as abnormal data identifiers, the data identifiers corresponding to the data sizes that do not meet the preset conditions in the target key-value pair data; and determining, as abnormal key-value pair data, the key-value pair data corresponding to the abnormal data identifiers in the master database. Abnormal data can be found in the master database in real time, avoiding the expansion of problems and improving the stability of the master database operation.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data processing, and in particular, to a data processing method, a data processing device, an electronic device, and a storage medium. Background Art

[0002] Redis (Remote Dictionary Server), that is, remote dictionary service, is an open-source, network-supported, log-based, Key-Value database that can be memory-based or persistent. There is a mapping relationship between Key and Value. Redis has advantages such as rich data types and high performance.

[0003] In actual use, if there is a Value with a large data volume in Redis, when obtaining this Value with a large data volume through a Key, it will seriously affect the performance of Redis. Therefore, it is necessary to find out the Key corresponding to the Value with an overly large data volume and perform optimization processing. Currently, the commonly used method is to scan all data tables of Redis in an offline analysis manner to find out the Key corresponding to the Value with an overly large data volume. However, the search speed of this method is very slow, and abnormal data cannot be discovered in time, which may lead to a further decline in the performance of Redis. Summary of the Invention

[0004] Embodiments of the present invention provide a data processing method to solve the problem of slow search speed for abnormal data in the existing Redis storage system.

[0005] Correspondingly, embodiments of the present invention further provide a data processing device to ensure the implementation and application of the above method.

[0006] To solve the above problems, embodiments of the present invention disclose a data processing method, which is characterized in that it is applied to a Redis storage system. The Redis storage system includes a main database, and key-value pair data is stored in the main database. The key-value pair data includes a data identifier and data. The method includes:

[0007] Call the master-slave synchronization interface of the Redis storage system to obtain an operation instruction for the key-value pair data;

[0008] Statistically calculate the data size of the data that has changed according to the operation instruction;

[0009] Store the data identifier and the data size as target key-value pair data;

[0010] Determine the data identifier corresponding to the data size that does not meet the preset condition in the target key-value pair data as an abnormal data identifier;

[0011] Determine the key-value pair data corresponding to the abnormal data identifier in the main database as abnormal key-value pair data.

[0012] Optionally, the storing the data identifier and the data size as the target key-value pair data includes:

[0013] Determine whether the data identifier exists in the target key-value pair data;

[0014] If so, store the data identifier and the data size as target key-value pair data according to the operation instruction; if not, convert the data identifier and the data size into new key-value pair data and store the new key-value pair data in the target key-value pair data.

[0015] Optionally, the storing the data identifier and the data size as the target key-value pair data according to the operation instruction includes:

[0016] If the operation instruction is a modification instruction, obtain the original data size corresponding to the data identifier from the target key-value pair data and modify the original data size to the data size;

[0017] If the operation instruction is a deletion instruction, delete the key-value pair data corresponding to the data identifier in the target key-value pair data.

[0018] Optionally, the converting the data identifier and the data size into new key-value pair data includes:

[0019] Obtain the main database identifier of the main database;

[0020] Combine the main database identifier and the data identifier to obtain a specified data identifier;

[0021] Convert the data identifier into the specified data identifier;

[0022] Convert the specified data identifier and the data size into new key-value pair data.

[0023] Optionally, the determining the data identifier corresponding to the data size that does not meet the preset condition in the target key-value pair data as an abnormal data identifier includes:

[0024] Compare the data size in the target key-value pair data with a preset threshold;

[0025] When the data size exceeds the preset threshold, determine the data identifier corresponding to the data size as an abnormal data identifier.

[0026] Optionally, the method further includes:

[0027] Use the data identifier corresponding to the abnormal data identifier as the target data identifier;

[0028] Generate a prompt message for prompting that the key-value pair data corresponding to the target data identifier in the master database is abnormal according to the target data identifier.

[0029] An embodiment of the present invention also discloses a data processing device, which is characterized in that it is applied to a Redis storage system. The Redis storage system includes a master database, and key-value pair data is stored in the master database. The key-value pair data includes a data identifier and data. The device includes:

[0030] An operation instruction acquisition module, configured to call the master-slave synchronization interface of the Redis storage system to acquire an operation instruction for the key-value pair data;

[0031] A data size statistics module, configured to count the data size of the data changed according to the operation instruction;

[0032] A target key-value pair data conversion module, configured to store the data identifier and the data size as target key-value pair data;

[0033] An abnormal data identifier determination module, configured to determine the data identifier corresponding to the data size that does not meet the preset condition in the target key-value pair data as the abnormal data identifier;

[0034] An abnormal key-value pair data determination module, configured to determine the key-value pair data corresponding to the abnormal data identifier in the master database as the abnormal key-value pair data.

[0035] Optionally, the target key-value pair data conversion module includes:

[0036] A target key-value pair data conversion sub-module, configured to determine whether the data identifier exists in the target key-value pair data; if so, store the data identifier and the data size as target key-value pair data according to the operation instruction; if not, convert the data identifier and the data size into new key-value pair data, and store the new key-value pair data in the target key-value pair data.

[0037] Optionally, the target key-value pair data conversion sub-module is configured to, if the operation instruction is a modification instruction, obtain the original data size corresponding to the data identifier from the target key-value pair data, and modify the original data size to the data size; if the operation instruction is a deletion instruction, delete the key-value pair data corresponding to the data identifier in the target key-value pair data.

[0038] Optionally, the target key-value pair data conversion sub-module is configured to

[0039] obtain the main database identifier of the main database;

[0040] combine the main database identifier and the data identifier to obtain a specified data identifier;

[0041] convert the data identifier into the specified data identifier;

[0042] convert the specified data identifier and the data size into new key-value pair data.

[0043] Optionally, the abnormal data identifier determination module is configured to

[0044] compare the data size in the target key-value pair data with a preset threshold;

[0045] When the data size exceeds the preset threshold, determine the data identifier corresponding to the data size as an abnormal data identifier.

[0046] Optionally, the device further includes:

[0047] use the data identifier corresponding to the abnormal data identifier as the target data identifier;

[0048] generate a prompt message for prompting that there is an abnormality in the key-value pair data corresponding to the target data identifier in the main database according to the target data identifier.

[0049] An embodiment of the present invention discloses an electronic device, including a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the data processing method described above are implemented.

[0050] An embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0051] The embodiments of the present invention have the following advantages:

[0052] In an embodiment of the present invention, the master-slave synchronization interface of the Redis storage system is called to obtain operation instructions for the key-value pair data of the master database, and the data size of the data changed according to the operation instructions is counted. The data identifier and data size of the key-value pair data are stored as target key-value pair data. Furthermore, the data identifier corresponding to the data size that does not meet the preset conditions in the target key-value pair data is determined as an abnormal data identifier. Finally, the key-value pair data corresponding to the abnormal data identifier in the master database is determined as abnormal key-value pair data. In the embodiment of the present invention, a statistical program is used to obtain data operation instructions for the master database, and the data size after the change is counted by analyzing the operation instructions. The data identifier and data size of the source data are converted into target key-value pair data. Furthermore, the abnormal data identifier can be found according to the data size in the target key-value pair data, and the abnormal data corresponding to the abnormal data identifier can be found in the master database in real time, avoiding the expansion of problems and improving the stability of the operation of the master database. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of the steps of an embodiment of a data processing method of the present invention;

[0054] Figure 2 It is a schematic diagram of the process of an embodiment of a data processing method of the present invention;

[0055] Figure 3 It is a structural block diagram of an embodiment of a data processing device of the present invention;

[0056] Figure 4 It is a structural block diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] As a high-performance Key-Value database, Redis is widely used in many network systems. However, in actual scenarios, there may be some Keys in Redis that correspond to Values with a very large data capacity. Such Keys are generally called large Keys. When the large Keys in Redis are accessed, it will seriously affect the performance of Redis. Therefore, it is necessary to find out the large Keys in Redis and perform optimization processing. A common way to find large Keys is to scan all the data tables of Redis in an offline analysis manner. However, the search speed of this method is very slow, and the search time may be several hours or more, and abnormal data cannot be found in time.

[0059] In view of the above problems, an embodiment of the present invention proposes a data processing method, which is applied to a Redis storage system. The Redis storage system includes a main database. By disguising a statistical program as a slave database of the main database, the statistical program can synchronize key-value pair data from the main database in real time, and find out abnormal key-value pair data according to the data size in the key-value pair data, so as to quickly find out abnormal data from the main database in real time, avoid the expansion of problems, and improve the stability of the operation of the main database.

[0060] Refer to Figure 1 , which is a flowchart of the steps of a data processing method provided in an embodiment of the present invention, and is applied to a Redis storage system. The Redis storage system includes a main database, and the main database stores key-value pair data. The key-value pair data includes a data identifier and data, and specifically may include the following steps:

[0061] Step 101, call the master-slave synchronization interface of the Redis storage system to obtain an operation instruction for the key-value pair data;

[0062] The Redis storage system supports master-slave replication. Master-slave replication enables data to be replicated from one database server to other servers. When replicating data, one server acts as the master server, and the remaining servers act as slave servers. Specifically, data synchronization can be performed by calling the master-slave synchronization interface of the Redis storage system.

[0063] Among them, the Redis storage system may include one or more main databases. The Redis storage system may be implemented by an independent server or by a server cluster composed of multiple servers. The embodiment of the present invention does not limit this. The database of the Redis storage system is a Key-Value database, that is, the data in the main database is key-value pair data. The key is the data identifier used to mark the data in a group of key-value pair data, and the value is the data in a group of key-value pair data. There is a mapping relationship between the key and the value, that is, one key corresponds to one value.

[0064] In the embodiments of the present invention, key-value data (Key-Value) includes a data identifier and data, where the key (Key) is the data identifier and the value (Value) is the data. Specifically, the data identifier can be a custom string, which can include numbers, letters, or words, such as "Key1", "D1", "data a", etc. The data is the actual data content stored in the database, and the types of data can include integers, floating-point numbers, strings, etc. The key and the value correspond to each other, that is, the data identifier and the data are in one-to-one correspondence. For example, the data corresponding to "Key1" is 123456, the data corresponding to "D1" is "123.4", and the data corresponding to "data a" is "Monday". A simple representation can be <Key1, 123456>. This representation will also be used when introducing key-value data below.

[0065] In a specific implementation, the key-value data in the main database can be operated on through operation instructions. Among them, the operation instructions can include add instructions, modify instructions, and delete instructions. The add instruction can add key-value data in the main database, the modify instruction can modify the key-value data in the main database, and the delete instruction can delete the key-value data in the main database. For example, a modify instruction is "Set Key1 1111", that is, the Value corresponding to the key-value data with the key "Key1" is set to "1111".

[0066] Specifically, in actual applications, the key-value data in the main database is modified through operation instructions. Therefore, by obtaining the operation instructions for the key-value data in the main database, the key-value data in the main database can be statistically analyzed. Specifically, a statistical program can be used to call the master-slave synchronization interface of the Redis storage system, and based on this master-slave synchronization interface, obtain the operation instructions for the key-value data of the main data.

[0067] Step 102: Statistically analyze the data size of the data after being changed according to the operation instruction;

[0068] Among them, the data size refers to the capacity size of the storage medium occupied by the data in the key-value data when stored in the main data. For example, the data size of the data in a group of key-value data <Key1, Value> is 4, while the data size of the data in another group of key-value data <Key2, Value> is 10, etc.

[0069] In a specific implementation, the statistical program can analyze the obtained operation instructions and count the data size of the data that has been changed after the operation instructions. In an example, assume that a set of key-value pair data in the main database is <Key1, Value1>. At this time, the operation instruction obtained by the statistical program for this set of key-value pair data is "Set Key1 Value2". By analysis, it can be known that the operation instruction is a modification instruction, and the statistical program can count the data size of the modified Value2.

[0070] Step 103: Store the data identifier and the data size as target key-value pair data;

[0071] In a specific implementation, the statistical program counts the data size of the key-value pair data in the main data, but does not store the original key-value pair data. Instead, it converts the original key-value pair data into target key-value pair data according to a preset rule for storage. Specifically, it can obtain the data identifier in the original key-value pair data and count the data size of the data in the key-value pair data, combine the data identifier and the data size into a new key-value pair data, and store it as the target key-value pair data. For example, for the above key-value pair data <Key1, Value1>, the data size of Value1 is 10. After being modified by the operation instruction "Set Key1 Value2", the data size of Value2 is 20. The statistical program can store the data identifier "Key1" and the data size 20 after being changed by the operation instruction as the target key-value pair data <Key1, 20>.

[0072] In an exemplary embodiment, step 103, storing the data identifier and the data size as the target key-value pair data, includes:

[0073] Determine whether the data identifier exists in the target key-value pair data;

[0074] If so, store the data identifier and the data size as target key-value pair data according to the operation instruction; if not, convert the data identifier and the data size into new key-value pair data and store the new key-value pair data in the target key-value pair data.

[0075] In a specific implementation, the statistical program can count the data size of the key-value pair data after the operation instruction is changed each time an operation instruction for the main database is obtained, and store it as target key-value pair data in the form of <data identifier, data size>. Therefore, multiple groups of target key-value pair data can be stored in the cache of the statistical program.

[0076] Therefore, before the statistical program stores the key-value pair data after being changed by the operation instruction as the target key-value pair data each time, it is necessary to determine whether the data identifier of the key-value pair data exists in the target key-value pair data in the cache.

[0077] If the data identifier already exists in the target key-value pair data, it means that the target key-value pair data already stores the data size of the key-value pair data before the change of this operation instruction. Therefore, according to this operation instruction, the data identifier and the changed data size can be stored as the target key-value pair data. Specifically, it can be to update the target key-value pair data according to this operation instruction.

[0078] In an exemplary embodiment, the storing the data identifier and the data size as the target key-value pair data according to the operation instruction includes:

[0079] If the operation instruction is a modification instruction, obtain the original data size corresponding to the data identifier from the target key-value pair data, and modify the original data size to the data size;

[0080] If the operation instruction is a deletion instruction, delete the key-value pair data corresponding to the data identifier in the target key-value pair data.

[0081] In an example, if the target key-value pair data stores two groups of data <Key1, 10> and <Key2, 20>. Specifically, if the operation instruction is a modification instruction, the data size corresponding to the data identifier in the target key-value pair data can be updated. For example, if the operation instruction is "Set Key1 Value1", and the statistical program calculates that the data size of Value1 is 100, then <Key1, 10> will be updated to <Key1, 100>. If the operation instruction is a deletion instruction, the data size corresponding to the data identifier in the target key-value pair data can be deleted. For example, if the operation instruction is "Delete Key2", and this operation instruction is specifically to delete the key-value pair data corresponding to "Key2", then <Key2, 20> in the target key-value pair data can be deleted. After the above changes, at this time, the target key-value pair data stores only one group of data <Key1, 100>.

[0082] If the data identifier does not exist in the target key-value pair data, it means that the target key-value pair data has not yet stored the data related to this key-value pair data. Therefore, the data identifier and the data size corresponding to this operation instruction can be stored as new key-value pair data in the target key-value pair data.

[0083] In an exemplary embodiment, the converting the data identifier and the data size into new key-value pair data includes:

[0084] Obtain the main database identifier of the main database;

[0085] Combine the main database identifier and the data identifier to obtain a specified data identifier;

[0086] Convert the data identifier to the specified data identifier;

[0087] Convert the specified data identifier and the data size into new key-value pair data.

[0088] Among them, the main database has a main database identifier, and the main database identifier can be the name or number of the main database, etc., which can distinguish the main database. For example, the main database identifier of main database 1 is DB1.

[0089] In practical applications, there may be multiple main databases in the Redis storage system. Each main database stores key-value pair data. In specific applications, the statistical program can obtain operation instructions for multiple different main databases at the same time. The data identifiers of the key-value pair data in different main databases can be the same. Therefore, when the statistical program statistically analyzes multiple main databases at the same time, the data identifier can be further transformed before storing the data identifier and the data size as the target key-value pair data, so as to avoid statistical errors caused by the same data identifiers of the key-value pair data in different main databases.

[0090] Specifically, it can be to combine the main database identifier and the data identifier to obtain a specified data identifier, then convert the data identifier to the specified data identifier, and then convert the specified data identifier and the data size into new key-value pair data.

[0091] In one example, assume that there are master databases DB1 and DB2 in a Redis storage system. The key-value pair data <Key1, Value1> is stored in DB1, and the key-value pair data <Key1, Value2> is stored in DB2. Before storing the target key-value pair data, the statistical program can first obtain the master data identifiers of these two master data, which are "DB1" and "DB2" in this example. Then, combine the master database identifier and the data identifier to obtain a specified data identifier, that is, combine "DB1" and "Key1" to get "DB1.Key1", and combine "DB2" and "Key1" to get "DB2.Key1". Then, convert the original data identifier to the specified data identifier, that is, convert "Key1" in the master database DB1 to "DB1.Key1", and convert "Key1" in the master database DB2 to "DB2.Key1". At this point, the data identifier has been converted to the specified data identifier. Further, the specified data identifier and the data size can be converted into new key-value pair data. Assume that the data size of Value1 is 10 and the data size of Value2 is 20. Then, <Key1, Value1> in DB1 is converted to <DB1.Key1, 10> after conversion, and <Key1, Value2> in DB2 is converted to <DB2.Key1, 20> after conversion. Thus, the conversion of new key-value pair data is completed, and the key-value pair data in multiple master databases is distinguished.

[0092] It should be noted that when the statistical program determines whether a certain data identifier exists in the target key-value pair data in the cache, it makes the determination according to the data identifier form when the target key-value pair data is stored. It can determine the data identifier or the specified data identifier. That is, if the data identifier is not converted to the specified data identifier when the statistical program stores the target key-value pair data, then when making the determination, it determines the data identifier; if the data identifier is converted to the specified data identifier when the statistical program stores the target key-value pair data, then when making the determination, it determines the specified data identifier.

[0093] Step 104: Determine the data identifier corresponding to the data size that does not meet the preset condition in the target key-value pair data as the abnormal data identifier;

[0094] Among them, the preset condition can be a preset data size threshold. In a specific implementation, the statistical program can count the data sizes saved in the target key-value pair data. When a certain data size exceeds the preset data size threshold, the data identifier corresponding to the data size is marked as the abnormal data identifier.

[0095] In an exemplary embodiment, step 105, determining the data identifier corresponding to the data size that does not meet the preset condition in the target key-value pair data as the abnormal data identifier, includes:

[0096] Compare the data size in the target key-value pair data with a preset threshold;

[0097] When the data size exceeds the preset threshold, determine the data identifier corresponding to the data size as an abnormal data identifier.

[0098] Specifically, the statistical program can compare all the data sizes in the target key-value pair data with the preset threshold in sequence. When a certain data size exceeds the preset threshold, determine the data identifier corresponding to the data size as an abnormal data identifier. For example, the preset threshold is 100, and a set of target key-value pair data A is <DB1.Key1, 200>. After comparison, it can be seen that the data size of A exceeds the preset threshold. Therefore, the data identifier "DB1.Key1" corresponding to the data size of A can be determined as an abnormal data identifier.

[0099] Step 105: Determine the key-value pair data corresponding to the abnormal data identifier in the main database as abnormal key-value pair data.

[0100] Specifically, after determining the abnormal data identifier, the key-value pair data corresponding to it can be found in the main database according to the abnormal data identifier, and the key-value pair data is determined as abnormal key-value pair data. Specifically, according to the above conversion rule of the specified data identifier, the abnormal data identifier can be reversely converted to obtain the original data identifier in the main database corresponding to the abnormal data identifier, and then the key-value pair data corresponding to the original data identifier is determined as abnormal key-value pair data.

[0101] In the above example, "DB1.Key1" is the abnormal data identifier. When converting the specified data identifier, the main database identifier is "DB1". Therefore, the abnormal data identifier can be reversely processed, that is, removing "DB1", and the original data identifier "Key1" in the main database can be obtained. Then, the corresponding key-value pair data can be found according to "Key1" and determined as abnormal key-value pair data.

[0102] In an exemplary embodiment, the method further includes:

[0103] Use the data identifier corresponding to the abnormal data identifier as the target data identifier;

[0104] Generate a prompt message for prompting that there is an abnormality in the key-value pair data corresponding to the target data identifier in the main database according to the target data identifier.

[0105] After the statistical program finds the abnormal data identifier, according to the combination rule during the conversion of the specified data identifier, the data identifier corresponding to the abnormal data identifier can be obtained. This data identifier is the original data identifier that has not been converted in the master data. Then, taking this data identifier as the target data identifier, according to the target data identifier, the corresponding key-value pair data can be found in the master database. Therefore, a prompt message can be generated to indicate that there is an abnormality in this key-value pair data in the master data. The prompt message can specifically be to prompt that the target data identifier is abnormal, or directly prompt that there is an abnormality in the key-value pair data in the master data corresponding to the target data identifier. The embodiments of the present invention do not limit this.

[0106] To enable those skilled in the art to better understand the technical solutions of the embodiments of the present invention, the following is an example and a combination Figure 2 to illustrate the embodiments of the present invention.

[0107] Refer to Figure 2 , which shows a schematic flowchart of an embodiment of a data processing method of the present invention. Among them, RedisMaster is the master database, and the statistical program includes a synchronization protocol module, a statistical analysis module, and a notification module. Specifically, the synchronization protocol module can call the master-slave synchronization interface of the Redis storage system to obtain operation instructions for the key-value pair data in the master database; the statistical analysis module can analyze the operation instructions, count the data size of the data after being changed by the operation instructions, and determine the abnormal data identifier based on the data size; after the notification module finds the abnormal data, it generates a prompt message according to the information of the abnormal data to prompt that there is abnormal data in the master database.

[0108] In an example, assume that there are three groups of key-value pair data in the master database, namely <Key1, Value1>, <Key2, Value2>, and <Key3, Value3>.

[0109] 1. First, call the master-slave synchronization interface of the Redis storage system to obtain operation instructions for the key-value pair data in the master database;

[0110] 2. The statistical program calculates the data size of the key-value pair data after being changed according to the operation instruction. Specifically, the key-value pair data changed by the operation instruction can be stored as target key-value pair data in the form of <data identifier, data size (Size)>. <Key1, Value1>, <Key2, Value2>, and <Key3, Value3> can be respectively converted into <Key1, Size1>, <Key2, Size2>, and <Key3, Size3> and stored in the memory of the statistical program. Specifically, before a certain key-value pair data is stored as target key-value pair data, it can be determined whether the data identifier of this key-value pair data exists in the target key-value pair data. If it exists, the data corresponding to this data identifier in the target key-value pair data is updated according to the operation instruction. If it does not exist, a new set of data is added to the target key-value pair data. When the statistical program simultaneously calculates multiple main databases, the data identifier and the main data identifier can also be combined to obtain a specified data identifier, and then stored as target key-value pair data in the form of <specified data identifier, data size>.

[0111] 3. The statistical program can preset a data size threshold, and then judge one by one whether the size in the target key-value pair data exceeds the preset threshold. When the size of a certain group of target key-value pair data exceeds the preset threshold, it is determined that the data identifier of this group of target key-value pair data is an abnormal data identifier;

[0112] 4. After determining the abnormal data identifier, it can be determined that the key-value pair data corresponding to the abnormal data identifier in the main database is abnormal data. Therefore, corresponding prompt information can be generated to prompt the existence of abnormal data and can locate this abnormal data.

[0113] In practical applications, if there are many services with large Keys, including user growth services, message push services, literature services, recommendation services, etc., accessing large Keys often causes Redis service blockage and affects the service effect. Taking the message push service as an example, the service uses the device channel as the Key and a certain type of device ID as the Value, and continuously appends the same type of device ID into the Value of the same Key, resulting in the size of the Value corresponding to this Key reaching 270MB (data unit, megabyte). During a certain period, the service application frequently reads the full amount of data of this Key, and each read takes more than 1 second, and it is read multiple times per minute, resulting in other normal Redis read requests being blocked and affecting the normal operation of the service. At this time, the statistical program is used to calculate large Keys in real time and promptly inform the service party of the existence of large Keys, and the service party can quickly optimize the large Keys to avoid continuous impact on the production environment.

[0114] In an embodiment of the present invention, the master-slave synchronization interface of the Redis storage system is called to obtain operation instructions for the key-value pair data of the master database, and the data size of the data changed according to the operation instructions is counted. The data identifier and the data size of the key-value pair data are stored as target key-value pair data. Furthermore, the data identifier corresponding to the data size that does not meet the preset condition in the target key-value pair data is determined as an abnormal data identifier. Finally, the key-value pair data corresponding to the abnormal data identifier in the master database is determined as abnormal key-value pair data. In the embodiment of the present invention, a statistical program is used to obtain the data operation instructions for the master database, and the data size after the change is counted by analyzing the operation instructions. The data identifier and the data size of the source data are converted into target key-value pair data. Furthermore, the abnormal data identifier can be found according to the data size in the target key-value pair data, and the abnormal data corresponding to the abnormal data identifier can be found in the master database in real time, avoiding the expansion of problems and improving the stability of the operation of the master database.

[0115] It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0116] Refer to Figure 3 , which shows a structural block diagram of an embodiment of a data processing device of the present invention, applied to a Redis storage system. The Redis storage system includes a master database, and the master database stores key-value pair data. The key-value pair data includes a data identifier and data. The embodiment of the present invention specifically may include the following modules:

[0117] An operation instruction acquisition module, configured to call the master-slave synchronization interface of the Redis storage system to obtain operation instructions for the key-value pair data;

[0118] A data size statistics module, configured to count the data size of the data changed according to the operation instructions;

[0119] A target key-value pair data conversion module, configured to store the data identifier and the data size as target key-value pair data;

[0120] An abnormal data identifier determination module, configured to determine the data identifier corresponding to the data size that does not meet the preset condition in the target key-value pair data as an abnormal data identifier;

[0121] An abnormal key-value pair data determination module, configured to determine the key-value pair data corresponding to the abnormal data identifier in the main database as abnormal key-value pair data.

[0122] In an exemplary embodiment, the target key-value pair data conversion module includes:

[0123] A target key-value pair data conversion sub-module, configured to determine whether the data identifier exists in the target key-value pair data; if so, store the data identifier and the data size as the target key-value pair data according to the operation instruction; if not, convert the data identifier and the data size into new key-value pair data, and store the new key-value pair data in the target key-value pair data.

[0124] In an exemplary embodiment, the target key-value pair data conversion sub-module is configured to, if the operation instruction is a modification instruction, obtain the original data size corresponding to the data identifier from the target key-value pair data, and modify the original data size to the data size; if the operation instruction is a deletion instruction, delete the key-value pair data corresponding to the data identifier in the target key-value pair data.

[0125] In an exemplary embodiment, the target key-value pair data conversion sub-module is configured to

[0126] Obtain the main database identifier of the main database;

[0127] Combine the main database identifier and the data identifier to obtain a specified data identifier;

[0128] Convert the data identifier into the specified data identifier;

[0129] Convert the specified data identifier and the data size into new key-value pair data.

[0130] In an exemplary embodiment, the abnormal data identifier determination module is configured to

[0131] Compare the data size in the target key-value pair data with a preset threshold;

[0132] When the data size exceeds the preset threshold, determine the data identifier corresponding to the data size as an abnormal data identifier.

[0133] In an exemplary embodiment, the apparatus further includes:

[0134] Use the data identifier corresponding to the abnormal data identifier as the target data identifier;

[0135] Generate a prompt message for indicating that there is an abnormality in the key-value pair data corresponding to the target data identifier in the master database according to the target data identifier.

[0136] For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, please refer to the partial description of the method embodiments.

[0137] An embodiment of the present invention further provides an electronic device, such as Figure 4 shown, including a processor 401, a communication interface 402, a memory 403, and a communication bus 404. Among them, the processor 401, the communication interface 402, and the memory 403 complete mutual communication through the communication bus 404.

[0138] The memory 403 is used to store a computer program.

[0139] When the processor 401 is used to execute the program stored on the memory 403, the following steps are implemented:

[0140] Call the master-slave synchronization interface of the Redis storage system to obtain an operation instruction for the key-value pair data.

[0141] Statistically calculate the data size of the data after being changed according to the operation instruction.

[0142] Store the data identifier and the data size as target key-value pair data.

[0143] Determine the data identifier corresponding to the data size that does not meet the preset condition in the target key-value pair data as an abnormal data identifier.

[0144] Determine the key-value pair data corresponding to the abnormal data identifier in the master database as abnormal key-value pair data.

[0145] Optionally, the storing the data identifier and the data size as the target key-value pair data includes:

[0146] Judge whether the data identifier exists in the target key-value pair data.

[0147] If so, store the data identifier and the data size as target key-value pair data according to the operation instruction; if not, convert the data identifier and the data size into new key-value pair data, and store the new key-value pair data in the target key-value pair data.

[0148] Optionally, the storing the data identifier and the data size as the target key-value pair data according to the operation instruction includes:

[0149] If the operation instruction is a modification instruction, obtain the original data size corresponding to the data identifier from the target key-value pair data, and modify the original data size to the data size;

[0150] If the operation instruction is a deletion instruction, delete the key-value pair data corresponding to the data identifier in the target key-value pair data.

[0151] Optionally, the converting the data identifier and the data size into new key-value pair data includes:

[0152] Obtain the main database identifier of the main database;

[0153] Combine the main database identifier and the data identifier to obtain a specified data identifier;

[0154] Convert the data identifier into the specified data identifier;

[0155] Convert the specified data identifier and the data size into new key-value pair data.

[0156] Optionally, the determining, as abnormal data identifiers, the data identifiers corresponding to the data sizes in the target key-value pair data that do not meet the preset conditions includes:

[0157] Compare the data sizes in the target key-value pair data with a preset threshold;

[0158] When the data size exceeds the preset threshold, determine the data identifier corresponding to the data size as an abnormal data identifier.

[0159] Optionally, the method further includes:

[0160] Use the data identifier corresponding to the abnormal data identifier as the target data identifier;

[0161] Generate a prompt message for prompting that there is an abnormality in the key-value pair data corresponding to the target data identifier in the main database according to the target data identifier.

[0162] The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0163] The communication interface is used for communication between the above-mentioned terminal and other devices.

[0164] The memory may include a random access memory (RAM) and may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0165] The aforementioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0166] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium storing instructions, which, when running on a computer, cause the computer to execute the data processing method described in any one of the above embodiments.

[0167] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which, when running on a computer, cause the computer to execute the data processing method described in any one of the above embodiments.

[0168] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0169] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including an..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0170] Each embodiment in this specification is described in a related manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0171] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, Applied to a Redis storage system, the Redis storage system includes a master database, and key-value pair data is stored in the master database. The key-value pair data includes a data identifier and data. The method includes: Disguise a statistical program as a slave database of the master database, and use the statistical program to call the master-slave synchronization interface of the Redis storage system to obtain operation instructions for the key-value pair data. Use the statistical program to count the data size of the data that has been changed according to the operation instructions. Use the statistical program to store the data identifier and the data size as target key-value pair data. Use the statistical program to determine the data identifier corresponding to the data size that does not meet the preset conditions in the target key-value pair data as an abnormal data identifier. Determine the key-value pair data corresponding to the abnormal data identifier in the master database as abnormal key-value pair data.

2. The method according to claim 1, wherein The step of using the statistical program to store the data identifier and the data size as the target key-value pair data includes: Judge whether the data identifier exists in the target key-value pair data. If so, store the data identifier and the data size as target key-value pair data according to the operation instructions; if not, convert the data identifier and the data size into new key-value pair data, and store the new key-value pair data in the target key-value pair data.

3. The method according to claim 2, wherein The step of storing the data identifier and the data size as target key-value pair data according to the operation instructions includes: If the operation instruction is a modification instruction, obtain the original data size corresponding to the data identifier from the target key-value pair data, and modify the original data size to the data size. If the operation instruction is a deletion instruction, delete the key-value pair data corresponding to the data identifier in the target key-value pair data.

4. The method according to claim 2, characterized in that, The step of converting the data identifier and the data size into new key-value pair data includes: Obtain the master database identifier of the master database. Combine the master database identifier and the data identifier to obtain a specified data identifier. Convert the data identifier into the specified data identifier. Convert the specified data identifier and the data size into new key-value pair data.

5. The method according to claim 1, wherein The step of using the statistical program to determine the data identifier corresponding to the data size that does not meet the preset conditions in the target key-value pair data as an abnormal data identifier includes: Compare the data size in the target key-value pair data with a preset threshold. When the data size exceeds the preset threshold, determine the data identifier corresponding to the data size as an abnormal data identifier.

6. The method according to claim 1, characterized in that, The method further includes: Use the data identifier corresponding to the abnormal data identifier as a target data identifier. Generate a prompt message for prompting that there is an abnormality in the key-value pair data corresponding to the target data identifier in the master database according to the target data identifier.

7. A data processing device, characterized in that, Applied to a Redis storage system, the Redis storage system includes a master database, and key-value pair data is stored in the master database. The key-value pair data includes a data identifier and data. The device includes: An operation instruction acquisition module, configured to disguise a statistical program as a slave database of the master database, and use the statistical program to call the master-slave synchronization interface of the Redis storage system to obtain operation instructions for the key-value pair data; A data size statistics module, configured to use the statistical program to count the data size of the data that has been changed according to the operation instructions; A target key-value pair data conversion module, configured to use the statistical program to store the data identifier and the data size as target key-value pair data; An abnormal data identifier determination module, configured to use the statistical program to determine the data identifier corresponding to the data size that does not meet the preset conditions in the target key-value pair data as an abnormal data identifier; An abnormal key-value pair data determination module, configured to determine the key-value pair data corresponding to the abnormal data identifier in the master database as abnormal key-value pair data.

8. The device according to claim 7, characterized in that, The target key-value pair data conversion module includes: A target key-value pair data conversion sub-module, configured to determine whether the data identifier exists in the target key-value pair data; if so, store the data identifier and the data size as target key-value pair data according to the operation instructions; if not, convert the data identifier and the data size into new key-value pair data, and store the new key-value pair data in the target key-value pair data.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images