Data processing method and device
By using regular expression matching conditions on the Redis server to optimize the key operations of the Redis database, the time-consuming and memory-consuming problems in the existing technology are solved, and more efficient key renaming and deletion operations are achieved.
Patent Information
- Application Number
- CN202110542176.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-18
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2041-05-18
AI Technical Summary
Existing Redis clients are time-consuming and memory-intensive to rename or delete keys in a Redis database, and are unable to efficiently process a class of keys with common naming characteristics.
The Redis server receives the client's operation commands, uses regular expression matching conditions to traverse and execute operations, reduces network latency and memory usage, and optimizes operation instructions by carrying regular expressions.
It reduces the operation time between the Redis client and the server, reduces the client load, and improves operation efficiency.
Smart Images
Figure CN113505020B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communication technology, and in particular to a data processing method and device. Background Art
[0002] The Remote Dictionary Server (Redis) database is a high-performance key-value database that can be used to store real-time data on network devices. If a network device unexpectedly restarts, it can retrieve previously stored data from the Redis database and use this data to recover services. Network devices can perform operations on the Redis database, such as adding, deleting, modifying, and querying data.
[0003] Currently, the following methods exist for modifying and deleting keys: RENAME operation, corresponding command is RENAME oldkey newkey; RENAMENX oldkey newkey; DEL operation, corresponding command is DEL key, which deletes the specified key.
[0004] According to the support of the existing Redis database for operation commands, each time a key is renamed or deleted, only the specified key can be renamed or deleted, and a class of keys with common naming characteristics cannot be renamed or deleted.
[0005] The following takes the renaming key operation in the Redis database as an example to illustrate.
[0006] Assume that the key consists of the following fields: userinfo, modulename, datatype, dataattrtype, attrvalue, and privatekey. Fields are separated by ":". The userinfo field is a fixed string; modulename is the name of the module that accesses the data (in string format. Because multiple business modules may need to write data to the Redis database, the business module name must be stored to distinguish the data stored in each module); the datatype field is the data type of the accessed data (in numeric format, numbered by the business module that needs to store the data in the Redis database); the dataattrtype field is the current data classification (in string format). Classification types include up type or global type; the attrvalue field (in numeric format) stores the value corresponding to the dataattrtype type. For example, if the dataattrtype field indicates that the current key is classified as up, the attrvalue field stores the up ID; the privatekey field stores the private information of each key.
[0007] In one scenario, if the up ID of an online service changes from 1024 to 1025, each business module must update the attrvalue of the dataattrtype field from 1024 to 1025, regardless of the value of the datatype field or the privatekey field. For example, if the key is userinfo:ucm:1:up:1024:sessionid100, the key must be updated to userinfo:ucm:1:up:1025:sessionid100.
[0008] In the existing implementation, if a certain type of key is to be renamed, the Redis client must first obtain the specified type of keys from the Redis server in batches using the SCAN command, and then rename each key after obtaining it.
[0009] like Figure 1 As shown, Figure 1 Schematic diagram of performing key renaming operations for existing Redis clients. Figure 1In the example, the Redis client sends a SCAN command containing a regular expression to the Redis server. The Redis server retrieves a specified number (for example, 5000) of keys from the Redis database that match the regular expression. The Redis server sends a Reply message containing 5000 keys to the Redis client. The Redis client iterates over each key, modifies the up ID included in each key, and generates a new key.
[0010] The Redis client sends 5000 RENAME commands to the Redis server, each containing a new key. After the Redis server executes the RENAME command, it sends a reply message back to the Redis client containing the result of the RENAME command.
[0011] After receiving the Reply message, the Redis client again obtains the next batch of specified number of keys from the Redis server and repeats the above operation until no key that meets the requirements of the regular expression can be obtained.
[0012] The existing implementation of key renaming in the Redis client also brings the following problems: key renaming is very time-consuming, and a large amount of time is spent in network latency; moreover, the Redis client cannot request too many keys at a time; otherwise, a large amount of local memory will be occupied when renaming the keys. Summary of the Invention
[0013] In view of this, the present application provides a data processing method and device to solve the problem in existing solutions that when a Redis client operates data in a Redis database, time is wasted and a large amount of memory of the Redis client is occupied.
[0014] In a first aspect, the present application provides a data processing method, which is applied to a Redis server and includes:
[0015] Receive a first operation command sent by a Redis client, where the first operation command includes an operation attribute and a matching condition;
[0016] According to the matching condition, traverse each key stored in the Redis database and obtain at least one original key that meets the matching condition;
[0017] According to the operation attribute, perform a corresponding operation on the at least one original key and record the operation result;
[0018] Send a response message to the Redis client, where the response message includes the operation result.
[0019] In a second aspect, the present application provides a data processing device, which is applied to a Redis server, and includes:
[0020] A receiving unit, configured to receive a first operation command sent by a Redis client, where the first operation command includes an operation attribute and a matching condition;
[0021] an acquisition unit, configured to traverse each key stored in the Redis database according to the matching condition and acquire at least one original key that meets the matching condition;
[0022] an execution unit, configured to execute a corresponding operation on the at least one original key according to the operation attribute and record the operation result;
[0023] A sending unit is used to send a response message to the Redis client, where the response message includes the operation result.
[0024] In a third aspect, the present application provides a network device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to execute the method provided in the first aspect of the present application.
[0025] Therefore, by applying the data processing method and device provided in this application, the Redis server receives a first operation command sent by the Redis client, the first operation command including operation attributes and matching conditions. Based on the matching conditions, the Redis server traverses each stored key in the Redis database and obtains at least one original key that meets the matching conditions. Based on the operation attributes, the Redis server performs a corresponding operation on the at least one original key and records the operation result. The Redis server sends a response message to the Redis client, the response message including the operation result.
[0026] In this way, the problem of the Redis client wasting time and occupying a large amount of memory when operating data in the Redis database in the existing solution is solved. At the same time, in this application, by carrying regular expressions in the operation instructions, the operation time between the Redis client and the Redis server is reduced, and the load on the Redis client is reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 Schematic diagram of performing key rename operation for existing Redis clients;
[0028] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;
[0029] Figure 3 A structural diagram of a data processing device provided in an embodiment of the present application;
[0030] Figure 4 The network device hardware structure provided in the embodiment of the present application. DETAILED DESCRIPTION
[0031] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
[0032] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the corresponding listed items.
[0033] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0034] The data processing method provided in the embodiment of the present application is described in detail below. Figure 2 , Figure 2 Flowchart of the data processing method provided in the embodiment of the present application. The method is applied to the Redis server, and the data processing method provided in the embodiment of the present application may include the following steps.
[0035] Step 210: Receive a first operation command sent by a Redis client, where the first operation command includes an operation attribute and a matching condition.
[0036] Specifically, when the Redis client needs to perform operations such as deletion and modification on data stored in the Redis database, the Redis client generates a first operation command, which includes an operation attribute and a matching condition.
[0037] The Redis client sends a first operation command to the Redis server.
[0038] Furthermore, the matching condition may be specifically a regular expression.
[0039] In one example, the operation attribute may specifically be a rename operation. When the first operation command is a rename operation, the first operation command further includes a replacement condition.
[0040] For example, the first operation command is specifically: RENAME MATCH find_regex replace_regex; wherein the matching condition is "find_regex", that is, the regular expression to be matched; and the replacement result is "replace_regex", that is, the string after replacement.
[0041] In another example, the operation attribute may be specifically a delete operation. For example, the first operation command may be specifically: DELMATCH find_regex; wherein the matching condition is "find_regex", that is, the regular expression to be matched.
[0042] It should be noted that the Redis client can also perform operations such as adding and searching on the Redis database. For example, when the Redis client stores new data in the Redis database, it is an adding operation; when the Redis client searches for data in the Redis database, it is a searching operation. In the embodiment of the present application, no changes are involved in operations such as adding and searching. The Redis client can still store new data in the Redis database in the same way as the existing adding operation without using matching conditions. The Redis client can still search for data in the Redis database in the same way as the existing searching operation without using the matching conditions involved in this step.
[0043] Step 220: According to the matching condition, traverse each key stored in the Redis database and obtain at least one original key that meets the matching condition.
[0044] Specifically, according to the description of step 210, after receiving the first operation command, the Redis server obtains the operation attributes and matching conditions from the first operation command. Based on the operation attributes, the Redis server determines the specific operation to be performed by the first operation command, such as a rename operation or a delete operation.
[0045] Based on the matching conditions, the Redis server traverses each key stored in the Redis database and obtains at least one original key that meets the matching conditions.
[0046] It is understandable that, in one implementation, when the Redis server traverses each key stored in the Redis database according to the matching conditions, there is no original key in the Redis database that meets the matching conditions. If there is no original key in the Redis database that meets the matching conditions, the Redis server sends a response message to the Redis client, and the response message carries "empty" content, so that the Redis client determines that there is no original key in the Redis database that meets the matching conditions. In one example, the original key is composed of the following multiple fields: userinfo field, modulename field, datatype field, dataattrtype field, attrvalue field, and privatekey field.
[0047] In the embodiment of the present application, the matching conditions also include matching partitioning conditions and variable partitioning conditions. The variable partitioning conditions are fields in the matching conditions that can be divided into variables, and the matching partitioning conditions are the remaining fields in the matching conditions except the variable partitioning conditions.
[0048] When the first operation command is specifically a rename operation, the first operation command is specifically:
[0049] RENAME MATCH / ^userinfo:ipoe:(\d+):up:1024:(\S+)$ / userinfo:ipoe:$1:up:1025:$2
[0050] Each field is separated by a comma (:). "userinfo" corresponds to the "userinfo field," "ipoe" corresponds to the "modulename field," "(\d+)" corresponds to the "datatype field," "up" corresponds to the "dataattrtype field," "1024" corresponds to the "attrvalue field," and "(\S+)" corresponds to the "privatekey field."
[0051] The specific format for "find_regex" is " / ^userinfo:ipoe:(\d+):up:1024:(\S+)$ / "; the specific format for "replace_regex" is "userinfo:ipoe:$1:up:1025:$2". "\d+" matches one or more digits; "\S+" matches one or more characters; and "()" indicates that the current string is a variable.
[0052] In an embodiment of the present application, the aforementioned first operation command includes two variables, namely "(\d+)" and "(\S+)". The Redis server determines two variables from the first operation command based on the variable partitioning condition, and configures the two variables as "$1" and "$2". At the same time, the Redis server determines other fields except "(\d+)" and "(\S+)" as matching partitioning conditions, that is, the Redis server uses "userinfo", "ipoe", "up", and "1024" as matching partitioning conditions, and obtains the original key that matches the values of the above fields from the Redis database.
[0053] It should be noted that the process of determining the variable and the matching value on the Redis server can be performed simultaneously without regard to the time sequence.
[0054] Furthermore, after the Redis server obtains at least one original key that meets the matching conditions, it obtains at least one variable that meets the variable partitioning conditions from each original key according to the aforementioned process of determining the variable from the first operation command, and configures the variable as "$n".
[0055] For example, if the original key is userinfo:ipoe:1:up:1024:session1, then "$1" is 1 and "$2" is session1.
[0056] Furthermore, when the first operation command is a delete operation, the Redis server does not need to determine the variables in the same way as described above. For example, in the case of DEL MATCH / ^userinfo:ipoe:2:up:1024:session2$ / , "^userinfo:ipoe:2:up:1024:session2$" is the matching condition. The Redis server retrieves the original key from the Redis database that matches the values of the above fields.
[0057] It should be noted that the above-mentioned matching conditions include two special symbols "^" and "$". The content between the two special symbols is an exact matching condition, that is, the entire segment is matched, starting with "userinfo:", ending with "session2", and the characters between "userinfo:" and "session2" are the original key of "ipoe:2:up:1024:". If the matching conditions do not include the two special symbols, the matching conditions are fuzzy matching conditions, that is, no matter where the fuzzy matching conditions are in the key, if the key includes the module matching conditions, the key is the original key that meets the matching conditions. For example, if the characters corresponding to the fuzzy matching conditions are in the middle position of the key, the key is the original key that meets the matching conditions. For example, "abcdef userinfo:ipoe:2:up:1024:session2ghijklmn".
[0058] The aforementioned exact matching condition can be applied to rename operations and delete operations.
[0059] In an embodiment of the present application, when the Redis server performs a delete operation, the Redis server can also determine the variables in the aforementioned manner. For example, in the case of DEL MATCH / ^userinfo:ipoe:(\d+):up:1024:(\S+)$ / , "^userinfo:ipoe:(\S+):up:1024:(\S+)$" is the matching condition. The Redis server retrieves the original key from the Redis database that matches the values of the aforementioned fields. This allows for more efficient deletion of a specific type of key.
[0060] Step 230: Perform a corresponding operation on the at least one original key according to the operation attribute and record the operation result.
[0061] Specifically, according to the description of step 220, after the Redis server obtains at least one variable that meets the variable partitioning conditions from each original key, it performs a corresponding operation on the at least one original key according to the operation attributes and records the operation result.
[0062] In one example, when the first operation command is a rename operation, the Redis server populates the obtained at least one variable with the corresponding position in the replacement condition to obtain at least one reorganized key. The Redis server stores the at least one reorganized key and the value corresponding to the at least one reorganized key in the Redis database and deletes the at least one original key. The Redis server records the number of the at least one reorganized key stored in the Redis database.
[0063] In the example of step 220 , the replacement result is “replace_regex”, which is specifically “userinfo:ipoe:$1:up:1025:$2”.
[0064] The Redis server populates at least one variable from the original key into the "$1" and "$2" positions in the replacement result to create a reconstructed key. Specifically, the reconstructed key is "userinfo:ipoe:1:up:1025:session1".
[0065] In another example, when the first operation command is a delete operation, the Redis server obtains at least one original key that meets the matching condition from the Redis database and deletes the at least one original key. The Redis server records the number of the at least one original key deleted from the Redis database.
[0066] Step 240: Send a response message to the Redis client, where the response message includes the operation result.
[0067] Specifically, according to the description of step 230, after recording the operation result, the Redis server generates a response message, which includes the operation result. The operation result can specifically be the number of original keys renamed / deleted by the Redis server.
[0068] The Redis server sends a response message to the Redis client. After receiving the response message, the Redis client obtains the operation result from it. Based on the operation result, the Redis client determines whether the Redis server has executed the first operation command and the number of original keys renamed / deleted.
[0069] The Redis client can subsequently rename / delete the original keys based on the number of keys. For example, if the Redis server has renamed a certain type of original key in the Redis database and the client needs to rename another type of original key, the Redis client will send the operation command to the Redis server again.
[0070] Therefore, by applying the data processing method and device provided in this application, the Redis server receives a first operation command sent by the Redis client, the first operation command including operation attributes and matching conditions. Based on the matching conditions, the Redis server traverses each stored key in the Redis database and obtains at least one original key that meets the matching conditions. Based on the operation attributes, the Redis server performs a corresponding operation on the at least one original key and records the operation result. The Redis server sends a response message to the Redis client, the response message including the operation result.
[0071] In this way, the problem of the Redis client wasting time and occupying a large amount of memory when operating data in the Redis database in the existing solution is solved. At the same time, in this application, by carrying regular expressions in the operation instructions, the operation time between the Redis client and the Redis server is reduced, and the load on the Redis client is reduced.
[0072] Based on the same inventive concept, the embodiment of the present application also provides a data processing device corresponding to the data processing method. Figure 3 , Figure 3 This is a structural diagram of a data processing device provided in an embodiment of the present application. The device is applied to a Redis server, and the device includes:
[0073] A receiving unit 310 is configured to receive a first operation command sent by a Redis client, where the first operation command includes an operation attribute and a matching condition;
[0074] The acquisition unit 320 is configured to traverse each key stored in the Redis database according to the matching condition and obtain at least one original key that meets the matching condition;
[0075] An execution unit 330 is configured to perform a corresponding operation on the at least one original key according to the operation attribute and record the operation result;
[0076] The sending unit 340 is configured to send a response message to the Redis client, where the response message includes the operation result.
[0077] Optionally, the matching condition includes a variable partitioning condition;
[0078] The acquiring unit 320 is further configured to acquire, according to the variable partitioning condition, at least one variable that meets the variable partitioning condition from each original key.
[0079] Optionally, the operation attribute is specifically a rename operation, and the first operation command further includes a replacement result;
[0080] The execution unit 330 is specifically configured to fill the at least one variable into a corresponding position in the replacement result to obtain at least one reorganization key;
[0081] Storing the at least one reorganized key and the value corresponding to the at least one reorganized key in the Redis database, and deleting the at least one original key;
[0082] Record the number of the at least one reorganization key stored in the Redis database.
[0083] Optionally, the operation attribute is specifically a delete operation; the execution unit 330 is specifically configured to delete the at least one original key;
[0084] Record the number of the at least one original key deleted from the Redis database.
[0085] Optionally, the matching condition is specifically a regular expression.
[0086] Therefore, by applying the data processing device provided by the present application, the Redis server receives a first operation command sent by the Redis client, where the first operation command includes an operation attribute and a matching condition. Based on the matching condition, the Redis server traverses each stored key in the Redis database and obtains at least one original key that meets the matching condition. Based on the operation attribute, the Redis server performs a corresponding operation on the at least one original key and records the operation result. The Redis server sends a response message to the Redis client, where the response message includes the operation result.
[0087] In this way, the problem of the Redis client wasting time and occupying a large amount of memory when operating data in the Redis database in the existing solution is solved. At the same time, in this application, by carrying regular expressions in the operation instructions, the operation time between the Redis client and the Redis server is reduced, and the load on the Redis client is reduced.
[0088] Based on the same inventive concept, the embodiment of the present application further provides a network device, such as Figure 4 As shown, it includes a processor 410, a transceiver 420 and a machine-readable storage medium 430. The machine-readable storage medium 430 stores machine-executable instructions that can be executed by the processor 410. The processor 410 is prompted by the machine-executable instructions to execute the data processing method provided in the embodiment of the present application. Figure 3 The data processing device shown can be used as Figure 4 The network device hardware structure shown is implemented.
[0089] The computer-readable storage medium 430 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Alternatively, the computer-readable storage medium 430 may be at least one storage device located remotely from the processor 410.
[0090] The processor 410 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0091] In the embodiment of the present application, the processor 410 reads the machine-executable instructions stored in the machine-readable storage medium 430, and the machine-executable instructions enable the processor 410 itself and call the transceiver 420 to execute the data processing method described in the aforementioned embodiment of the present application.
[0092] In addition, an embodiment of the present application provides a machine-readable storage medium 430, which stores machine-executable instructions. When called and executed by the processor 410, the machine-executable instructions prompt the processor 410 itself and the calling transceiver 420 to execute the data processing method described in the aforementioned embodiment of the present application.
[0093] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0094] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.
[0095] As for the data processing device and machine-readable storage medium embodiments, since the method contents involved are basically similar to the aforementioned method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0096] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A data processing method, characterized in that: The method is applied to the Redis server, and the method includes: Receive a first operation command sent by a Redis client, where the first operation command includes an operation attribute and a matching condition, where the matching condition includes a matching partitioning condition and a variable partitioning condition; According to the matching condition, traverse each key stored in the Redis database and obtain at least one original key that meets the matching condition; According to the operation attribute, perform a corresponding operation on the at least one original key and record the operation result; Send a response message to the Redis client, where the response message includes the operation result.
2. The method according to claim 1, characterized in that The matching conditions include variable partitioning conditions; After obtaining at least one original key that meets the matching condition, the method further includes: According to the variable partitioning condition, at least one variable that meets the variable partitioning condition is obtained from each original key.
3. The method according to claim 2, characterized in that The operation attribute is specifically a rename operation, and the first operation command further includes a replacement result; The performing a corresponding operation on the at least one original key according to the operation attribute and recording the operation result specifically includes: Filling the at least one variable into a corresponding position in the replacement result to obtain at least one recombinant key; Storing the at least one reorganized key and the value corresponding to the at least one reorganized key in the Redis database, and deleting the at least one original key; Record the number of the at least one reorganization key stored in the Redis database.
4. The method according to claim 1, wherein The operation attribute is specifically a delete operation; The performing a corresponding operation on the at least one original key according to the operation attribute and recording the operation result specifically includes: Deleting the at least one original key; Record the number of the at least one original key deleted from the Redis database.
5. The method according to claim 1, wherein The matching condition is specifically a regular expression.
6. A data processing device, characterized in that: The device is applied to a Redis server, and includes: A receiving unit, configured to receive a first operation command sent by a Redis client, wherein the first operation command includes an operation attribute and a matching condition, wherein the matching condition includes a matching partitioning condition and a variable partitioning condition; an acquisition unit, configured to traverse each key stored in the Redis database according to the matching condition and acquire at least one original key that meets the matching condition; an execution unit, configured to execute a corresponding operation on the at least one original key according to the operation attribute and record the operation result; A sending unit is used to send a response message to the Redis client, where the response message includes the operation result.
7. The device according to claim 6, characterized in that The matching conditions include variable partitioning conditions; The acquisition unit is further configured to acquire, according to the variable partitioning condition, at least one variable that meets the variable partitioning condition from each original key.
8. The device according to claim 7, characterized in that The operation attribute is specifically a rename operation, and the first operation command further includes a replacement result; The execution unit is specifically configured to fill the at least one variable into a corresponding position in the replacement result to obtain at least one reorganization key; Storing the at least one reorganized key and the value corresponding to the at least one reorganized key in the Redis database, and deleting the at least one original key; Record the number of the at least one reorganization key stored in the Redis database.
9. The device according to claim 6, characterized in that The operation attribute is specifically a delete operation; The execution unit is specifically configured to delete the at least one original key; Record the number of the at least one original key deleted from the Redis database.
10. The device according to claim 6, characterized in that The matching condition is specifically a regular expression.