Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a data processing method and device, electronic equipment and storage medium. The method comprises: receiving a data processing request sent by a client, the data processing request being used to indicate the Nth query and carrying a target cursor parameter; the target cursor parameter is an encoding parameter generated in the (N-1)th query, N being a positive integer greater than 1; the target key corresponding to the target cursor parameter is obtained from the mapping relationship between the cursor parameter and the key, the target key indicating the end position of the (N-1)th query in the distributed key-value database; and the (N)th query is performed on the distributed key-value database according to the target key. According to the technical scheme of the present disclosure, the cursor parameter is configured according to the demand of the original command, the corresponding relationship between the cursor parameter and the key is stored, so that the starting position of the query can be accurately analyzed based on the target cursor parameter, and the distributed key-value database has good compatibility.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of Internet, and particularly relates to a data processing method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the rapid development of NoSQL databases (non-relational databases), Redis (Remote Dictionary Service) databases are widely used in e-commerce, social networking, gaming, education, finance, media and other fields. Many distributed KV databases of content platforms are compatible with Redis protocol.

[0003] Taking the Redis SCAN command (a command for incrementally iterating and querying elements in a set) as an example, the SCAN command uses high-bit carry addition to traverse the first-dimensional array, and the index of the array is returned as a cursor each time. However, since the distributed KV database cannot be compatible with the native SCAN command, the cursor in the native SCAN command usually needs to be replaced from an integer to the last key (string) returned to the client in the current iteration. Using non-Redis native commands, the business code usually needs to use a customized SDK (Software Development Kit). Therefore, with the rapid development and application of NoSQL databases, there is an urgent need to provide a solution that can improve the compatibility of distributed KV databases with different protocols. SUMMARY

[0004] The present disclosure provides a data processing method and device, electronic equipment, computer readable storage medium and computer program product to at least solve the problem of compatibility of distributed KV databases with different protocols in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, comprising:

[0006] receiving a data processing request sent by a client, the data processing request being used to indicate an Nth query, the data processing request carrying a target cursor parameter; the target cursor parameter is an encoding parameter generated in an (N-1)th query and sent to the client after the (N-1)th query is completed, and N is any positive integer greater than 1;

[0007] querying a target key corresponding to the target cursor parameter from a stored mapping relationship between cursor parameters and keys, the target key indicating an end position of the (N-1)th query in a distributed key-value database;

[0008] The Nth query is performed on the distributed key-value database, starting from the target key.

[0009] In one of the embodiments, the method further comprises:

[0010] In the Nth-1 query, a unique cursor identifier corresponding to the Nth-1 query is obtained;

[0011] A first time indication parameter corresponding to first time information is determined, the first time information indicating time information of performing the Nth-1 query;

[0012] The target cursor parameter is generated according to the cursor identifier and the first time indication parameter;

[0013] After the Nth-1 query is completed, the target cursor parameter is sent to the client.

[0014] In one of the embodiments, the method further comprises:

[0015] After the Nth-1 query is completed, an end key indicated by an end position of the Nth-1 query is determined;

[0016] A time-to-live value corresponding to the first time indication parameter is determined;

[0017] The time-to-live value and the target cursor parameter are spliced to obtain a cursor key, and a mapping relationship between the cursor key and the end key is established.

[0018] In one of the embodiments, the method further comprises:

[0019] When the cursor cleaning condition is met, a cleaning time interval is determined according to second time information when the cursor cleaning condition is met and a preset cursor survival time length, the cursor survival time length indicating a storable time length of the mapping relationship;

[0020] The cursor key whose time-to-live value is within the cleaning time interval is obtained;

[0021] The mapping relationship between the cursor key and the end key within the cleaning time interval is deleted.

[0022] In one of the embodiments, the target key corresponding to the target cursor parameter is obtained from the stored mapping relationship between the cursor parameter and the key, comprising:

[0023] The first time indication parameter is parsed from the target cursor parameter;

[0024] A current time-to-live value is generated according to the first time indication parameter;

[0025] concatenate the current time-to-live value and the target cursor parameter to obtain a cursor key corresponding to the target cursor parameter;

[0026] According to the cursor key corresponding to the target cursor parameter, the target key is obtained from the mapping relationship between the stored cursor key and the end key.

[0027] In one of the embodiments, the generating a current time-to-live value according to the first time indication parameter comprises:

[0028] determining a second time indication parameter corresponding to current time information;

[0029] In the case that the first time indication parameter is consistent with the second time indication parameter, the current time-to-live value is generated according to the current time information and the first time indication parameter;

[0030] In the case that the first time indication parameter is inconsistent with the second time indication parameter, the current time-to-live value is generated according to the current time information, a cursor time-to-live duration and the first time indication parameter.

[0031] In one of the embodiments, the determining a first time indication parameter corresponding to the first time information comprises:

[0032] obtaining a preset cursor time-to-live duration, the cursor time-to-live duration being used to indicate a storable duration of the mapping relationship;

[0033] obtaining a preset unit time parameter from the first time information as the first time indication parameter according to the cursor time-to-live duration and a preset data bit number.

[0034] In one of the embodiments, before the obtaining a unique cursor identifier corresponding to the Nth query, the method further comprises:

[0035] generating the unique cursor identifier by an ID generator built in the distributed key-value database, and storing the cursor identifier in a local memory.

[0036] According to a second aspect of the embodiments of the present disclosure, a data processing apparatus is provided, comprising:

[0037] The receiving module is configured to perform receiving a data processing request sent by a client, the data processing request being used to instruct to perform an Nth query and carrying a target cursor parameter, the target cursor parameter being an encoded parameter generated when an (N-1)th query is performed and being sent to the client after the (N-1)th query is completed, the N being any positive integer greater than 1.

[0038] The query module is configured to query a target key corresponding to the target cursor parameter from a stored mapping relationship between cursor parameters and keys, the target key indicating an end position of the N-1th query in the distributed key-value database.

[0039] The request processing module is configured to perform the Nth query on the distributed key-value database starting from the target key.

[0040] In one of the embodiments, the apparatus further comprises an identification obtaining module configured to obtain a unique cursor identification corresponding to the N-1th query when the N-1th query is performed.

[0041] The first time parameter determining module is configured to determine a first time indication parameter corresponding to first time information, the first time information indicating time information of performing the N-1th query.

[0042] The cursor parameter generating module is configured to generate the target cursor parameter according to the cursor identification and the first time indication parameter.

[0043] The sending module is configured to send the target cursor parameter to the client after the N-1th query is completed.

[0044] In one of the embodiments, the apparatus further comprises:

[0045] The end key determining module is configured to determine an end key indicated by the end position of the N-1th query after the N-1th query is completed.

[0046] The time value determining module is configured to determine a time-to-live value corresponding to the first time indication parameter.

[0047] The cursor key generating module is configured to splice the time-to-live value and the target cursor parameter to obtain a cursor key, and establish a mapping relationship between the cursor key and the end key.

[0048] In one of the embodiments, the apparatus further comprises:

[0049] The time interval determining module is configured to determine a cleaning time interval according to second time information when the cursor cleaning condition is met and a preset cursor survival time length indicating a storable time length of the mapping relationship, in a case that the cursor cleaning condition is met.

[0050] The cursor key obtaining module is configured to obtain a cursor key of the time-to-live value within the cleaning time interval.

[0051] A deleting module is configured to perform deleting a mapping relationship between a cursor key and an end key in the cleaning time interval.

[0052] In one of the embodiments, the querying module comprises:

[0053] A parsing unit is configured to perform parsing the target cursor parameter to obtain the first time indication parameter.

[0054] A time value determining unit is configured to perform generating a current survival time value according to the first time indication parameter.

[0055] A cursor key generating unit is configured to perform splicing the current survival time value and the target cursor parameter to obtain a cursor key corresponding to the target cursor parameter.

[0056] A querying unit is configured to perform querying the target key from a stored mapping relationship between a cursor key and an end key according to the cursor key corresponding to the target cursor parameter.

[0057] In one of the embodiments, the time value determining unit is configured to perform determining a second time indication parameter corresponding to current time information; in a case where the first time indication parameter is consistent with the second time indication parameter, generating the current survival time value according to the current time information and the first time indication parameter; in a case where the first time indication parameter is inconsistent with the second time indication parameter, generating the current survival time value according to the current time information, a cursor survival time length and the first time indication parameter.

[0058] In one of the embodiments, the first time parameter determining module comprises:

[0059] A survival time length obtaining unit is configured to perform obtaining a preset cursor survival time length, the cursor survival time length being used to indicate a storable time length of the mapping relationship.

[0060] A time parameter obtaining unit is configured to perform obtaining a time parameter of a preset unit from the first time information as the first time indication parameter according to the cursor survival time length and a preset data bit number.

[0061] In one of the embodiments, further comprising an identifier storing module configured to perform generating a unique cursor identifier by an ID generator built in the distributed key-value database, and storing the cursor identifier into a local memory; and the identifier obtaining module is configured to obtain a cursor identifier corresponding to the N-1th query from the local memory.

[0062] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0063] a processor;

[0064] a memory for storing instructions executable by the processor;

[0065] The processor is configured to execute the instructions to implement the data processing method according to any one of the embodiments of the first aspect.

[0066] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the data processing method according to any one of the embodiments of the first aspect.

[0067] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, the computer program product comprises instructions, when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the data processing method according to any one of the embodiments of the first aspect.

[0068] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0069] The target cursor parameter is obtained by encoding according to the demand of the original command. The target cursor parameter is sent to the client after the N-1th(N is a positive integer greater than 1) query is completed, and the corresponding relationship between the target cursor parameter and the target key (the end position of the N-1th query in the distributed key-value database) is stored in the mapping relationship between the cursor parameter and the key. Therefore, when the Nth command is called, the corresponding target key can be obtained by querying according to the target cursor parameter, and the Nth query can be performed from the target key in the distributed key-value database, so that the starting position of the Nth query can be accurately parsed based on the target cursor parameter, ensuring data consistency and good compatibility of the distributed database.

[0070] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0071] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0072] Figure 1 is a data processing method according to an exemplary embodiment.

[0073] Figure 2is a flow chart of a data processing method according to an exemplary embodiment.

[0074] Figure 3 is a flow chart of generating a cursor parameter according to an exemplary embodiment.

[0075] Figure 4 is a schematic diagram of determining a time indication parameter according to an exemplary embodiment.

[0076] Figure 5 is a flow chart of cleaning a mapping relationship between a cursor parameter and a key according to an exemplary embodiment.

[0077] Figure 6 is a flow chart of calibrating a time indication parameter according to an exemplary embodiment.

[0078] Figure 7 is a flow chart of a data processing method according to an exemplary embodiment.

[0079] Figure 8 is a block diagram of a data processing apparatus according to an exemplary embodiment.

[0080] Figure 9 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0081] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings.

[0082] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0083] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0084] In order to solve the blocking situation caused by Redis single thread batch acquisition of the same style key, Redis provides the SCAN command and its related SSCAN command, HSCAN command and ZSCAN command (command for incrementally iterating the elements in the collection), through which the elements in the collection are incrementally iterated. Among them, the SCAN command is used to iterate the database keys in the current database; the SSCAN command is used to iterate the elements in the set key; the HSCAN command is used to iterate the key-value pairs in the hash key; and the ZSCAN command is used to iterate the elements (including element members and element scores) in the sorted set. The following are the syntaxes of the SCAN command, the SSCAN command, the HSCAN command and the ZSCAN command respectively:

[0085] SCAN cursor[MATCH pattern][COUNT count]

[0086] SSCAN key cursor[MATCH pattern][COUNT count]

[0087] HSCAN key cursor[MATCH pattern][COUNT count]

[0088] ZSCAN key cursor[MATCH pattern][COUNT count]

[0089] Wherein, cursor: cursor, indicating the starting position of traversal, when the cursor is 0, it indicates the start of traversing a collection, and when the returned cursor is 0, it indicates that the collection has been traversed.

[0090] [MATCH pattern]: regular expression used to filter matching elements.

[0091] [COUNT count]: the number of elements returned each time.

[0092] However, due to the difference in data storage structure, the distributed KV (key value) database cannot be fully compatible with the SCAN command and related commands, and therefore, the native cursor is usually replaced from an integer to a string. Taking the SCAN command as an example, the syntax of the SCAN command is changed from SCAN cursor[MATCH pattern][COUNT count] to SCAN key[MATCH pattern][COUNT count], wherein the key is the last key returned to the client in the last iteration. Since the above syntax is a non-Redis native command, secondary development is required to support the SDK, and multiple development languages such as Java, C / C++, PHP, Golang, Rust, etc. need to be adapted, which has high development and maintenance costs; and the non-Redis native command requires the business code to use the customized SDK, which is deeply bound to a specific database product, and the portability is limited.

[0093] To solve the above problems, the present disclosure provides a data processing method, which can be applied to an application environment as shown in Figure 1 The terminal 110 communicates with the server 120 through a network. The data storage system can store data required to be processed by the server 120. The data storage system can be integrated on the server 120, or placed on a cloud or other network server. The terminal 110 is deployed with a client, which can be but is not limited to a video type, a social type, an instant messaging type, an e-commerce type, etc. The server 120 is deployed with a distributed key value database based on the Redis protocol, and the server 120 can scan elements in the distributed key value database by using the SCAN command and related commands. In a specific implementation, the server 120 generates a target cursor parameter when performing the N-1th (N is any positive integer greater than 1) query, and sends the target cursor parameter to the terminal 110. The server 120 receives a data processing request sent by the terminal 110, and the data processing request is used to indicate the Nth query, and the data processing request carries the target cursor parameter. The server 120 queries the target key corresponding to the target cursor parameter from the stored mapping relationship between the cursor parameter and the key; and performs the Nth query on the distributed key value database starting from the target key.

[0094] The terminal 110 can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers.

[0095] Figure 2is a flowchart of a data processing method according to an exemplary embodiment, as Figure 2 As shown, the data processing method is used in a server and includes steps S210-S230.

[0096] In step S210, a data processing request sent by a client is received.

[0097] The data processing request can be, but is not limited to, a request for obtaining comment data, a request for obtaining session messages, a request for updating resources, and the like. In this embodiment, the data processing request carries a target cursor parameter, which is used to indicate an Nth iteration query, where N is any positive integer greater than 1. The target cursor parameter is an encoding parameter generated in an (N-1)th query and is used to uniquely represent the (N-1)th query.

[0098] Specifically, when the server performs an iteration query on the distributed key-value database by invoking a command in the (N-1)th time, a preset algorithm is used to generate the target cursor parameter. The encoding format of the target cursor parameter is determined according to the native protocol currently supported by the distributed key-value database. In one example, if the distributed key-value database is a database based on the Redis protocol, the target cursor parameter is a positive integer set according to the requirements of the Redis native command. After the (N-1)th query is completed, the server returns the target cursor parameter and an element set obtained by the query to the client. The preset algorithm can be implemented based on a database auto-increment ID, a UUID (universal unique identifier), or the like. It can be understood that the elements in the element set obtained in the (N-1)th query are elements that satisfy [MATCH pattern], and the number of elements in the set is not more than count in the syntax, or the set can be empty.

[0099] In step S220, a target key corresponding to the target cursor parameter is queried from a stored mapping relationship between cursor parameters and keys.

[0100] The target key indicates an end position of the (N-1)th query in the distributed key-value database.

[0101] Specifically, when the (N-1)th query is completed, the server obtains a target key corresponding to the end position of the query, generates a mapping relationship between the target cursor parameter and the target key by taking the target cursor parameter as the key and the target key as the value, and stores the mapping relationship in a cursor set. After the server receives the data processing request indicating the Nth query, the target key corresponding to the target cursor parameter carried in the data processing request is queried from the cursor set.

[0102] In step S230, the Nth query is performed on the distributed key-value database by taking the target key as a starting position.

[0103] Specifically, the server invokes the command for the Nth time, takes the target key as the starting position of the iterative query, and performs the Nth query on the distributed key-value database to obtain an element set corresponding to the Nth query. The server generates a next target cursor parameter and returns the next target cursor parameter and the element set obtained by the Nth query to the client. If the next target cursor parameter is not the cursor 0 indicating stop, steps S210-S230 can be repeated to continue the iterative query on the distributed key-value database; if the next target cursor parameter is the cursor 0 indicating stop, the traversal of the distributed key-value database is completed.

[0104] In the above data processing method, the target cursor parameter is obtained according to the requirement of the native command. After the (N-1)th query is completed, the target cursor parameter is sent to the client, and the correspondence between the target cursor parameter and the target key is stored in the mapping relationship between the cursor parameter and the key. Thus, when the command is invoked for the Nth time, the corresponding target key can be obtained according to the target cursor parameter, and the Nth query can be performed from the target key in the distributed key-value database. Thus, the starting position of the Nth query can be accurately parsed based on the target cursor parameter, ensuring data consistency and good compatibility of the distributed database.

[0105] In an example embodiment, a generation manner of the target cursor parameter is described. As shown in FIG. 3, the generation of the target cursor parameter can be implemented by steps S310-S340. Figure 3 As shown in FIG. 3, the generation of the target cursor parameter can be implemented by steps S310-S340.

[0106] In step S310, when the (N-1)th query is performed, a unique cursor identifier corresponding to the (N-1)th query is obtained.

[0107] In step S320, a first time indication parameter corresponding to first time information is determined.

[0108] In step S330, the target cursor parameter is generated according to the cursor identifier and the first time indication parameter.

[0109] In step S340, after the (N-1)th query is completed, the target cursor parameter is sent to the client.

[0110] The data format of the cursor identifier can be determined according to the native protocol currently compatible with the distributed key-value database. In an example, if the distributed key-value database is a database based on the Redis protocol, the cursor identifier is a unique positive integer.

[0111] The first time information represents time information of performing the (N-1)th query, which may, but is not limited to, include any one or more of an iteration date (for example, week, month, year, etc.), an iteration start time, an iteration end time, and any time during the iteration.

[0112] The data format of the first time indication parameter can be determined according to the native protocol currently compatible with the distributed key-value database. In one example, if the distributed key-value database is a database based on the Redis protocol, the first time indication parameter is a positive integer. In one example, the first time indication parameter can be a parameter value extracted from the first time information, for example, one or more values of the year, month, hour, etc. in the first time information are extracted as the first time indication parameter. In another example, the first time indication parameter can be a value obtained by conversion processing based on the first time information.

[0113] Specifically, an ID (identification) generator is deployed in the server, which can be implemented based on UUID, database auto-increment, snowflake algorithm, etc. When the server performs the N-1th query on the distributed key-value database, the ID generator is used to generate a unique cursor identifier corresponding to the N-1th query. The first time information corresponding to the N-1th query is obtained, and the corresponding first time indication parameter is obtained based on the first time information. The server can process the first time indication parameter and the cursor identifier based on a preset algorithm to obtain a target cursor parameter. After the N-1th query is completed, the target cursor parameter and the element set obtained by iterative query are returned to the client.

[0114] In this embodiment, by adding the time information of the iterative query in the cursor parameter, subsequent data applications such as quickly retrieving the cursor parameter from the cursor set and cleaning the cursor parameter are facilitated, and the maintainability and use efficiency of the data are improved.

[0115] In an example embodiment, the ID generator is built into the distributed key-value database, which can apply a continuous cursor identifier according to a certain step size, and cache these continuous cursor identifiers in the local memory. When the server performs the N-1th query, the cursor identifier corresponding to the N-1th query can be obtained from the local memory through the alloc ID (allocate ID) interface. In this embodiment, by pre-caching the cursor identifier in the local memory, the application of the cursor identifier can be completed in the local memory, and the efficiency of obtaining the cursor identifier can be improved.

[0116] In an example embodiment, step S320, determining the first time indication parameter corresponding to the first time information, includes: obtaining a preset cursor survival duration; obtaining a time parameter of a preset unit from the first time information as the first time indication parameter according to the cursor survival duration and a preset number of data bits.

[0117] The cursor survival time length represents the storable time length of the mapping relationship in the cursor set. For example, if the mapping relationship in the cursor set needs to be retained for 24 hours, the cursor survival time length can be 24 hours; if the mapping relationship in the cursor set needs to be retained for one month, the cursor survival time length can be one month. The actual demand can be configured.

[0118] The preset unit can be years, months, days, hours, minutes, etc., which can be determined according to the setting of the cursor survival time length. In one example, the cursor survival time length is 24 hours, and the preset unit is hour, so the first time indication parameter can be obtained according to the value of the preset unit in the first time information. In another example, the cursor survival time length is one month, so the first time indication parameter can be obtained according to the value of the preset unit in the first time information.

[0119] The preset data bit number can be determined according to the cursor time length and / or the preset unit. In one example, the target cursor parameter is 64 bits. If the cursor time length is 24 hours and the preset unit is hour, 3 bits of the target cursor parameter can be allocated to the first time indication parameter, and the remaining part can be allocated to the cursor identifier. The preset data bit number can also be a fixed number of bits, for example, the preset bit number can be configured as 4 bits. 4 bits include 32 bits, and when the preset unit is month, the value range of the first time indication parameter is the widest, which is 31, and 4 bits can meet most use requirements.

[0120] In one example, the encoding rule of the target cursor parameter can be |SLOT|ID|, wherein SLOT represents the first time indication parameter, occupying 4 bits; ID is the cursor identifier, occupying 60 bits. If the cursor survival time length is 24 hours and the preset unit is hour, Figure 4 The 24-hour ring SLOT is shown, that is, a total of 24 SLOTs need to be maintained, and each SLOT stores the clock (with an accuracy of hours) corresponding to the current SLOT value. Table 1 shows the clock distribution of the first time indication parameter. For example, the first time information is 2021-01-22 10:39:23, and the first time indication parameter is 11. If the cursor identifier is 12, the target cursor parameter (cursor) obtained through the preset algorithm is 6917529027641081843.

[0121] The preset algorithm can be:

[0122] cursor = unit64(11) << 59 | (12 & 0x07FFFFFFFFFFFFFF)

[0123] Table 1, clock distribution of the first time indication parameter:

[0124] Time indication parameter 1 2 3 4 Time information 2021-01-22 00 2021-01-22 01 2021-01-22 02 2021-01-22 03 Time indication parameter 5 6 7 8 Time information 2021-01-22 04 2021-01-22 05 2021-01-22 06 2021-01-22 07 Time indication parameter 9 10 11 12 Time information 2021-01-22 08 2021-01-22 09 2021-01-22 10 2021-01-22 11 Time indication parameter 13 14 15 16 Time information 2021-01-21 12 2021-01-21 13 2021-01-21 14 2021-01-21 15 Time indication parameter 17 18 19 20 Time information 2021-01-21 16 2021-01-21 17 2021-01-21 18 2021-01-21 19 Time indication parameter 21 22 23 24 Time information 2021-01-21 20 2021-01-21 21 2021-01-21 22 2021-01-21 23

[0125] In this embodiment, the time indication parameter in the cursor parameter is set based on the cursor storage time length, so that when subsequent data applications such as quickly retrieving cursor parameters from the cursor set, cleaning the cursor parameters, and the like are performed, the required cursor parameter can be quickly found from the cursor set, and the maintainability and use efficiency of the data are improved.

[0126] In an example embodiment, in the case where the target cursor parameter is generated according to the first time indication parameter, the method further includes: after the N-1th query is completed, determining an end key indicated by an end position of the N-1th query; determining a survival time value corresponding to the first time indication parameter; splicing the survival time value and the target cursor parameter to obtain a cursor key; and establishing a mapping relationship between the cursor key and the end key.

[0127] Specifically, after the N-1th query is completed, the server obtains an end key corresponding to an end position at the end of the query as a target key. Based on the first time indication parameter, a survival time value with a precision of the above-mentioned preset unit is obtained, for example, the first time information is 2021-01-22 10:39:23, the first time indication parameter is 11, and the survival time value generated is 2021-01-22 10:00:00. The server splices the survival time value and the target cursor parameter by taking the survival time value as a prefix or a suffix to obtain a cursor key. The mapping relationship between the cursor key and the target key is generated by taking the cursor key as the key and the target key as the value, and the mapping relationship is stored in the cursor set. In one example, the survival time value can be taken as a prefix, the survival time value and the target cursor parameter are spliced to obtain a cursor key. The encoding format is |TTL|cursor|, where TTL represents the survival time value, and cursor represents the target cursor parameter.

[0128] In one embodiment, for a distributed key-value database, a copy of the cursor set can be cached on a local server, so that the above-mentioned mapping relationship can be obtained by reading from the local server, avoiding reading from the disk, and improving the performance of reading data.

[0129] In this embodiment, after the server receives a data processing request indicating the Nth query, the first time indication parameter can be parsed from the target cursor parameter carried in the data request, and the current survival time value is generated based on the first time indication parameter and the current time information. The current survival time value and the target cursor parameter are spliced in the same splicing manner as when the cursor key is generated to obtain the cursor key corresponding to the target cursor parameter. Then, according to the cursor key corresponding to the target cursor parameter, the target key is obtained from the mapping relationship between the stored cursor key and the end key.

[0130] In one example, the first time information of the N-1th query is 2021-01-22 10:39:23, the first time indication parameter is 11, and the generated survival time value is 2021-01-22 10:00:00. The survival time value 2021-01-22 10:00:00 and the target cursor parameter are spliced to generate a cursor key, and the mapping relationship between the cursor key and the target key is stored. If a data processing request is received at 2021-01-22 10:39:44, the first time indication parameter is parsed from the target cursor parameter carried in the data processing request as 11. The current survival time value 2021-01-22 10:00:00 (with an accuracy of hours) can be generated according to the current time information (2021-01-22 10:39:44) and the first time indication parameter 11. Then, the cursor key is obtained by splicing the current survival time value 2021-01-22 10:00:00 and the target cursor parameter, and the corresponding target key is found based on the cursor key.

[0131] In one embodiment, if the current time information is inconsistent with the first time information, resulting in inconsistency between the current survival time value and the survival time value generated at the N-1th query, the corresponding target key cannot be found based on the currently obtained cursor key. The server can determine that the mapping relationship between the cursor key and the target key stored in the cursor set has expired, and return an error prompt information to the client. Continue to illustrate the above example. If a data processing request is received at 2021-01-23 10:39:44, the first time indication parameter is parsed from the target cursor parameter carried in the data processing request as 11. The current survival time value 2021-01-23 10:00:00 (with an accuracy of hours) can be generated according to the current time information (2021-01-23 10:39:44) and the first time indication parameter 11. Then, the cursor key is obtained by splicing the current survival time value 2021-01-23 10:00:00 and the target cursor parameter. However, since the cursor key stored in the cursor set is obtained by splicing the survival time value 2021-01-22 10:00:00 and the target cursor parameter, the corresponding target key cannot be found based on the currently generated cursor key. The server can return an expired error prompt information to the client.

[0132] In this embodiment, the survival time value is used as a prefix or suffix to generate a cursor key, and the mapping relationship between the cursor key and the target key is stored in the cursor set. When subsequent operations such as query and data cleaning of the cursor set are performed, the required data can be quickly located based on the survival time value, thereby improving the efficiency of subsequent data application.

[0133] In one example embodiment, the mapping relationship between the cursor key and the target key is stored in the cursor set, as followsFigure 5 As shown, the method further includes steps S510-S530.

[0134] In step S510, when the cursor cleaning condition is met, a cleaning time interval is determined according to the second time information when the cursor cleaning condition is met and the preset cursor survival time length.

[0135] In step S520, the cursor key with the survival time value in the cleaning time interval is obtained.

[0136] In step S530, the mapping relationship between the cursor key and the end key in the cleaning time interval is deleted.

[0137] The second time information is used to indicate whether the cursor cleaning condition is met, for example, a timing task is started inside the distributed key-value database, which is used to periodically clean the cursor set. If the current second time information meets the cleaning time of the timing task, for example, the cleaning time is 24:00 every day, and the current second time information is 24:00, it is determined that the second time information meets the cursor cleaning condition. The server obtains the same time information as the cursor survival time length based on the second time information, calculates the difference between the extracted time information and the cursor survival time length, and obtains the cleaning time interval. The server compares the survival time value in the cursor key with the cleaning time interval, and if the survival time value is in the cleaning time interval, the mapping relationship between the cursor key and the end key in the cleaning time interval is deleted.

[0138] In one example, the cursor survival time length is 24 hours, and the current time information is 2021-01-23 24:00:00. The cleaning time interval can be obtained as (negative infinity, 2021-01-22 24:00:00), that is, the mapping relationship between the cursor key and the end key with the survival time value before 2021-01-22 24:00:00 is deleted.

[0139] In this embodiment, the survival time value is used as the prefix or suffix to generate the cursor key, and the mapping relationship between the cursor key and the target key is stored in the cursor set, which facilitates subsequent data cleaning operation on the cursor set, and quickly locates the data to be cleaned based on the survival time value, thereby improving the subsequent data application efficiency and helping to improve the database performance.

[0140] In an example embodiment, as shown in Figure 6 The server generates the current survival time value based on the first time indication parameter and the current time information in response to the data processing request of the Nth query, which can be implemented by steps S610-S630.

[0141] In step S610, a second time indication parameter corresponding to the current time information is determined.

[0142] In step S620, in a case where the first time indication parameter is consistent with the second time indication parameter, a current survival time value is generated according to the current time information and the first time indication parameter.

[0143] In step S630, in a case where the first time indication parameter is inconsistent with the second time indication parameter, a current survival time value is generated according to the current time information, the cursor survival duration, and the first time indication parameter.

[0144] Specifically, for a distributed system, there is an objective clock deviation between different servers in a cluster, and in the case of NTP (Network Time Protocol) service guarantee, the clock deviation is usually less than 500 ms (milliseconds). Therefore, for the server currently receiving the data processing request, after receiving the data processing request, the first time indication parameter can be parsed from the target cursor parameter. The second time indication parameter corresponding to the current time information indicated by the clock of the server itself is obtained. The first time indication parameter and the second time indication parameter are compared. If they are consistent, a current survival time value is generated based on the current time information and the first time indication parameter.

[0145] If they are inconsistent, a current survival time value can be generated based on the current time information and the first time indication parameter, and a cursor key is generated using the current survival time value to query the target key. If the corresponding target key is not queried, the current time information can be adjusted based on the cursor survival duration to obtain the last period time information of the current time information, and a new current survival time value is generated according to the last period time information and the first time indication parameter, and a cursor key is generated using the new current survival time value to query the target key.

[0146] In one example, the first time indication information is 2021-05-31 10:59:59.123 (milliseconds), and the corresponding first time indication parameter is 11 (i.e., the first time indication parameter parsed from the target cursor parameter is 11). If the current time information is 2021-05-31 10:59:59.000, the corresponding second time indication parameter is also 11. The first time indication parameter and the second time indication parameter are consistent, and further according to the current time information 2021-05-31 10:59:59.000 and the first time indication parameter 11, a current survival time value of 2021-05-31 10:00:00 is generated.

[0147] If the current time information is 2021-05-31 11:00:00.123, the corresponding second time indication parameter is 12, and the first time indication parameter and the second time indication parameter are inconsistent. The server can first generate a current survival time value as 2021-05-31 10:00:00 according to the current time information 2021-05-31 11:00:00.123 and the first time indication parameter 11, and query the target key based on the current survival time value. If the target key can be queried based on the current survival time value, the Nth query is continued to be executed. If the target key cannot be queried, the server determines the last period time information (i.e., 2021-05-30 11:00:00.123) of the current time information according to the survival time length of the cursor (for example, 24 hours), generates a new current survival time value as 2021-05-30 10:00:00 according to the last period time information and the first time indication parameter, and queries the target key based on the new current survival time value.

[0148] In the embodiment, before the iterative query is executed, the clock deviation correction is performed on the current clock information of the server and the time indication information indicated in the cursor parameter, so as to avoid the problem of inconsistent clocks of different servers in the distributed system, and the accuracy of data query can be improved.

[0149] In an example embodiment, as shown in FIG. 7, Figure 7 a flowchart of a data processing method is provided, taking the example of the data processing method applied to the SCAN command, including steps S702-S720.

[0150] In step S702, a data processing request of a client is received, and the data processing request carries a query field and a target cursor parameter. The target cursor parameter of the first query is 0.

[0151] In step S704, according to the query field, the target cursor parameter, and the pre-configured element quantity, the calling instruction of the SCAN command is generated based on the syntax corresponding to the SCAN command.

[0152] In step S706, the SCAN command is called to perform iterative query on the distributed key-value database, and an element set is obtained.

[0153] In step S708, before the element set is returned to the client, a first time indication parameter is obtained according to the first time information indicated by the first query. A globally unique ID is obtained through an interface as a cursor identifier.

[0154] In step S710, a next target cursor parameter is generated according to the first time indication parameter and the cursor identifier.

[0155] In step S712, a cursor key is generated according to the time-to-live value corresponding to the first time indication parameter and the next target cursor parameter. A mapping relationship between the cursor key and the target key is stored in the cursor set. The target key is the key corresponding to the end position of the first iteration query.

[0156] In step S714, the next target cursor parameter and the element set obtained by the iteration query are returned to the client.

[0157] In step S716, a new data processing request of the client is received, and the new data processing request carries the next target cursor parameter generated in step S710.

[0158] In step S718, a current time-to-live value is generated based on the current time information and the first time indication parameter parsed from the next target cursor parameter, and a cursor key is generated according to the current time-to-live value. The specific generation method of the current time-to-live value can refer to the above embodiment, which will not be described in detail here.

[0159] In step S720, the target key corresponding to the cursor key is queried from the cursor set.

[0160] The above steps S706 and S720 are repeated until the target cursor parameter 0 indicating stop is returned to the client, and the traversal of the distributed key-value database is completed.

[0161] It should be understood that although each step in the above flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise stated herein, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowchart can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0162] It can be understood that the same / similar parts of each embodiment of the above method in the specification can be mutually referred to, and each embodiment mainly explains the difference from other embodiments, and the related part can refer to the description of other method embodiments.

[0163] Figure 8 is a block diagram of a data processing apparatus 800 according to an exemplary embodiment. Referring to Figure 8 The apparatus includes a receiving module 802, a querying module 804, and a request processing module 806.

[0164] The receiving module 802 is configured to receive a data processing request sent by a client, the data processing request being used to instruct an Nth query, and carrying a target cursor parameter; the target cursor parameter is an encoded parameter generated in an (N-1) th query and sent to the client after the (N-1) th query is completed, and N is any positive integer greater than 1;

[0165] The querying module 804 is configured to query a target key corresponding to the target cursor parameter from a stored mapping relationship between a cursor parameter and a key.

[0166] The request processing module 806 is configured to perform the Nth query on the distributed key-value database with the target key as a starting position.

[0167] In an example embodiment, the apparatus 800 further includes an identification obtaining module configured to obtain a unique cursor identification corresponding to the (N-1) th query when the (N-1) th query is performed; a first time parameter determining module configured to determine a first time indication parameter corresponding to first time information, the first time information representing time information when the (N-1) th query is performed; a cursor parameter generating module configured to generate the target cursor parameter according to the cursor identification and the first time indication parameter; and a sending module configured to send the target cursor parameter to the client after the (N-1) th query is completed.

[0168] In an example embodiment, the apparatus 800 further includes an end key determining module configured to determine an end key indicated by an end position of the (N-1) th query after the (N-1) th query is completed; a time value determining module configured to determine a survival time value corresponding to the first time indication parameter; a cursor key generating module configured to splice the survival time value and the target cursor parameter to obtain a cursor key, and establish a mapping relationship between the cursor key and the end key; in this embodiment, the querying module 804 is configured to query the target key corresponding to the target cursor parameter from the stored mapping relationship between the cursor key and the end key.

[0169] In an example embodiment, the apparatus 800 further includes a time interval determining module configured to determine a cleaning time interval according to second time information when a cursor cleaning condition is met and a preset cursor survival time length, the cursor survival time length being used to indicate a storable time length of the mapping relationship, in a case where the cursor cleaning condition is met; a cursor key obtaining module configured to obtain a cursor key with a survival time value within the cleaning time interval; and a deleting module configured to delete the mapping relationship between the cursor key and the end key within the cleaning time interval.

[0170] In an example embodiment, the query module 804 comprises: a parsing unit configured to perform parsing a first time indication parameter from the target cursor parameter; a time value determination unit configured to perform determining a time to live value corresponding to the first time indication parameter; a cursor key generation unit configured to perform concatenating the time to live value and the target cursor parameter to obtain a cursor key corresponding to the target cursor parameter; and a query unit configured to perform querying the target key from a stored mapping relationship between the cursor key and the end key according to the cursor key corresponding to the target cursor parameter.

[0171] In one of the embodiments, the time value determination unit is configured to perform determining a second time indication parameter corresponding to the current time information; in a case where the first time indication parameter is consistent with the second time indication parameter, generating a current time to live value according to the current time information and the first time indication parameter; and in a case where the first time indication parameter is inconsistent with the second time indication parameter, generating the current time to live value according to the current time information, the cursor time to live value and the first time indication parameter.

[0172] In an example embodiment, the first time parameter determination module comprises: a time to live value acquisition unit configured to perform acquiring a preset cursor time to live value, the cursor time to live value being used to indicate a storable time length of the mapping relationship; and a time parameter acquisition unit configured to perform acquiring a time parameter of a preset unit from the first time information according to the cursor time to live value and a preset data bit number, as the first time indication parameter.

[0173] In an example embodiment, the apparatus further comprises an identifier storage module configured to perform generating a unique cursor identifier by an ID generator built in the distributed key-value database, and storing the cursor identifier into a local memory; and an identifier acquisition module configured to perform acquiring the cursor identifier corresponding to the N-1th query from the local memory.

[0174] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0175] Figure 9 FIG. 8 is a block diagram of an electronic device S00 for processing data according to an example embodiment. For example, the electronic device S00 can be a server. Refer to FIG. 1 for the electronic device S00. Figure 9The electronic device S00 includes a processing component S20, which further includes one or more processors, and a memory resource represented by a memory S22 for storing instructions, such as application programs, executable by the processing component S20. The application programs stored in the memory S22 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute the instructions to perform the above method.

[0176] The electronic device S00 can further include a power component S24 configured to perform power management of the electronic device S00, a wired or wireless network interface S26 configured to connect the electronic device S00 to a network, and an input and output (I / O) interface S28. The electronic device S00 can operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0177] In an exemplary embodiment, a computer readable storage medium including instructions, such as the memory S22 including instructions, is also provided, which can be executed by the processor of the electronic device S00 to complete the above method. The storage medium can be a computer readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0178] In an exemplary embodiment, a computer program product including instructions, which can be executed by the processor of the electronic device S00 to complete the above method, is also provided.

[0179] It should be noted that the above-described apparatus, electronic device, computer readable storage medium, computer program product, etc. according to the description of the method embodiments can also include other implementations, and the specific implementation can refer to the description of the related method embodiments, which will not be described here.

[0180] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the aspects of the present disclosure disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations resulting from combinations, sub-combinations, and equivalents of the fundamental features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0181] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings and that various modifications and changes can be made therein without departing from the scope thereof. The scope of the present disclosure is indicated by the appended claims rather than by the description and examples.

Claims

1. A data processing method, characterized by, The method comprises: receiving a data processing request sent by a client, the data processing request being used to indicate an Nth query, the data processing request carrying a target cursor parameter; the target cursor parameter being an encoded parameter generated in an (N-1)th query and being sent to the client after the (N-1)th query is completed, N being any positive integer greater than 1; querying a target key corresponding to the target cursor parameter from a stored mapping relationship between cursor parameters and keys, the target key indicating an end position of the (N-1)th query in a distributed key-value database; in the mapping relationship, the target cursor parameter is used as a key, and the target key is used as a value; performing the Nth query on the distributed key-value database with the target key as a starting position; a target cursor parameter generation process comprising: obtaining a cursor identifier corresponding to the (N-1)th query when the (N-1)th query is performed; determining a first time indication parameter corresponding to first time information, the first time information indicating time information of performing the (N-1)th query; and generating the target cursor parameter according to the cursor identifier and the first time indication parameter; querying the target key corresponding to the target cursor parameter from the stored mapping relationship between cursor parameters and keys comprises: parsing the first time indication parameter from the target cursor parameter; generating a current survival time value according to the first time indication parameter; splicing the current survival time value and the target cursor parameter to obtain a cursor key corresponding to the target cursor parameter; and querying the target key from a stored mapping relationship between the cursor key and an end key according to the cursor key corresponding to the target cursor parameter; a mapping relationship construction process between the cursor key and the end key comprising: determining an end key indicated by an end position of the (N-1)th query after the (N-1)th query is completed; determining a survival time value corresponding to the first time indication parameter; and splicing the survival time value and the target cursor parameter to obtain a cursor key, and establishing a mapping relationship between the cursor key and the end key.

2. The data processing method according to claim 1, characterized in that, The method further comprises: sending the target cursor parameter to the client after the (N-1)th query is completed.

3. The data processing method of claim 1, wherein, The method further comprises: in a case where a cursor cleaning condition is met, determining a cleaning time interval according to second time information when the cursor cleaning condition is met and a preset cursor survival time length, the cursor survival time length being used to indicate a storable time length of the mapping relationship; obtaining a cursor key whose survival time value is within the cleaning time interval; deleting a mapping relationship between the cursor key and an end key within the cleaning time interval.

4. The data processing method of claim 1, wherein, The current survival time value is generated according to the first time indication parameter, comprising: determining a second time indication parameter corresponding to current time information; in a case where the first time indication parameter and the second time indication parameter are consistent, generating the current survival time value according to the current time information and the first time indication parameter; In a case where the first time indication parameter is inconsistent with the second time indication parameter, the current survival time value is generated according to the current time information, the cursor survival time length and the first time indication parameter.

5. The data processing method according to any one of claims 2 to 4, characterized in that, The first time indication parameter corresponding to the first time information is determined, including: The preset cursor survival time length is acquired, and the cursor survival time length is used for indicating the storable time length of the mapping relationship. The time parameter of the preset unit is acquired from the first time information according to the cursor survival time length and a preset data bit number, as the first time indication parameter.

6. The data processing method according to any one of claims 2 to 4, characterized in that, Before the acquiring of the unique cursor identifier corresponding to the Nth query, including: The unique cursor identifier is generated by an ID generator built in the distributed key-value database, and the cursor identifier is stored in the local memory.

7. A data processing apparatus, characterized by Including: The receiving module is configured to receive a data processing request sent by a client, the data processing request being used for indicating an Nth query, and the data processing request carrying a target cursor parameter; the target cursor parameter is an encoding parameter generated at an N-1th query and sent to the client after the N-1th query is completed, and N is any positive integer greater than 1; The query module is configured to query a target key corresponding to the target cursor parameter from a stored mapping relationship between a cursor parameter and a key, and the target key indicates an end position of the N-1th query in the distributed key-value database; the target cursor parameter is used as the key, and the target key is used as the value in the mapping relationship; The request processing module is configured to perform the Nth query on the distributed key-value database from the target key as a starting position; The target cursor parameter generation process includes: acquiring a cursor identifier corresponding to the N-1th query at the N-1th query; determining a first time indication parameter corresponding to first time information, the first time information indicating time information of performing the N-1th query; and generating the target cursor parameter according to the cursor identifier and the first time indication parameter; The query module is further configured to parse the first time indication parameter from the target cursor parameter; generate a current survival time value according to the first time indication parameter; splice the current survival time value and the target cursor parameter to obtain a cursor key corresponding to the target cursor parameter; and query the target key from a stored mapping relationship between the cursor key and an end key according to the cursor key corresponding to the target cursor parameter; The mapping relationship construction process between the cursor key and the end key includes: determining an end key indicated by an end position of the N-1th query after the N-1th query is completed; determining a survival time value corresponding to the first time indication parameter; splicing the survival time value and the target cursor parameter to obtain a cursor key, and establishing a mapping relationship between the cursor key and the end key.

8. An electronic device, comprising: Including: A processor; A memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The instructions in the computer readable storage medium, when executed by the processor of the electronic device, enable the electronic device to perform the data processing method according to any one of claims 1 to 6.

10. A computer program product, comprising instructions therein, characterized in that, The instructions, when executed by the processor of the electronic device, enable the electronic device to perform the data processing method according to any one of claims 1 to 6.

Images