Data processing method, device and equipment, computer readable storage medium and product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a computer readable storage medium and a product, and the method comprises the steps that search information input by a user is acquired, and the search information comprises identification information and a time range of the user; determining at least one data fragment matched with the search information in a key value database, storing index data of a plurality of users in the key value database, and performing fragment storage on the index data by taking a preset time interval as fragment granularity; and in response to the condition that the at least one data fragment meets a preset adjustment condition, adjusting the fragment granularity of the at least one data fragment, and obtaining the adjusted at least one data fragment, so that the associated parameters of the adjusted data fragments meet the preset condition. Therefore, the data fragments with small capacity can be automatically merged, the storage cost is reduced, the data fragments with large capacity are automatically split, and the problems that the data fragments with large capacity occupy large service bandwidth and are high in delay in the data reading and writing process are solved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, computer-readable storage medium, and product. Background Technology

[0002] A key-value (KV) database is a storage system based on key-value pairs, where data is stored in the form of key-value pairs. Each key is unique and corresponds to a specific value (data shard).

[0003] In related technologies, data shards corresponding to different key-value pairs in a KV database may have uneven storage sizes. Therefore, reading and writing data to large data shards often consumes a lot of service bandwidth, has high latency, and poor read / write performance. Conversely, reading and writing data to small data shards also generates a call traffic, increasing service costs. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, device, computer-readable storage medium, and product to address the technical problem of negative impacts on system capacity, data read / write costs, and performance caused by uneven data fragmentation density in KV databases.

[0005] In a first aspect, embodiments of this disclosure provide a data processing method, including:

[0006] Obtain search information input by the user, the search information including the user's identification information and time range;

[0007] In a preset key-value database, at least one data shard that matches the search information is determined. The key-value database stores index data for multiple users. The index data is sharded and stored with a preset time interval as the sharding granularity.

[0008] In response to the at least one data shard satisfying a preset adjustment condition, the sharding granularity of the at least one data shard is adjusted to obtain the at least one adjusted data shard, so that the association parameters of the adjusted data shards satisfy the preset condition.

[0009] In a second aspect, embodiments of this disclosure provide a data processing apparatus, comprising:

[0010] The acquisition module is used to acquire search information input by the user, the search information including the user's identification information and time range;

[0011] The determination module is used to determine at least one data shard that matches the search information in a preset key-value database. The key-value database stores index data of multiple users, and the index data is sharded and stored at a preset time interval as the sharding granularity.

[0012] An adjustment module is used to adjust the granularity of the at least one data shard in response to the at least one data shard meeting a preset adjustment condition, so as to obtain the at least one data shard after adjustment, so that the association parameters of the adjusted data shards meet the preset condition.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the data processing method described in the first aspect and various possible designs of the first aspect.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method described in the first aspect and various possible designs of the first aspect.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect and various possible designs of the first aspect.

[0018] The data processing method, apparatus, device, computer-readable storage medium, and product provided in this embodiment store index data of multiple users in a key-value database and shard the index data at a preset time interval as the sharding granularity. Therefore, the data capacity of different data shards varies. To avoid the negative impact of unevenly distributed data sharding on system capacity, cost, and performance, during data querying, if at least one data shard obtained based on search information is determined to meet preset adjustment conditions, the sharding granularity of that at least one data shard is adjusted so that the association parameters of each adjusted data shard meet the preset conditions. This allows for flexible adjustment of the sharding granularity of multiple data shards, enabling automatic merging of small-capacity data shards to reduce storage costs, and automatic splitting of large-capacity data shards to reduce the problems of high bandwidth consumption, high latency, and poor read / write performance associated with large-capacity data shards during data read / write processes. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic flowchart illustrating the data processing method provided in this embodiment of the disclosure;

[0021] Figure 2 A schematic flowchart of a data processing method provided in yet another embodiment of this disclosure;

[0022] Figure 3 This is a schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0023] Figure 4 A schematic flowchart of a data processing method provided in yet another embodiment of this disclosure;

[0024] Figure 5 This is a schematic diagram of a storage fragmentation state machine provided in an embodiment of this disclosure;

[0025] Figure 6 A schematic flowchart of a data processing method provided in yet another embodiment of this disclosure;

[0026] Figure 7 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this disclosure;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0030] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0031] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0032] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0033] To address the technical problem of the negative impact on system capacity, data read / write costs, and performance caused by uneven data fragmentation density in KV databases, this disclosure provides a data processing method, apparatus, device, computer-readable storage medium, and product.

[0034] It should be noted that the data processing methods, apparatus, devices, computer-readable storage media and products provided in this disclosure can be applied to any KV database sharding and balancing scenario.

[0035] Key-value (KV) databases are storage systems based on key-value pairs, where data is stored in key-value pairs. Each key is unique and corresponds to a specific value (data shard). KV databases excel in fast data storage and retrieval. Commonly used KV databases include Redis, Memcached, and LevelDB.

[0036] In data retrieval scenarios, to enable the retrieval of data within a specific time range, the index data associated with each user in a key-value database can be sharded and stored according to a preset time granularity. For example, the index data for a particular user can be sharded and stored on a daily basis. Therefore, each user can be associated with multiple data shards. However, since the index data varies from day to day, the capacity of the data shards also varies. Using this method for data storage may lead to technical problems such as high bandwidth consumption, high latency, poor performance, or increased service costs.

[0037] In solving the aforementioned technical problems, the inventors discovered through research that adjustment conditions can be preset. These adjustment conditions can be that the average capacity of at least one data fragment currently being searched is not within a preset capacity range. Therefore, during data reading, if it is determined that at least one data fragment being read currently meets the preset adjustment conditions, the granularity of at least one data fragment can be adjusted to obtain at least one adjusted data fragment, so that the correlation parameters of each adjusted data fragment meet the preset conditions.

[0038] Figure 1 This is a flowchart illustrating the data processing method provided in the embodiments of this disclosure, as shown below. Figure 1 As shown, the method includes:

[0039] Step 101: Obtain the search information input by the user, including the user's identification information and time range.

[0040] Step 102: Determine at least one data shard that matches the search information in a preset key-value database. The key-value database stores index data for multiple users. The index data is sharded and stored at a preset time interval as the sharding granularity.

[0041] Step 103: In response to the at least one data shard satisfying the preset adjustment conditions, adjust the sharding granularity of the at least one data shard to obtain the at least one adjusted data shard, so that the association parameters of the adjusted data shards satisfy the preset conditions.

[0042] In this embodiment, the execution entity is a data processing device. This data processing device can be coupled to a key-value database. Therefore, the granularity of the data sharding can be flexibly adjusted when the user performs data read and write operations.

[0043] In this embodiment, the KV database stores index data for multiple users, and the index data is sharded and stored with a preset time interval as the sharding granularity. For example, since the KV database stores index data for multiple users, the index data can be stored with a granularity of natural days. Therefore, the index data associated with user A within the past 180 days corresponds to 180 data shards. The key value associated with each data shard can be user A's identification information and timestamp information.

[0044] Optionally, users can perform data retrieval operations in the KV database according to actual needs. In response to a user-triggered data retrieval operation, the search information input by the user can be obtained, including the user's identification information and a time range. At least one data shard matching the search information can be determined in a preset key-value database. For example, if it is necessary to read the index data of user A for 30-40 days, then 10 data shards with timestamps within the 30-40 day range can be obtained from the 180 data shards associated with user A.

[0045] Furthermore, since index data in a key-value database is stored in shards based on time range, the capacity of different shards may vary. Inconsistent shard sizes can lead to higher bandwidth consumption, higher latency, poorer performance, or increased service costs.

[0046] Therefore, adjustment conditions can be preset. After acquiring at least one data shard, it can be determined whether the at least one data shard meets the preset adjustment conditions. If it does, the sharding granularity of the at least one data shard can be adjusted to obtain an adjusted at least one data shard, so that the association parameters of the adjusted data shards meet the preset conditions. Here, the association parameter can be the capacity associated with the data shard. The preset condition can be that the average capacity corresponding to at least one data shard falls within a preset capacity range. Alternatively, the preset condition can be that the capacity of each data shard falls within a preset capacity range.

[0047] As one feasible approach, the data processing device can also determine at preset time intervals whether the granularity of the currently stored data shards in the KV database needs to be adjusted. For example, it can determine daily whether the average capacity of the currently stored data shards in the KV database falls within a preset capacity range; if it does not, an adjustment operation for the granularity of the shards is triggered. Alternatively, it can determine at preset time intervals whether the currently stored data shards in the KV database include any data shard whose capacity falls outside the capacity range, and perform splitting or merging operations on that data shard to ensure that the capacity of the adjusted data shard falls within the capacity range.

[0048] Users can adjust this time interval according to their actual needs, so that the cycle of score granularity adjustment operation is more in line with the user's actual needs.

[0049] The data processing method provided in this embodiment stores index data for multiple users in a key-value database and segments the index data into fragments with a preset time interval as the fragment granularity. Therefore, different data fragments have different data capacities. To avoid the negative impact of unevenly distributed data fragments on system capacity, cost, and performance, during data querying, if at least one data fragment obtained based on search information is determined to meet preset adjustment conditions, the fragment granularity of that at least one data fragment is adjusted so that the association parameters of each adjusted data fragment meet the preset conditions. This allows for flexible adjustment of the fragment granularity of multiple data fragments, enabling automatic merging of small-capacity data fragments to reduce storage costs, and automatic splitting of large-capacity data fragments to mitigate the problems of high bandwidth consumption, high latency, and poor read / write performance associated with large-capacity data fragments during data read / write processes.

[0050] Figure 2 This is a flowchart illustrating a data processing method provided in yet another embodiment of the present disclosure. Based on any of the above embodiments, as follows... Figure 2 As shown, step 103 includes:

[0051] Step 201: In response to the at least one data shard meeting the preset adjustment conditions, trigger a shard adjustment task and send the shard adjustment task to a preset message queue.

[0052] Step 202: Control the preset sharding service module to asynchronously obtain sharding adjustment tasks from the message queue and adjust the sharding granularity of the at least one data shard.

[0053] In this embodiment, to improve the efficiency of data sharding adjustment, a message queue can be pre-set. Therefore, after triggering a sharding adjustment task, the sharding adjustment task can be placed in the message queue so that the preset sharding service module can asynchronously read each sharding adjustment task and realize the sharding adjustment operation.

[0054] Optionally, when at least one data shard meets preset adjustment conditions, a shard adjustment task can be triggered and sent to a preset message queue. Additionally, a sharding service module can be pre-configured. This module can perform splitting or merging operations on data shards based on their actual capacity to adjust the current sharding granularity. This sharding service module can asynchronously retrieve shard adjustment tasks from the message queue and adjust the sharding granularity of at least one data shard.

[0055] Figure 3 This is a schematic diagram illustrating an application scenario provided by an embodiment of this disclosure, such as... Figure 3As shown, during data read operation 31, if at least one data shard read from the KV database 32 is detected to meet preset adjustment conditions, a shard adjustment task can be triggered. This shard adjustment task is sent to a preset general load balancing queue so that a preset load balancing task distribution service can distribute it to a message queue 33. The preset sharding service module 34 can then asynchronously retrieve the shard adjustment task from the message queue 33 and adjust the sharding granularity of the at least one data shard. The data generated during the adjustment process can be stored in a preset Hive data warehouse.

[0056] The data processing method provided in this embodiment sets up a message queue. After triggering the sharding adjustment task, the sharding adjustment task can be sent to the message queue, thus enabling asynchronous sharding balancing operations and improving the efficiency of data sharding adjustment.

[0057] Figure 4 This is a flowchart illustrating a data processing method provided in yet another embodiment of the present disclosure. Based on any of the above embodiments, as follows... Figure 4 As shown, step 103 includes:

[0058] Step 401: Determine the fragment capacity corresponding to each data fragment.

[0059] Step 402: Calculate the average capacity corresponding to the at least one data shard.

[0060] Step 403: In response to the average capacity not being within the preset capacity range, the granularity of the at least one data shard is adjusted.

[0061] In this embodiment, after obtaining at least one data shard based on search information, the shard capacity corresponding to each data shard can be determined, and the average capacity corresponding to at least one data shard can be calculated.

[0062] Optionally, a capacity range can be preset. When the capacity of a data shard falls within this range, the data read / write process is most efficient, and service bandwidth and read / write performance are saved. Therefore, in response to the average capacity not falling within the preset capacity range, the sharding granularity of the data shards can be adjusted based on this capacity range so that the adjusted capacity of the data shards falls within that range.

[0063] The data processing method provided in this embodiment, by pre-setting a capacity range, can accurately determine whether a granularity adjustment operation is needed based on the capacity range and the average capacity corresponding to at least one data shard. Furthermore, it can accurately determine the current adjustment method.

[0064] Furthermore, based on any of the above embodiments, step 403 includes:

[0065] If the average capacity exceeds the capacity range, then the at least one data shard is split to reduce the shard granularity corresponding to the at least one data shard.

[0066] If the average capacity is lower than the capacity range, then a merging operation is performed on the at least one data shard to improve the shard granularity corresponding to the at least one data shard.

[0067] In this embodiment, a capacity range can be preset. This range can be set by the user according to actual needs or it can be a preset empirical value. When the capacity of the data shard falls within this capacity range, the data read and write process is most efficient, and service bandwidth and read / write performance are saved.

[0068] Furthermore, after determining the average capacity corresponding to at least one data shard, the sharding granularity of the data shards can be adjusted based on this capacity range so that the adjusted capacity of the data shards falls within that range. Therefore, if the average capacity exceeds the capacity range, a split operation is performed on at least one data shard to reduce the sharding granularity corresponding to at least one data shard. If the average capacity is below the capacity range, a merge operation is performed on at least one data shard to increase the sharding granularity corresponding to at least one data shard.

[0069] The data processing method provided in this embodiment performs a splitting operation on at least one data fragment when the average capacity exceeds the capacity range, thereby reducing the fragment granularity corresponding to at least one data fragment. When the average capacity is below the capacity range, a merging operation is performed on at least one data fragment, thereby increasing the fragment granularity corresponding to at least one data fragment. This ensures that the capacity of the adjusted data fragments falls within the capacity range, avoiding technical problems such as large service bandwidth consumption, high latency, poor performance, or increased service costs caused by inconsistent fragment sizes.

[0070] Furthermore, based on any of the above embodiments, the splitting operation on the at least one data shard includes:

[0071] For each data shard, the data shard is split into a preset number of sub-shards to obtain at least one adjusted data shard, so that the association parameters of each adjusted data shard meet preset conditions.

[0072] Alternatively, the splitting operation on the at least one data fragment includes:

[0073] In the at least one data shard, a target shard with a capacity exceeding the capacity range is identified.

[0074] The target fragment is split according to the stated capacity range, so that the capacity of the split target fragment is within the stated capacity range.

[0075] In this embodiment, when the average capacity of at least one data shard exceeds the capacity range, a split operation is performed on at least one data shard.

[0076] Optionally, at least one data shard may correspond to the same capacity, so a preset number can be set in advance, which can be set by the user according to actual needs. Therefore, for each data shard, the data shard can be split into a preset number of sub-shards to obtain at least one adjusted data shard, so that the association parameters of each adjusted data shard meet preset conditions.

[0077] As one possible approach, since at least one data shard may correspond to different capacity sizes, the splitting operation can be performed only on data shards whose capacity exceeds the capacity range. Therefore, a target shard with a capacity exceeding the capacity range can be identified from at least one data shard. The target shard is then split according to the capacity range so that the capacity of the split target shard is within the capacity range.

[0078] The data processing method provided in this embodiment enables rapid data fragmentation by setting a preset quantity, ensuring that the capacity of the fragmented data is within the specified range. Alternatively, by splitting a target fragment whose capacity exceeds the specified range according to the specified range, the capacity of the split target fragment can be kept within the specified range. This allows for flexible determination of the fragmentation adjustment method based on the actual scenario, improving the applicability of the fragmentation adjustment operation.

[0079] Furthermore, based on any of the above embodiments, the merging operation on the at least one data fragment includes:

[0080] The preset number of data fragments are merged into a single adjusted data fragment.

[0081] Alternatively, the splitting operation on the at least one data fragment includes:

[0082] In the at least one data shard, a target shard with a capacity lower than the capacity range is determined.

[0083] The target fragments are merged according to the stated capacity range, so that the capacity of the merged target fragments is within the stated capacity range.

[0084] In this embodiment, when the average capacity of at least one data shard is less than a preset capacity range, at least one data shard can be merged.

[0085] Optionally, a preset number can be set, which can be configured by the user according to actual needs. Therefore, a preset number of data shards can be merged into an adjusted data shard, ensuring that the capacity of the merged target shard is within the specified capacity range.

[0086] As one feasible approach, for at least one data shard, a merging operation can be performed on data shards whose capacity is less than the capacity range. Therefore, a target shard with a capacity less than the capacity range can be identified among at least one data shard. The target shard is then merged according to the capacity range so that the capacity of the merged target shard is within the capacity range.

[0087] The data processing method provided in this embodiment enables the rapid merging of data fragments based on a preset quantity, ensuring that the capacity of the merged data fragments remains within the specified capacity range. Alternatively, by merging target fragments whose capacity exceeds the capacity range according to the specified capacity range, the capacity of the merged target fragments can be brought within the specified capacity range. This allows for flexible determination of the fragment adjustment method based on the actual scenario, improving the applicability of the fragment adjustment operation.

[0088] Furthermore, based on any of the above embodiments, after adjusting the granularity of the at least one data shard to obtain the adjusted at least one data shard, the method further includes:

[0089] Based on the adjustment operation for the at least one data shard, the metadata in the preset shard is updated. The metadata includes the current adjustment status, shard granularity, and adjustment time of each data shard. The adjustment status includes normal status, split status, and merge status.

[0090] In this embodiment, for each key-value data pair, preset metadata can be stored in a preset shard in the KV database. This metadata may include the current adjustment status of the key-value data pair, the current shard granularity, and the adjustment time. The adjustment status includes normal status, split status, and merge status.

[0091] Furthermore, when adjusting the granularity of data shards, the metadata in the preset shards can be updated based on the adjustment operation for at least one data shard, so as to update the latest state of the shard data written in the metadata.

[0092] Figure 5 This is a schematic diagram of a storage fragmentation state machine provided in an embodiment of this disclosure, as shown below. Figure 5As shown, the adjustment states can include merge state 51, normal state 52, and split state 53. In merge state 51, status = 2; in normal state 52, status = 1; and in split state 53, status = 0.

[0093] The data processing method provided in this embodiment can accurately record the current sharding status by updating the preset metadata when adjusting the sharding granularity of the data shards, and can accurately perform subsequent data reading operations by recording the adjusted sharding granularity.

[0094] Furthermore, based on any of the above embodiments, after updating the metadata in the preset fragment, the method further includes:

[0095] In response to the content search operation triggered by the user, metadata is obtained from the preset segment.

[0096] Based on the sharding granularity in the metadata and the search information associated with the content search operation, the target data shard that matches the search information is searched in the key-value database.

[0097] In this embodiment, after completing the granularity adjustment operation for at least one data shard, the user can still perform subsequent data read and write operations.

[0098] Optionally, users can trigger a content search operation based on actual needs. In order to accurately determine the data shards to be read based on the content search operation, metadata can be obtained from the preset shards, and data reading operations can be performed based on the current shard granularity recorded in the metadata.

[0099] Furthermore, based on the sharding granularity in the metadata and the search information associated with the content search operation, the target data shards that match the search information can be searched in the key-value database.

[0100] For example, before adjusting the sharding granularity, data could be stored at the calendar day level. Therefore, the index data associated with user A for the past 180 days would correspond to 180 data shards. If the user needed to read the index data for days 30-40, they would need to obtain 10 data shards. After adjusting the sharding granularity, the sharding granularity can be adjusted so that index data for every two days is stored in one data shard. Therefore, if the user needs to read the index data for days 30-40, they would need to obtain 5 data shards.

[0101] The data processing method provided in this embodiment records the adjusted sharding granularity in the metadata, so that when the user triggers a data read operation again, the target data shard that matches the search information can be accurately determined based on the adjusted sharding granularity, thus avoiding the impact of changes in sharding granularity on subsequent data read and write operations.

[0102] Figure 6 This is a flowchart illustrating a data processing method provided in yet another embodiment of the present disclosure. Based on any of the above embodiments, as follows... Figure 6 As shown, the method further includes:

[0103] Step 601: Determine at least one adjusted shard in the key-value database that is currently in a split or merge state based on the metadata.

[0104] Step 602: Determine the adjustment duration corresponding to each adjustment segment based on the adjustment time and the current time.

[0105] Step 603: If the adjustment duration exceeds the preset duration threshold, then the adjustment segment is readjusted.

[0106] In this embodiment, when adjusting the granularity of data shards, the adjustment status can be recorded in the metadata of the preset shard. This metadata allows the adjustment duration for each data shard to be determined. For data shards with excessively long adjustment durations, to avoid errors during the adjustment process that could affect subsequent data read and write operations, they can be set to an expired state, allowing for subsequent readjustment of expired adjustment tasks.

[0107] Optionally, at least one adjustment shard in the key-value database currently in a split or merge state can be determined based on metadata. The adjustment duration for each adjustment shard is determined based on the adjustment time and the current time. For example, the difference between the timestamp of the current time and the timestamp of the adjustment time can be determined as the current adjustment duration. Furthermore, a duration threshold can be preset. This duration threshold can be a value set by the user according to actual needs, or it can be a preset empirical value; this disclosure does not impose any restrictions on this. If the adjustment duration exceeds the preset duration threshold, the adjustment operation for the adjustment shard is repeated.

[0108] The data processing method provided in this embodiment records the adjustment status of the shard adjustment operation in the metadata, so that when the adjustment time exceeds the time threshold, the adjustment shard can be readjusted to avoid failures during the long adjustment operation.

[0109] Figure 7 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this disclosure, such as... Figure 7As shown, the device includes: an acquisition module 71, a determination module 72, and an adjustment module 73. The acquisition module 71 is used to acquire search information input by the user, the search information including the user's identification information and a time range. The determination module 72 is used to determine at least one data fragment matching the search information in a preset key-value database. The key-value database stores index data for multiple users, and the index data is fragmented and stored at a preset time interval as the fragment granularity. The adjustment module 73 is used to adjust the fragment granularity of the at least one data fragment in response to the at least one data fragment meeting preset adjustment conditions, obtaining an adjusted at least one data fragment, so that the association parameters of the adjusted data fragments meet the preset conditions.

[0110] Furthermore, based on any of the above embodiments, the adjustment module is configured to: trigger a shard adjustment task in response to the at least one data shard meeting a preset adjustment condition, and send the shard adjustment task to a preset message queue; control a preset shard service module to asynchronously retrieve the shard adjustment task from the message queue and adjust the sharding granularity of the at least one data shard.

[0111] Further, based on any of the above embodiments, the adjustment module is configured to: determine the shard capacity corresponding to each data shard; calculate the average capacity corresponding to the at least one data shard; and adjust the shard granularity of the at least one data shard in response to the average capacity not being within a preset capacity range.

[0112] Furthermore, based on any of the above embodiments, the adjustment module is configured to: if the average capacity exceeds the capacity range, perform a splitting operation on the at least one data shard to reduce the shard granularity corresponding to the at least one data shard; if the average capacity is lower than the capacity range, perform a merging operation on the at least one data shard to increase the shard granularity corresponding to the at least one data shard.

[0113] Further, based on any of the above embodiments, the adjustment module is configured to: split each data shard into a preset number of sub-shards to obtain at least one adjusted data shard, so that the association parameters of each adjusted data shard meet preset conditions. Alternatively, the adjustment module is configured to: determine a target shard among the at least one data shards whose capacity exceeds the capacity range; and perform a splitting operation on the target shard according to the capacity range, so that the capacity of the split target shard is within the capacity range.

[0114] Further, based on any of the above embodiments, the adjustment module is configured to: merge a preset number of data shards into an adjusted data shard. Alternatively, the adjustment module is configured to: determine a target shard with a capacity lower than the capacity range among the at least one data shard; and perform a merging operation on the target shard according to the capacity range, so that the capacity of the merged target shard is within the capacity range.

[0115] Furthermore, based on any of the above embodiments, the apparatus further includes: an adjustment module, configured to update metadata in a preset shard based on the adjustment operation for the at least one data shard, wherein the metadata includes the current adjustment status, shard granularity, and adjustment time of each data shard, and the adjustment status includes a normal state, a split state, and a merge state.

[0116] Furthermore, based on any of the above embodiments, the apparatus further includes: an acquisition module, configured to acquire metadata from the preset shards in response to a content search operation triggered by the user; and a search module, configured to search the key-value database for a target data shard that matches the search information based on the shard granularity in the metadata and the search information associated with the content search operation.

[0117] Furthermore, based on any of the above embodiments, the apparatus further includes: a determining module, configured to determine at least one adjustment shard in the key-value database currently in a split or merged state based on the metadata; a processing module, configured to determine the adjustment duration corresponding to each adjustment shard based on the adjustment time and the current time; and an adjustment module, configured to re-adjust the adjustment shard if the adjustment duration exceeds a preset duration threshold.

[0118] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0119] To implement the above embodiments, this disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method as described in any of the above embodiments.

[0120] To implement the above embodiments, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method as described in any of the above embodiments.

[0121] To implement the above embodiments, this disclosure also provides an electronic device, including: a processor and a memory;

[0122] The memory stores computer-executed instructions;

[0123] The processor executes computer execution instructions stored in the memory, causing the processor to perform the data processing method as described in any of the above embodiments.

[0124] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 800 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0125] like Figure 8 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0126] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0127] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0128] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0129] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0130] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0131] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0134] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] In a first aspect, according to one or more embodiments of the present disclosure, a data processing method is provided, comprising:

[0137] Obtain search information input by the user, the search information including the user's identification information and time range;

[0138] In a preset key-value database, at least one data shard that matches the search information is determined. The key-value database stores index data for multiple users. The index data is sharded and stored with a preset time interval as the sharding granularity.

[0139] In response to the at least one data shard satisfying a preset adjustment condition, the sharding granularity of the at least one data shard is adjusted to obtain the at least one adjusted data shard, so that the association parameters of the adjusted data shards satisfy the preset condition.

[0140] According to one or more embodiments of this disclosure, adjusting the granularity of the at least one data shard in response to the at least one data shard satisfying a preset adjustment condition includes:

[0141] In response to the fact that at least one data shard meets the preset adjustment conditions, a shard adjustment task is triggered and the shard adjustment task is sent to a preset message queue;

[0142] The preset sharding service module asynchronously retrieves sharding adjustment tasks from the message queue and adjusts the sharding granularity of the at least one data shard.

[0143] According to one or more embodiments of this disclosure, adjusting the granularity of the at least one data shard in response to the at least one data shard satisfying a preset adjustment condition includes:

[0144] Determine the fragment capacity corresponding to each data fragment;

[0145] Calculate the average capacity corresponding to the at least one data shard;

[0146] If the average capacity is not within the preset capacity range, the granularity of the at least one data shard is adjusted.

[0147] According to one or more embodiments of this disclosure, the adjustment of the granularity of the at least one data shard in response to the average capacity not being within a preset capacity range includes:

[0148] If the average capacity exceeds the capacity range, then the at least one data shard is split to reduce the shard granularity corresponding to the at least one data shard.

[0149] If the average capacity is lower than the capacity range, then a merging operation is performed on the at least one data shard to improve the shard granularity corresponding to the at least one data shard.

[0150] According to one or more embodiments of this disclosure, the splitting operation on the at least one data fragment includes:

[0151] For each data shard, the data shard is split into a preset number of sub-shards to obtain at least one adjusted data shard, so that the capacity information associated with the at least one adjusted data shard is within a preset capacity range.

[0152] Alternatively, the splitting operation on the at least one data fragment includes:

[0153] In the at least one data shard, a target shard with a capacity exceeding the capacity range is identified;

[0154] The target fragment is split according to the stated capacity range, so that the capacity of the split target fragment is within the stated capacity range.

[0155] According to one or more embodiments of this disclosure, the merging operation on the at least one data fragment includes:

[0156] Merge a preset number of data fragments into a single adjusted data fragment;

[0157] Alternatively, the splitting operation on the at least one data fragment includes:

[0158] In the at least one data shard, a target shard with a capacity lower than the capacity range is identified;

[0159] The target fragments are merged according to the stated capacity range, so that the capacity of the merged target fragments is within the stated capacity range.

[0160] According to one or more embodiments of this disclosure, after adjusting the granularity of the at least one data shard to obtain the adjusted at least one data shard, the method further includes:

[0161] Based on the adjustment operation for the at least one data shard, the metadata in the preset shard is updated. The metadata includes the current adjustment status, shard granularity, and adjustment time of each data shard. The adjustment status includes normal status, split status, and merge status.

[0162] According to one or more embodiments of this disclosure, after updating the metadata in the preset fragment, the method further includes:

[0163] In response to the content search operation triggered by the user, metadata is obtained from the preset segment;

[0164] Based on the sharding granularity in the metadata and the search information associated with the content search operation, the target data shard that matches the search information is searched in the key-value database.

[0165] According to one or more embodiments of this disclosure, the method further includes:

[0166] Based on the metadata, determine at least one adjusted shard in the key-value database that is currently in a split or merge state;

[0167] The adjustment duration for each adjustment segment is determined based on the adjustment time and the current time.

[0168] If the adjustment duration exceeds the preset duration threshold, the adjustment segment will be readjusted.

[0169] Secondly, according to one or more embodiments of the present disclosure, a data processing apparatus is provided, comprising:

[0170] The acquisition module is used to acquire search information input by the user, the search information including the user's identification information and time range;

[0171] The determination module is used to determine at least one data shard that matches the search information in a preset key-value database. The key-value database stores index data of multiple users, and the index data is sharded and stored at a preset time interval as the sharding granularity.

[0172] An adjustment module is used to adjust the granularity of the at least one data shard in response to the at least one data shard meeting a preset adjustment condition, so as to obtain the at least one data shard after adjustment, so that the association parameters of the adjusted data shards meet the preset condition.

[0173] According to one or more embodiments of this disclosure, the adjustment module is configured to:

[0174] In response to the fact that at least one data shard meets the preset adjustment conditions, a shard adjustment task is triggered and the shard adjustment task is sent to a preset message queue;

[0175] The preset sharding service module asynchronously retrieves sharding adjustment tasks from the message queue and adjusts the sharding granularity of the at least one data shard.

[0176] According to one or more embodiments of this disclosure, the adjustment module is configured to:

[0177] Determine the fragment capacity corresponding to each data fragment;

[0178] Calculate the average capacity corresponding to the at least one data shard;

[0179] If the average capacity is not within the preset capacity range, the granularity of the at least one data shard is adjusted.

[0180] According to one or more embodiments of this disclosure, the adjustment module is configured to:

[0181] If the average capacity exceeds the capacity range, then the at least one data shard is split to reduce the shard granularity corresponding to the at least one data shard.

[0182] If the average capacity is lower than the capacity range, then a merging operation is performed on the at least one data shard to improve the shard granularity corresponding to the at least one data shard.

[0183] According to one or more embodiments of this disclosure, the adjustment module is configured to:

[0184] For each data shard, the data shard is split into a preset number of sub-shards to obtain at least one adjusted data shard, so that the capacity information associated with the at least one adjusted data shard is within a preset capacity range.

[0185] Alternatively, the adjustment module is used to:

[0186] In the at least one data shard, a target shard with a capacity exceeding the capacity range is identified;

[0187] The target fragment is split according to the stated capacity range, so that the capacity of the split target fragment is within the stated capacity range.

[0188] According to one or more embodiments of this disclosure, the adjustment module is configured to:

[0189] Merge a preset number of data fragments into a single adjusted data fragment;

[0190] Alternatively, the adjustment module is used to:

[0191] In the at least one data shard, a target shard with a capacity lower than the capacity range is identified;

[0192] The target fragments are merged according to the stated capacity range, so that the capacity of the merged target fragments is within the stated capacity range.

[0193] According to one or more embodiments of this disclosure, the apparatus further includes:

[0194] The adjustment module is used to update the metadata in the preset shards based on the adjustment operation for the at least one data shard. The metadata includes the current adjustment status, shard granularity, and adjustment time of each data shard. The adjustment status includes normal status, split status, and merge status.

[0195] According to one or more embodiments of this disclosure, the apparatus further includes:

[0196] The acquisition module is used to acquire metadata from the preset segment in response to the content search operation triggered by the user;

[0197] The search module is used to search the key-value database for target data fragments that match the search information based on the fragment granularity in the metadata and the search information associated with the content search operation.

[0198] According to one or more embodiments of this disclosure, the apparatus further includes:

[0199] The determination module is used to determine, based on the metadata, at least one adjustment shard in the key-value database that is currently in a split or merge state;

[0200] The processing module is used to determine the adjustment duration corresponding to each adjustment segment based on the adjustment time and the current time;

[0201] The adjustment module is used to readjust the adjustment segment if the adjustment duration exceeds a preset duration threshold.

[0202] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0203] The memory stores computer-executed instructions;

[0204] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the data processing method as described in the first aspect and various possible designs of the first aspect.

[0205] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the data processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0206] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect and various possible designs of the first aspect.

[0207] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0208] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0209] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A data processing method, characterized in that, include: Obtain search information input by the user, the search information including the user's identification information and time range; In a preset key-value database, at least one data shard that matches the search information is determined. The key-value database stores index data for multiple users. The index data is sharded and stored with a preset time interval as the sharding granularity. In response to the at least one data shard satisfying a preset adjustment condition, the sharding granularity of the at least one data shard is adjusted to obtain the at least one adjusted data shard, so that the association parameters of the adjusted data shards satisfy the preset condition.

2. The method according to claim 1, characterized in that, The step of adjusting the granularity of the at least one data fragment in response to the at least one data fragment meeting a preset adjustment condition includes: In response to the fact that at least one data shard meets the preset adjustment conditions, a shard adjustment task is triggered and the shard adjustment task is sent to a preset message queue; The preset sharding service module asynchronously retrieves sharding adjustment tasks from the message queue and adjusts the sharding granularity of the at least one data shard.

3. The method according to claim 1, characterized in that, The step of adjusting the granularity of the at least one data fragment in response to the at least one data fragment meeting a preset adjustment condition includes: Determine the fragment capacity corresponding to each data fragment; Calculate the average capacity corresponding to the at least one data shard; If the average capacity is not within the preset capacity range, the granularity of the at least one data shard is adjusted.

4. The method according to claim 3, characterized in that, In response to the average capacity not falling within a preset capacity range, the granularity of the at least one data shard is adjusted, including: If the average capacity exceeds the capacity range, then the at least one data shard is split to reduce the shard granularity corresponding to the at least one data shard. If the average capacity is lower than the capacity range, then a merging operation is performed on the at least one data shard to improve the shard granularity corresponding to the at least one data shard.

5. The method according to claim 4, characterized in that, The splitting operation on the at least one data fragment includes: For each data shard, the data shard is split into a preset number of sub-shards to obtain at least one adjusted data shard, so that the capacity information associated with the at least one adjusted data shard is within a preset capacity range. Alternatively, the splitting operation on the at least one data fragment includes: In the at least one data shard, a target shard with a capacity exceeding the capacity range is identified; The target fragment is split according to the stated capacity range, so that the capacity of the split target fragment is within the stated capacity range.

6. The method according to claim 4, characterized in that, The merging operation on the at least one data fragment includes: Merge a preset number of data fragments into a single adjusted data fragment; Alternatively, the splitting operation on the at least one data fragment includes: In the at least one data shard, a target shard with a capacity lower than the capacity range is identified; The target fragments are merged according to the stated capacity range, so that the capacity of the merged target fragments is within the stated capacity range.

7. The method according to any one of claims 1-6, characterized in that, After adjusting the granularity of the at least one data shard to obtain the adjusted at least one data shard, the method further includes: Based on the adjustment operation for the at least one data shard, the metadata in the preset shard is updated. The metadata includes the current adjustment status, shard granularity, and adjustment time of each data shard. The adjustment status includes normal status, split status, and merge status.

8. The method according to claim 7, characterized in that, After updating the metadata in the preset shards, the process further includes: In response to the content search operation triggered by the user, metadata is obtained from the preset segment; Based on the sharding granularity in the metadata and the search information associated with the content search operation, the target data shard that matches the search information is searched in the key-value database.

9. The method according to claim 7, characterized in that, The method further includes: Based on the metadata, determine at least one adjusted shard in the key-value database that is currently in a split or merge state; The adjustment duration for each adjustment segment is determined based on the adjustment time and the current time. If the adjustment duration exceeds the preset duration threshold, the adjustment segment will be readjusted.

10. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire search information input by the user, the search information including the user's identification information and time range; The determination module is used to determine at least one data shard that matches the search information in a preset key-value database. The key-value database stores index data of multiple users, and the index data is sharded and stored at a preset time interval as the sharding granularity. An adjustment module is used to adjust the granularity of the at least one data shard in response to the at least one data shard meeting a preset adjustment condition, so as to obtain the at least one data shard after adjustment, so that the association parameters of the adjusted data shards meet the preset condition.

11. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the data processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 9.

Images