A method and apparatus for data storage, reading and deletion

Abstract

The application discloses a data storage, reading and deleting method and device, and relates to the technical field of large-scale data management. A specific implementation of the method comprises the following steps: in response to a data storage request, obtaining current segment number allocation information of to-be-stored data, and listening to an update state of segment meta information corresponding to the to-be-stored data; according to a preset segmenting rule and the current segment number allocation information, performing segmenting processing on the to-be-stored data, storing the data after the segmenting processing, and determining update information of the segment meta information; and in the case that the update state is not updated, updating the segment meta information according to the update information. The implementation effectively solves the problems of storage, reading and deletion of large data in Redis cache, and through the listening to the update state of the segment meta information, the accuracy of data storage and reading is ensured.

Description

Technical Field

[0001] This invention relates to the field of large-scale data management technology, and in particular to a method and apparatus for data storage, retrieval and deletion. Background Technology

[0002] Redis, or Remote Dictionary Service, is a commonly used data caching solution for high-concurrency scenarios due to its fast read / write speeds, support for data persistence, and support for various data types. However, Redis's high-performance operations are typically only suitable for small datasets. For large datasets, such as those exceeding 5MB, read / write operations often encounter anomalies. Current common solutions include adjusting the system's time thresholds on the Redis server and adding retry logic to accommodate large datasets, or storing large datasets separately on another system.

[0003] In the process of realizing this invention, the inventors discovered the following problems in the prior art:

[0004] While adjusting system parameters and retry logic to adapt to the data can solve a small portion of data read / write problems, it cannot fundamentally solve the problem of reading / writing large amounts of data and will also affect the normal business operations of other data on the Redis server. On the other hand, the method of storing data separately not only directly affects the read speed but also consumes more hardware and software resources, which cannot meet the needs of practical applications. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method and apparatus for data storage, retrieval, and deletion. Based on the obtained current segment number allocation information and segmentation rules of the data to be stored, the data is segmented and stored. If the segment metadata status is not updated, the segment metadata is updated. Based on the already stored data, data retrieval and deletion operations are performed according to the segment metadata. The method of the present invention effectively solves the problem of storing, retrieving, and deleting large volumes of data in Redis cache, and ensures the accuracy of data storage and retrieval by monitoring the update status of segment metadata.

[0006] To achieve the aforementioned objective, according to a first aspect of the present invention, a data storage method is provided, comprising:

[0007] In response to a data storage request, the system obtains the current segment number allocation information of the data to be stored and monitors the update status of the segment metadata corresponding to the data to be stored.

[0008] According to the preset segmentation rules and the current segment number allocation information, the data to be stored is segmented, the segmented data is stored, and the update information of the segmented metadata is determined.

[0009] If the update status is not updated, the segment metadata is updated according to the update information.

[0010] Optionally, before obtaining the current segment number allocation information of the data to be stored, the method includes: determining that the amount of data to be stored exceeds a preset processing threshold.

[0011] Optionally, the method includes: before segmenting the data to be stored, obtaining segmentation metadata corresponding to the data to be stored, storing the segmentation metadata in a local cache, and determining the data to be stored based on the segmentation metadata; after updating the segmentation metadata based on the update information, obtaining the segmentation metadata from the local cache, and deleting the historical stored data corresponding to the segmentation metadata based on the segmentation metadata.

[0012] Optionally, the data to be stored is segmented according to a preset segmentation rule and the current segment number allocation information, including: splitting the data to be stored into multiple segments according to a segmentation threshold in the preset segmentation rule; determining the current segment start number of the data to be stored according to the prefix information and number information in the current segment number allocation information; allocating a corresponding segment data index to each segment data according to the current segment start number and the number of segments; and determining the update information of the segment metadata, including: using the current segment start number and the number of segments as the update information of the segment metadata.

[0013] Optionally, storing the segmented data includes: when the number of segmented data exceeds a preset threshold, determining the corresponding cache slot for each segmented data according to the segmented data index, and storing the segmented data in the same cache slot in batches using multiple threads.

[0014] Optionally, after storing the segmented data, the method further includes: if the update status is updated, deleting the segmented data, re-acquiring the current segment number allocation information of the data to be stored, monitoring the update status of the segmented metadata, and segmenting the data to be stored according to the re-acquired current segment number allocation information and the preset segmentation rules.

[0015] According to a second aspect of the present invention, a method for reading data is provided, comprising:

[0016] In response to a data read request, segmented metadata of the data to be read is obtained; the data to be read is stored using any data storage method of the first aspect of the present invention.

[0017] Based on the number of segments and the starting segment number in the segmented metadata, determine the segmented data index corresponding to the data to be read;

[0018] Based on the segmented data index corresponding to the data to be read, read the segmented data corresponding to the data to be read; assemble the segmented data corresponding to the data to be read to obtain the data to be read.

[0019] Optionally, the method further includes: if reading the segmented data corresponding to the data to be read fails, re-acquiring the segmented metadata of the data to be read, determining the segmented data index corresponding to the data to be read based on the number of segments and the starting segment number in the re-read segmented metadata, and reading the segmented data.

[0020] Optionally, reading the segmented data corresponding to the data to be read according to the segmented data index includes: when the number of segments corresponding to the data to be read exceeds a preset number threshold, determining the cache slot of the corresponding segmented data according to each segmented data index of the data to be read, and batch reading the segmented data in the same cache slot in multiple threads.

[0021] According to a third aspect of the present invention, a method for data deletion is provided, comprising:

[0022] In response to a data deletion request, segment metadata corresponding to the data to be deleted is obtained; the data to be deleted is stored using any data storage method of the first aspect of the present invention.

[0023] Based on the number of segments and the starting segment number in the segmented metadata, determine the segmented data index corresponding to the data to be deleted;

[0024] Delete the data to be deleted based on the segmented data index corresponding to the data to be deleted.

[0025] Optionally, deleting the data to be deleted according to the segment data index corresponding to the data to be deleted includes: when the number of segments corresponding to the data to be deleted exceeds a preset number threshold, determining the cache slot of the corresponding segment data according to each segment data index corresponding to the data to be read, and deleting the segment data in the same cache slot in batches using multiple threads.

[0026] According to a fourth aspect of the present invention, a data storage apparatus is provided, comprising:

[0027] The first metadata acquisition module is used to respond to data storage requests, acquire the current segment number allocation information of the data to be stored, and monitor the update status of the segment metadata corresponding to the data to be stored.

[0028] The first data segmentation module is used to segment the data to be stored according to the preset segmentation rules and the current segment number allocation information, store the segmented data, and determine the update information of the segmented metadata.

[0029] The data storage module is used to update the segmented metadata based on the update information when the update status is not updated.

[0030] According to a fifth aspect of the present invention, a data reading apparatus is provided, comprising:

[0031] The second metadata acquisition module is used to acquire segmented metadata of the data to be read in response to a data reading request; the data to be read is stored using any data storage method of the first aspect of the present invention.

[0032] The second data segmentation module is used to determine the segmented data index corresponding to the data to be read based on the number of segments and the starting segment number in the segmentation metadata.

[0033] The data reading module is used to read the segmented data corresponding to the data to be read according to the segmented data index corresponding to the data to be read; and to assemble the segmented data corresponding to the data to be read to obtain the data to be read.

[0034] According to a sixth aspect of the present invention, a data deletion apparatus is provided, comprising:

[0035] The third-level metadata acquisition module is used to acquire segment metadata corresponding to the data to be deleted in response to a data deletion request; the data to be deleted is stored using any data storage method of the first aspect of the present invention.

[0036] The third data segmentation module is used to determine the segmented data index corresponding to the data to be deleted based on the number of segments and the starting segment number in the segmentation metadata.

[0037] The data deletion module is used to delete the data to be deleted according to the segmented data index corresponding to the data to be deleted.

[0038] According to a seventh aspect of the present invention, an electronic device for data storage, retrieval, and deletion is provided, comprising:

[0039] One or more processors;

[0040] Storage device for storing one or more programs.

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiments of the present invention.

[0042] According to an eighth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods provided in the first, second, and third aspects of the present invention.

[0043] According to a ninth aspect of the present invention, a computer program product is provided. One such computer program product includes a computer program that, when executed by a processor, implements the data query method provided in the embodiments of the present application.

[0044] One embodiment of the invention has the following advantages or beneficial effects: by responding to a data storage request, obtaining the current segment number allocation information of the data to be stored, and monitoring the update status of the segment metadata corresponding to the data to be stored; by segmenting the data to be stored according to the preset segmentation rules and the current segment number allocation information, storing the segmented data, and determining the update information of the segment metadata; by updating the segment metadata according to the update information when the update status is not updated, and by performing data reading and deletion operations based on the segment metadata based on the already stored data, the technical solution effectively solves the problem of storing, reading, and deleting large amounts of data in the Redis cache, and by monitoring the update status of the segment metadata, the accuracy of data storage and reading is ensured. Attached Figure Description

[0045] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0046] Figure 1 This is a schematic diagram of the main flow of a data storage method according to an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram illustrating the segmentation processing principle of the data to be stored in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of multi-threaded batch storage according to an embodiment of the present invention;

[0049] Figure 4 This is a detailed flowchart illustrating the data storage process according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the main flow of a data reading method according to an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of multi-threaded batch reading according to an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the data reading process according to an embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of the main flow of a data deletion method according to an embodiment of the present invention;

[0054] Figure 9 This is a schematic diagram of the data deletion process according to an embodiment of the present invention;

[0055] Figure 10 This is a schematic diagram of the main modules of a data storage device according to an embodiment of the present invention;

[0056] Figure 11 This is a schematic diagram of the main modules of a data reading device according to an embodiment of the present invention;

[0057] Figure 12 This is a schematic diagram of the main modules of a data deletion device according to an embodiment of the present invention;

[0058] Figure 13 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0059] Figure 14 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0060] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0061] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0062] Currently, for Redis read / write issues involving large volumes of data, adjusting system parameters and retry logic to adapt to the data can solve a small portion of the read / write problems, but it cannot fundamentally solve the read / write problem for large volumes of data and will also affect the normal business of other data on the Redis instance. On the other hand, the method of storing data separately not only directly affects the read speed but also consumes more hardware and software resources, which cannot meet the needs of practical applications.

[0063] To address the aforementioned problems in existing technologies, this invention proposes a method for data storage, retrieval, and deletion. Based on the obtained current segment number allocation information and segmentation rules of the data to be stored, the data is segmented and stored. If the segment metadata status is not updated, the segment metadata is updated. Based on the already stored data, data retrieval and deletion operations are performed according to the segment metadata. This method effectively solves the problem of storing, retrieving, and deleting large volumes of data in Redis cache, and ensures the accuracy of data storage and retrieval by monitoring the update status of segment metadata.

[0064] In the description of the embodiments of the present invention, the terms involved and their meanings are as follows:

[0065] CAS (Compare and Swap) is an optimistic locking mechanism for handling resource modifications in concurrent scenarios. Before modification, the resource's value is A. If it's changed to B, the modification will only succeed if the resource's value is still A at the time of commit. Otherwise, the modification will fail. CAS can lead to the ABA problem: if thread 1 obtains the resource's value as A before modification, and then thread 2 also obtains the resource's value as A, and thread 2 modifies the resource's value to A and commits successfully, then if thread 1 changes the resource's value to B, thread 1 will assume that the resource's value hasn't been modified by other threads before its own modification and will commit successfully. However, the resource's value has actually been modified by thread 2.

[0066] Redis Transactions: Redis transactions differ from those in typical relational databases. Before a transaction begins, it supports watching a set of keys, with their values ​​acting as optimistic locks. When executing the transaction on the Redis server, it first checks if the values ​​corresponding to the watched keys have been modified. If modified, the transaction is abandoned. If not modified, the transaction proceeds. Redis's watch mechanism avoids the ABA problem because Redis is single-threaded and does not handle concurrency, therefore it does not use CAS (Compare-and-Swap) mechanisms. Commands within a transaction execute independently. Even if one command fails, it will not affect the execution of other commands.

[0067] Slots: Redis clusters have 16,384 built-in slots. Each node in the Redis cluster is responsible for several slots. When reading or writing key-value pairs to the Redis cluster, the hash algorithm of the key is used to divide the number of slots by the hash value. Only after the key is located can the read or write operation be performed.

[0068] Dirty read: A transaction reads data that another transaction has not yet committed.

[0069] Dirty write: A transaction modifies data that has not yet been modified by another transaction.

[0070] incrby: is an atomic increment or decrement operation;

[0071] Atomicity: refers to the fact that an operation is indivisible; either all operations occur or none occur.

[0072] Figure 1 This is a schematic diagram of the main flow of a data storage method according to an embodiment of the present invention, such as... Figure 1 As shown, the data storage method of this embodiment of the invention includes the following steps S101 to S103.

[0073] Step S101: In response to the data storage request, obtain the current segment number allocation information of the data to be stored, and monitor the update status of the segment metadata corresponding to the data to be stored.

[0074] According to an embodiment of the present invention, before obtaining the current segment number allocation information of the data to be stored, the method includes: determining that the amount of data to be stored exceeds a preset processing threshold.

[0075] Specifically, to address the storage problem of large volumes of data in Redis, this invention primarily targets large volumes of data. Here, "data volume" can be understood as the amount of storage space occupied by the data, or the number of sub-data entries included in the data, etc., without specific limitations. Before storing data, it is necessary to determine the volume of the data to be stored, confirming that it exceeds a preset processing threshold. This threshold can be set based on business experience or flexibly configured according to the needs of the scenario.

[0076] Furthermore, in a concurrent environment, for key-value data stored in Redis, multiple threads may simultaneously perform storage operations on the same key with different data values. For example, multiple business accounts may concurrently update data in a specific data storage unit or storage address using multiple threads. This embodiment of the invention considers the concurrency conflicts that may occur in a multi-threaded environment. Before storage processing, segmentation metadata is allocated to the data to be stored to record segmentation information. The original key of the data to be stored is used as the key of the segmentation metadata, and the update status of the segmentation metadata is monitored. Specifically, the received data storage request is parsed to obtain the current segment number allocation information required for subsequent segmentation processing of the data to be stored, as well as the original key of the data to be stored. Based on the original key, the segmentation metadata corresponding to the data to be stored can be obtained. The update status of the segmentation metadata in the subsequent processing flow is obtained by monitoring the key of the segmentation metadata through Redis's watch mechanism.

[0077] In this embodiment of the invention, the segment number allocation information is dynamically updated in real time, with a set initial value used to determine the index of the data to be stored in the Redis database. Since this embodiment is primarily applied to multiple threads simultaneously performing storage operations on the same key with different data values, the segment number allocation information is dynamically updated by incrementing or decrementing according to each storage operation, providing a dedicated and unique index for subsequent data storage. For example, a specific shard in a sharded database can be used as the range for segment number allocation, with the segment number allocation information incrementing or decrementing within this specified range. Alternatively, the entire storage space of the database can be used as the range for segment number allocation, or different business accounts can be used as the range for segment number allocation, with different business accounts associated with different segment number allocation ranges to meet the segment number allocation needs of various business scenarios.

[0078] Step S102: According to the preset segmentation rules and the current segment number allocation information, the data to be stored is segmented, the segmented data is stored, and the update information of the segmented metadata is determined.

[0079] It should be noted that before segmenting the data to be stored, algorithms with suitable compression ratios and decompression speeds, or efficient serialization algorithms, can be used to compress the data to minimize its size. Furthermore, the data to be stored in this embodiment can be a byte array, or data converted from other data types to a byte array, to facilitate segmentation and splitting.

[0080] According to an embodiment of the present invention, the method includes: before segmenting the data to be stored, obtaining segmentation metadata corresponding to the data to be stored, storing the segmentation metadata in a local cache, and determining the data to be stored based on the segmentation metadata; after updating the segmentation metadata based on the update information, obtaining the segmentation metadata from the local cache, and deleting the historical stored data corresponding to the segmentation metadata based on the segmentation metadata.

[0081] Specifically, before segmenting the data to be stored, segmentation metadata is obtained and stored locally. On the one hand, when the data to be stored is uncertain and requires logical operations on the currently stored data to determine it, the segmentation metadata is used to obtain the currently stored data, and then logical operations are performed on the currently stored data to obtain the specific value of the data to be stored. On the other hand, after updating the segmentation metadata according to the update information and completing the storage of the data to be stored, the data to be stored has been stored and taken effect, and the aforementioned currently stored data has become historical stored data. The historical stored data can be deleted by using the segmentation metadata obtained during the data storage request stored in the cache.

[0082] According to another embodiment of the present invention, the data to be stored is segmented according to a preset segmentation rule and the current segment number allocation information, including: splitting the data to be stored into multiple segments according to a segmentation threshold in the preset segmentation rule; determining the current segment start number of the data to be stored according to the prefix information and number information in the current segment number allocation information; allocating a corresponding segment data index to each segment data according to the current segment start number and the number of segments; and determining the update information of the segment metadata, including: using the current segment start number and the number of segments as the update information of the segment metadata.

[0083] Specifically, the data to be stored is split according to the segmentation threshold in the preset segmentation rules. For example, if the size of the data to be stored is 101K and the segmentation threshold is 10K, then 101K is split into segments of 10K each. The first 10 segments are all 10K in size, and the 11th segment is 1K in size, resulting in 11 segments of data.

[0084] Furthermore, considering the multi-threaded concurrency scenario in this embodiment of the invention, this embodiment uses Redis's `incrby` command, an atomic counting method, and sets a non-business type of Key-Value data: segment number allocation information. The Key represents a preset prefix, which can be a custom string; the corresponding Value is an allocatable number, which is updated in real-time as the data to be stored is stored. The prefix and number information can be used to determine the current segment starting number. For example, if the Key of the segment number allocation information is defined as `segm`, and its current Value is 0, then the current segment starting number can be determined as `segm-0`.

[0085] Based on the number of segments and the starting segment number of the current segment, a corresponding segment data index can be assigned to each segment. For example, if there are 5 segments, obtain the current segment number allocation information, and use Redis's atomic counting command `incrby` on the segment number allocation information. After incrementing the value of the segment number allocation data by 5, the count result is returned as 5. Since the count result of `incrby` is 5, we know that the number of subsequent allocable numbers is 5, which means that the number of the last segment data in the current segment is 4. Combining the segment data index definition rule agreed in the segmentation rules, which is: based on the increment of the original key of the data to be stored - the starting segment number, we can obtain the segment data indices corresponding to the 5 segments as follows: original key - segm0, original key - segm1, original key - segm2, original key - segm3, and original key - segm4. It should be noted that in this embodiment of the invention, incrby is set to increment by 1 each time. In actual use, by setting the definition rules of the segmented data index in the segmentation rules, the count can be incremented or decremented according to a specified number. This embodiment of the invention does not make specific limitations.

[0086] In addition, the segmentation metadata mainly includes the number of segments and the starting segment number. Through the above processing, the current starting segment number and the number of segmented data can be used as the update information of the segmentation metadata corresponding to the data to be stored.

[0087] By using Redis's atomic counting `incrby` command to request a unique starting segment number for each thread, and then assigning a corresponding segment data index to each segment based on the starting segment number, concurrency issues in the allocation of indexes for different segments between threads are avoided.

[0088] Figure 2This is a schematic diagram illustrating the segmentation process of data to be stored according to an embodiment of the present invention. The diagram shows the original Key and Value of the data to be stored being segmented. Based on the segmentation threshold in the preset segmentation rules, the data to be stored is divided into n+1 segments, resulting in a segment count of n+1, corresponding to the segment_count in the segment metadata value of the diagram. Based on the obtained current segment number allocation information, the starting segment number of the current segment can be determined, which is the start_segment in the segment metadata value of the diagram. The original Key of the data to be stored is used as the Key of the segment metadata, resulting in the segment metadata shown in the diagram. According to the preset segmentation rules, the original Key - starting segment number is used as the segment data index Key for the first segment. Following the order of the segment data, a corresponding segment data index Key is assigned to each segment data according to a rule that increments by 1 each time, i.e., original Key - starting segment number, original Key - starting segment number + 1, original Key - starting segment number + 2, ..., original Key - starting segment number + n.

[0089] According to another embodiment of the present invention, storing segmented data includes: when the number of segmented data exceeds a preset quantity threshold, determining the corresponding cache slot for each segmented data according to the segmented data index of each segmented data, and batch storing the segmented data in the same cache slot in multiple threads.

[0090] Specifically, through the data segmentation process described above, large volumes of data are divided into several segments. These segments can then be indexed and stored in Redis. It's important to note that this storage of segmented data only involves storing the segmented data in Redis; the corresponding segment metadata is not updated. It remains the same metadata obtained in the initial response to the data storage request, and therefore is not visible externally. External interfaces will not read this segmented data, thus avoiding the dirty read problem.

[0091] Furthermore, to improve storage efficiency, when the number of segmented data exceeds a preset threshold, the cache slot corresponding to each segmented data index key can be obtained through the Redis keyslot command based on the index key of each segmented data. Segmented data in the same slot can be placed in a batch of appropriate size. The Redis mset command can be used to batch store segmented data in the same slot in multiple batches. To further accelerate the storage speed, a multi-threaded concurrent storage approach can be adopted, sending storage requests to multiple Redis nodes simultaneously.

[0092] Figure 3This is a schematic diagram of multi-threaded batch storage according to an embodiment of the present invention. In the diagram, segmented data with the same slot are placed in one thread, such as segmented data 1, segmented data 3, ..., segmented data 9 having the same slot, and are collectively placed in thread 1. Multiple threads are used to write to multiple Redis nodes simultaneously, realizing multi-threaded batch storage and improving storage efficiency.

[0093] By using the multi-threaded batch storage described above, the number of RPC calls to Redis in segmented data storage can be effectively reduced, greatly improving the storage speed of segmented data and thus increasing storage efficiency.

[0094] Step S103: If the update status is not updated, update the segmented metadata according to the update information.

[0095] Specifically, after storing the segmented data as described above, the `muti` command is used to start a Redis transaction that updates the segmented metadata. The number of segments and the starting segment number obtained from the segmentation process are used as update information, and the update command for the segmented metadata is input. The Redis transaction is committed, and the update status of the segmented metadata corresponding to the data to be stored is obtained. If the update status of the segmented metadata has not been updated since the data storage request was received, that is, since the initial listening to the segmented metadata, the segmented metadata has not been modified by other threads, the commit is successful, and the update of the segmented metadata takes effect.

[0096] In addition, after successful submission, the data to be stored has been successfully stored. Accordingly, when the data storage request is initially received, the historical segment data pointed to by the segment metadata obtained based on the locally stored segment metadata is no longer needed and can be deleted.

[0097] According to another embodiment of the present invention, when the update status is updated, the segmented data is deleted, the current segment number allocation information of the data to be stored is re-acquired, and the update status of the segment metadata corresponding to the data to be stored is monitored. The data to be stored is segmented according to the re-acquired current segment number allocation information and the preset segmentation rules.

[0098] Specifically, if the update status is "updated," it means that between the initial monitoring of the segment metadata update status and the submission of the Redis transaction to update the segment metadata, another thread has already updated the segment metadata. Therefore, this Redis transaction to update the segment metadata has failed, and consequently, the storage operation for the data to be stored has failed. It is necessary to roll back and delete the segment data already stored in Redis. Since Redis transactions themselves do not have a rollback mechanism, you can write your own code to delete the data. Furthermore, to better meet practical application needs, a retry count can be set. After the Redis transaction fails to commit, the latest current segment number allocation information can be retrieved again, and the update status of the segment metadata can be monitored. Based on the re-retrieved current segment number allocation information and the preset segmentation rules, the latest data to be stored can be segmented.

[0099] The update status of segment metadata determines whether a Redis transaction is allowed to update segment metadata. If segment metadata is modified by other threads, the Redis transaction will fail to commit and the segment metadata will not be allowed to be updated. This can effectively avoid the dirty write problem of segment metadata.

[0100] Figure 4This is a detailed flowchart illustrating the data storage process according to an embodiment of the present invention. In response to a data storage request, the current segment number allocation information of the data to be stored is obtained, and the original key of the data to be stored in the storage request is used as the key of the segment metadata corresponding to the data to be stored. To minimize the risk of dirty reads and dirty writes, upon receiving a data storage request, Redis's Watch mechanism is used to monitor the update status of the segment metadata by listening to the key of the segment metadata; and the segment metadata is read according to the key. To ensure that the segment metadata has not been modified to the greatest extent, a judgment logic is added to this step. If the read fails, it means that the segment metadata has been modified by other threads, and the current monitoring (unwatch) needs to be canceled, the latest segment number allocation information is obtained again, the key of the segment metadata is monitored again, and the latest segment metadata is read again; if the read succeeds, the data to be stored is segmented according to the preset segmentation rules to obtain segmented data; based on the number of segmented data and the determined starting segment number, each segment is assigned a segmented data segment. Each data segment is assigned a corresponding segment index; the segment data is stored according to the corresponding segment index; the update information of the segment metadata is determined; the Redis transaction for updating the segment metadata is invoked, and the Redis transaction is executed; the command to update the metadata is set; the Redis transaction for updating the segment metadata is committed, and if the update status of the monitored segment metadata is not updated, the Redis transaction for updating the segment metadata is committed successfully, the segment metadata is updated, the data is stored successfully, and the Redis transaction is committed successfully; the historical segment data corresponding to the segment metadata read when the data storage request is received is deleted; if the update status of the monitored segment metadata is updated, it means that the data storage failed, the Redis transaction commit failed, and the segment data of this segment split is deleted.

[0101] Through the above data storage process, it can be seen that this embodiment of the invention achieves the storage of large amounts of data through two-stage storage commit (storage of segmented data and commit update of segmented metadata). Combined with Redis's transaction-based determination of the update status of segmented metadata during commit, this approach effectively stores large volumes of data. Since Redis's watch mechanism is not based on CAS, it avoids the ABA problem. This means that even if other threads modify the segmented metadata, but the content remains unchanged, Redis will still consider the metadata to have been modified, effectively preventing dirty reads and dirty writes and maximizing the accuracy of data storage and retrieval.

[0102] Figure 5 This is a schematic diagram of the main flow of a data reading method according to an embodiment of the present invention, as shown below. Figure 5 As shown, the data reading method of this embodiment of the invention includes the following steps S501 to S503.

[0103] Step S501: In response to the data reading request, obtain the segmented metadata of the data to be read; the data to be read is stored using any data storage method of the first aspect of the present invention.

[0104] Specifically, based on the above data storage method, a reading operation is performed on the stored data, the original key of the data to be read in the data reading request is parsed, the original key is used as the segment metadata key of the data to be read, and the segment metadata is obtained according to the segment metadata key.

[0105] Step S502: Determine the segment data index corresponding to the data to be read based on the number of segments and the starting segment number in the segment metadata.

[0106] Specifically, based on the number of segments and the starting segment number in the segment metadata, the segment data index corresponding to the data to be read is determined according to the increment of the segment data index in the preset segmentation rules, starting from the starting segment number.

[0107] Step S503: Read the segmented data corresponding to the data to be read according to the segmented data index; assemble the segmented data corresponding to the data to be read to obtain the data to be read.

[0108] According to an embodiment of the present invention, reading segmented data corresponding to the data to be read according to the segmented data index corresponding to the data to be read includes: when the number of segments corresponding to the data to be read exceeds a preset number threshold, determining the cache slot of the corresponding segmented data according to each segmented data index corresponding to the data to be read, and reading the segmented data in the same cache slot in batches using multiple threads.

[0109] Specifically, when the number of segments corresponding to the data to be read exceeds a preset threshold, similar to the multi-threaded batch storage mentioned above, the cache slot corresponding to each segment data index key can be obtained through the Redis keyslot command based on the index key of each segment data. Segment data indexes in the same slot are placed in a batch of appropriate size. The Redis mset command can be used to read segment data stored in the same slot in multiple batches. To further speed up the reading process, a multi-threaded concurrent reading approach can be adopted to read multiple Redis nodes simultaneously.

[0110] Figure 6 This is a schematic diagram of multi-threaded batch reading according to an embodiment of the present invention. The diagram shows that segmented data indexes with the same slot are placed in one reading thread, and multiple segments of data with the same slot on the Redis node are read at once, using n threads for batch reading.

[0111] By using the multi-threaded batch segmented data reading described above, the number of RPC calls to Redis during segmented data reading can be effectively reduced, greatly improving the reading speed of segmented data and thus increasing reading efficiency.

[0112] Furthermore, based on the read segmented data and the numbering order in the segmented data index, the segmented data is assembled sequentially to finally obtain the data to be read. For byte array type data to be read, direct assembly is possible. However, if the data is of other types that have undergone type conversion to obtain byte values, further type conversion is required to convert it to the initial data type to meet the needs of practical applications.

[0113] According to another embodiment of the present invention, if reading the segmented data corresponding to the data to be read fails, the segmented metadata of the data to be read is re-acquired, and the segmented data index corresponding to the data to be read is determined based on the number of segments and the starting segment number in the re-read segmented metadata, and the segmented data is read.

[0114] Specifically, if a certain segment of data does not exist and reading the segment data fails, it means that the segment metadata has been updated and changed during the reading process. At this time, it is necessary to re-acquire the segment metadata of the data to be read and re-execute the data reading operation based on the newly acquired segment metadata.

[0115] Figure 7 This is a schematic diagram of the data reading process according to an embodiment of the present invention. The original key of the data to be read in the received data reading request is used as the key of the segment metadata corresponding to the data to be read, and the segment metadata is obtained. Based on the number of segments and the starting segment number in the segment metadata, a segment data index is obtained, and the segment data is read according to the segment data index. If the segment data is read successfully, the segment data is reassembled in order according to the order information included in the segment data index to obtain the data to be read. If the segment data reading fails, the current reading fails, and it is necessary to reread the latest segment metadata based on the original key in the reading request, and then read the new segment data and reassemble the new segment data.

[0116] Figure 8 This is a schematic diagram of the main flow of the data deletion method according to an embodiment of the present invention, as shown below. Figure 8 As shown, the data reading method of this embodiment of the invention includes the following steps S801 to S803.

[0117] Step S801: In response to the data deletion request, obtain the segment metadata corresponding to the data to be deleted; the data to be deleted is stored using any data storage method of the first aspect of the present invention.

[0118] Specifically, based on the above data storage method, a deletion operation is performed on the stored data, the original key of the data to be deleted in the data deletion request is parsed, the original key is used as the segment metadata key of the data to be deleted, and the segment metadata is obtained according to the segment metadata key.

[0119] Step S802: Determine the segment data index corresponding to the data to be deleted based on the number of segments and the starting segment number in the segment metadata.

[0120] Specifically, based on the number of segments and the starting segment number in the segment metadata, the segment data index corresponding to the data to be deleted is determined according to the increment of the segment data index in the preset segmentation rules, starting from the starting segment number.

[0121] Step S803: Delete the data to be deleted according to the segmented data index corresponding to the data to be deleted.

[0122] According to an embodiment of the present invention, deleting the data to be deleted according to the segment data index corresponding to the data to be deleted includes: when the number of segments corresponding to the data to be deleted exceeds a preset number threshold, determining the cache slot of the corresponding segment data according to each segment data index corresponding to the data to be read, and deleting the segment data in the same cache slot in batches using multiple threads.

[0123] Specifically, based on the segmented data indexes corresponding to the data to be deleted as determined above, the data to be deleted is deleted. When the number of segments exceeds a preset threshold, similar to the multi-threaded batch read described above, the cache slot corresponding to each segmented data index key can be obtained using the Redis keyslot command. Segmented data indexes in the same slot are placed in a batch of appropriate size. The Redis del command can then be used to batch delete segmented data stored in the same slot in multiple batches. To further accelerate the deletion speed, a multi-threaded concurrent deletion approach can be adopted, deleting multiple Redis nodes simultaneously. Multi-threaded batch deletion of segmented data effectively reduces the RPC calls to Redis during segmented data deletion, significantly improving the deletion speed and thus increasing deletion efficiency.

[0124] Figure 9 This is a schematic diagram of the data deletion process according to an embodiment of the present invention. The original key of the data to be deleted in the received data deletion request is used as the key of the segment metadata corresponding to the data to be deleted, and the segment metadata is obtained. Based on the number of segments and the starting segment number in the segment metadata, the segment data index is obtained. The segment data is deleted according to the segment data index, and the segment metadata is also deleted.

[0125] Figure 10 This is a schematic diagram of the main modules of a data storage device according to an embodiment of the present invention. Figure 10 As shown, the data storage device 1000 mainly includes a first metadata acquisition module 1001, a first data segmentation module 1002, and a data storage module 1003.

[0126] The first metadata acquisition module 1001 is used to respond to a data storage request, acquire the current segment number allocation information of the data to be stored, and monitor the update status of the segment metadata corresponding to the data to be stored.

[0127] The first data segmentation module 1002 is used to segment the data to be stored according to the preset segmentation rules and the current segment number allocation information, store the segmented data, and determine the update information of the segmentation metadata.

[0128] The data storage module 1003 is used to update the segmented metadata according to the update information when the update status is not updated.

[0129] According to an embodiment of the present invention, the data storage device 1000 further includes a data volume determination module (not shown in the figure), which is used to: determine that the data volume of the data to be stored exceeds a preset processing threshold before obtaining the current segment number allocation information of the data to be stored.

[0130] According to another embodiment of the present invention, the data storage device 1000 further includes a segmented data to be stored determination module (not shown in the figure), configured to: obtain segmented metadata corresponding to the data to be stored before segmenting the data to be stored, store the segmented metadata in a local cache, and determine the data to be stored based on the segmented metadata; the data storage device 1000 further includes a data deletion module (not shown in the figure), configured to: obtain the segmented metadata from the local cache after updating the segmented metadata according to the update information, and delete the historical stored data corresponding to the segmented metadata based on the segmented metadata.

[0131] According to another embodiment of the present invention, the first data segmentation module 1002 is further configured to: split the data to be stored into multiple segments according to a segmentation threshold in a preset segmentation rule; determine the current segment start number of the data to be stored according to the prefix information and number information in the current segment number allocation information; allocate a corresponding segment data index to each segment data according to the current segment start number and the number of segments data; and use the current segment start number and the number of segments data as update information for the segment metadata.

[0132] According to another embodiment of the present invention, the first data segmentation module 1002 is further configured to: when the number of segmented data exceeds a preset quantity threshold, determine the corresponding cache slot of the segmented data according to the segmented data index of each segmented data, and perform batch storage of the segmented data in the same cache slot in multiple threads.

[0133] According to another embodiment of the present invention, the data storage device 1000 further includes a data deletion module (not shown in the figure), configured to: after storing the segmented data, delete the segmented data when the update status is updated, re-acquire the current segment number allocation information of the data to be stored, monitor the update status of the segment metadata corresponding to the data to be stored, and perform segmentation processing on the data to be stored according to the re-acquired current segment number allocation information and the preset segmentation rules.

[0134] Figure 11 This is a schematic diagram of the main modules of a data reading device according to an embodiment of the present invention. Figure 11 As shown, the data reading device 1100 mainly includes a second metadata acquisition module 1101, a second data segmentation module 1102, and a data reading module 1103.

[0135] The second metadata acquisition module 1101 is used to acquire segmented metadata of the data to be read in response to a data reading request; the data to be read is stored using any data storage method of the first aspect of the present invention.

[0136] The second data segmentation module 1102 is used to determine the segmented data index corresponding to the data to be read based on the number of segments and the starting segment number in the segmentation metadata.

[0137] The data reading module 1103 is used to read the segmented data corresponding to the data to be read according to the segmented data index; and to assemble the segmented data corresponding to the data to be read to obtain the data to be read.

[0138] According to an embodiment of the present invention, the data reading device 1100 further includes a reading determination module (not shown in the figure), which is used to: in the case of failure to read the segmented data corresponding to the data to be read, re-acquire the segmented metadata of the data to be read, determine the segmented data index corresponding to the data to be read according to the number of segments and the segment start number in the re-read segmented metadata, and read the segmented data.

[0139] According to another embodiment of the present invention, the data reading module 1103 is further configured to: when the number of segments corresponding to the data to be read exceeds a preset number threshold, determine the cache slot of the corresponding segment data according to the index of each segment data corresponding to the data to be read, and perform batch reading of the segment data in the same cache slot in multiple threads.

[0140] Figure 12 This is a schematic diagram of the main modules of a data deletion device according to an embodiment of the present invention. Figure 12 As shown, the data deletion device 1200 mainly includes a third-dimensional information acquisition module 1201, a third-dimensional data segmentation module 1202, and a data deletion module 1203.

[0141] The third-level metadata acquisition module 1201 is used to acquire segment metadata corresponding to the data to be deleted in response to a data deletion request; the data to be deleted is stored using any data storage method of the first aspect of the present invention.

[0142] The third data segmentation module 1202 is used to determine the segmented data index corresponding to the data to be deleted based on the number of segments and the starting segment number in the segmentation metadata.

[0143] The data deletion module 1203 is used to delete the data to be deleted according to the segmented data index corresponding to the data to be deleted.

[0144] According to an embodiment of the present invention, the data deletion module 1203 is further configured to: when the number of segments corresponding to the data to be deleted exceeds a preset number threshold, determine the cache slot of the corresponding segment data according to the index of each segment data corresponding to the data to be read, and perform batch deletion of the segment data in the same cache slot in multiple threads.

[0145] Figure 13 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied.

[0146] like Figure 13 As shown, system architecture 1300 may include terminal devices 1301, 1302, and 1303, network 1304, and server 1305. Network 1304 is used as a medium to provide a communication link between terminal devices 1301, 1302, and 1303 and server 1305. Network 1304 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0147] Users can use terminal devices 1301, 1302, and 1303 to interact with server 1305 via network 1304 to receive or send messages, etc. Various communication client applications, such as data processing applications, can be installed on terminal devices 1301, 1302, and 1303 (for example only).

[0148] Terminal devices 1301, 1302, and 1303 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0149] Server 1305 can be a server providing various services, such as a backend management server (for example only) supporting data storage, retrieval, and deletion performed by users using terminal devices 1301, 1302, and 1303. The backend management server can respond to data storage requests by obtaining the current segment number allocation information of the data to be stored and monitoring the update status of the segment metadata corresponding to the data to be stored; according to preset segmentation rules and the current segment number allocation information, it can segment the data to be stored, store the segmented data, and determine the update information of the segment metadata; if the update status is not updated, it can update the segment metadata according to the update information, and feed back the processing result (e.g., storing data – for example only) to the terminal devices.

[0150] It should be noted that the data storage, reading and deletion methods provided in the embodiments of the present invention are generally executed by the server 1305, and correspondingly, the data storage, reading and deletion devices are generally located in the server 1305.

[0151] It should be understood that Figure 13 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0152] The following is for reference. Figure 14 It shows a schematic diagram of the structure of a computer system 1400 suitable for implementing terminal devices or servers of the present invention. Figure 14 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0153] like Figure 14As shown, the computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1402 or programs loaded from storage section 1408 into random access memory (RAM) 1403. The RAM 1403 also stores various programs and data required for the operation of the system 1400. The CPU 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0154] The following components are connected to I / O interface 1405: an input section 1406 including a keyboard, mouse, etc.; an output section 1407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to I / O interface 1405 as needed. Removable media 1411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1410 as needed so that computer programs read from them can be installed into storage section 1408 as needed.

[0155] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1409, and / or installed from removable medium 1411. When the computer program is executed by central processing unit (CPU) 1401, it performs the functions defined above in the system of this invention.

[0156] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0158] The units described in the embodiments of the present invention can be implemented in software or in hardware. The described units can also be housed in a processor; for example, a processor can be described as including: a first metadata acquisition module, a first data segmentation module, and a data storage module.

[0159] In some cases, the names of these modules do not constitute a limitation on the module itself. For example, the data storage module can also be described as "a module for updating the segmented metadata according to the update information when the update status is not updated".

[0160] On the other hand, the present invention also provides a computer-readable medium, which may be included in the device described in the embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: responding to a data storage request, acquiring current segment number allocation information of the data to be stored, and monitoring the update status of the segment metadata corresponding to the data to be stored; segmenting the data to be stored according to a preset segmentation rule and the current segment number allocation information, storing the segmented data, and determining update information of the segment metadata; and updating the segment metadata according to the update information when the update status is not updated.

[0161] The technical solution of the present invention has the following advantages or beneficial effects: by responding to a data storage request, obtaining the current segment number allocation information of the data to be stored, and monitoring the update status of the segment metadata corresponding to the data to be stored; by performing segmentation processing on the data to be stored according to the preset segmentation rules and the current segment number allocation information, storing the segmented data, and determining the update information of the segment metadata; by updating the segment metadata according to the update information when the update status is not updated, and by performing data reading and deletion operations based on the segment metadata based on the already stored data, the technical solution effectively solves the problem of storing, reading, and deleting large amounts of data in the Redis cache, and by monitoring the update status of the segment metadata, the accuracy of data storage and reading is ensured.

[0162] The specific embodiments described herein do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for data storage, characterized in that, include: In response to a data storage request, the system obtains the current segment number allocation information of the data to be stored and monitors the update status of the segment metadata corresponding to the data to be stored. According to the preset segmentation rules and the current segment number allocation information, the data to be stored is segmented, the segmented data is stored, and the update information of the segmented metadata is determined. The segment number allocation information is dynamically updated in real time and has a set initial value, used to determine the index of the data to be stored in the Redis database. To address conflicts in a multi-threaded concurrent environment, segment metadata is allocated to the data to be stored before storage processing to record segmentation information. The original key of the data to be stored is used as the key of the segment metadata, and the update status of the segment metadata is monitored. Monitoring the update status of the segment metadata corresponding to the data to be stored includes: parsing the received data storage request, obtaining the current segment number allocation information and the original key of the data to be stored required for subsequent segmentation processing, obtaining the segment metadata corresponding to the data to be stored based on the original key, and monitoring the key of the segment metadata through Redis's watch mechanism to obtain the update status of the segment metadata in subsequent processing flows. When multiple threads simultaneously perform storage operations on the same key with different data values, the segment number allocation information is dynamically updated by incrementing or decrementing according to each storage operation, providing a dedicated and unique index for subsequent data storage. This dynamic update of the segment number allocation information includes: using a specific shard in the database sharding as the range for segment number allocation, and updating the segment number allocation information by incrementing or decrementing within the specified range; or using the entire storage space of the database as the range for segment number allocation; or associating different business accounts with different segment number allocation ranges. If the update status is "not updated," the segment metadata is updated according to the update information. The method further includes: If the number of segmented data exceeds the preset threshold, the cache slot corresponding to each segmented data index is obtained through the Redis keyslot command based on the index of each segmented data. Segmented data with the same cache slot is placed in a batch, and the Redis mset command is used to perform batch storage of segmented data with the same cache slot in multiple batches. During batch storage, a multi-threaded concurrent storage method is adopted, and storage requests are sent to multiple Redis nodes at the same time.

2. The method according to claim 1, characterized in that, Before obtaining the current segment number allocation information of the data to be stored, the method includes: The amount of data to be stored is determined to exceed a preset processing threshold.

3. The method according to claim 1, characterized in that, The method includes: Before segmenting the data to be stored, obtain the segmentation metadata corresponding to the data to be stored, store the segmentation metadata in a local cache, and determine the data to be stored based on the segmentation metadata. After updating the segmented metadata based on the update information, the segmented metadata is retrieved from the local cache, and the historical stored data corresponding to the segmented metadata is deleted based on the segmented metadata.

4. The method according to claim 1, characterized in that, According to the preset segmentation rules and the current segment number allocation information, the data to be stored is segmented, including: According to the segmentation threshold in the preset segmentation rules, the data to be stored is split into multiple segments. Based on the prefix information and number information in the current segment number allocation information, determine the current segment start number of the data to be stored; Based on the current segment start number and the number of segment data, assign a corresponding segment data index to each segment data; Determining the update information of the segmented metadata includes: using the current segment start number and the number of segmented data as the update information of the segmented metadata.

5. The method according to claim 1, characterized in that, After storing the segmented data, the method further includes: If the update status is updated, delete the segmented data, reacquire the current segment number allocation information of the data to be stored, monitor the update status of the segment metadata corresponding to the data to be stored, and segment the data to be stored according to the reacquired current segment number allocation information and the preset segmentation rules.

6. A method for reading data, characterized in that, include: In response to a data read request, obtain the segmented metadata of the data to be read; The data to be read is stored using the method described in any one of claims 1-5; Based on the number of segments and the starting segment number in the segmented metadata, determine the segmented data index corresponding to the data to be read; Based on the segmented data index corresponding to the data to be read, read the segmented data corresponding to the data to be read; assemble the segmented data corresponding to the data to be read to obtain the data to be read.

7. The method according to claim 6, characterized in that, The method further includes: If reading the segmented data corresponding to the data to be read fails, the segmented metadata of the data to be read is re-acquired. Based on the number of segments and the starting segment number in the re-acquired segmented metadata, the segmented data index corresponding to the data to be read is determined, and the segmented data is read.

8. The method according to claim 6, characterized in that, Based on the segmented data index corresponding to the data to be read, read the segmented data corresponding to the data to be read, including: If the number of segments corresponding to the data to be read exceeds a preset threshold, the cache slot of the corresponding segment data is determined according to the index of each segment data corresponding to the data to be read, and the segment data in the same cache slot is read in batches by multiple threads.

9. A method for data deletion, characterized in that, include: In response to a data deletion request, obtain the segment metadata corresponding to the data to be deleted; The data to be deleted is stored using the method described in any one of claims 1-5; Based on the number of segments and the starting segment number in the segmented metadata, determine the segmented data index corresponding to the data to be deleted; Delete the data to be deleted based on the segmented data index corresponding to the data to be deleted.

10. The method according to claim 9, characterized in that, Delete the data to be deleted according to the segmented data index corresponding to the data to be deleted, including: If the number of segments corresponding to the data to be deleted exceeds a preset threshold, the cache slot of the corresponding segment data is determined according to the index of each segment data corresponding to the data to be deleted, and the segment data in the same cache slot is deleted in batches by multiple threads.

11. A data storage device, characterized in that, include: The first metadata acquisition module is used to respond to data storage requests, acquire the current segment number allocation information of the data to be stored, and monitor the update status of the segment metadata corresponding to the data to be stored. The first data segmentation module is used to segment the data to be stored according to the preset segmentation rules and the current segment number allocation information, store the segmented data, and determine the update information of the segmented metadata. The data storage module is used to update the segmented metadata based on the update information when the update status is not updated; The segment number allocation information is dynamically updated in real time and has a set initial value, used to determine the index of the data to be stored in the Redis database. To address conflicts in a multi-threaded concurrent environment, segment metadata is allocated to the data to be stored before storage processing to record segmentation information. The original key of the data to be stored is used as the key of the segment metadata, and the update status of the segment metadata is monitored. Monitoring the update status of the segment metadata corresponding to the data to be stored includes: parsing the received data storage request, obtaining the current segment number allocation information and the original key of the data to be stored required for subsequent segmentation processing, obtaining the segment metadata corresponding to the data to be stored based on the original key, and monitoring the key of the segment metadata through Redis's watch mechanism to obtain the update status of the segment metadata in subsequent processing flows. When multiple threads simultaneously perform storage operations on the same key with different data values, the segment number allocation information is dynamically updated by incrementing or decrementing according to each storage operation, providing a dedicated and unique index for subsequent data storage. The dynamic update of the segment number allocation information includes: using a specific shard in the database sharding as the range for segment number allocation, and updating the segment number allocation information by incrementing or decrementing within the specified range; or using the entire storage space of the database as the range for segment number allocation; or associating different business accounts with different segment number allocation ranges. The device is further configured to, when the number of segmented data exceeds a preset threshold, obtain the cache slot corresponding to each segmented data index using the Redis keyslot command, place the segmented data in the same cache slot in a batch, and use the Redis mset command to perform batch storage of the segmented data in the same cache slot in multiple batches. During batch storage, a multi-threaded concurrent storage method is adopted, and storage requests are sent to multiple Redis nodes simultaneously.

12. A data reading device, characterized in that, include: The second metadata acquisition module is used to acquire segmented metadata of the data to be read in response to a data read request. The data to be read is stored using the method described in any one of claims 1-5; The second data segmentation module is used to determine the segmented data index corresponding to the data to be read based on the number of segments and the starting segment number in the segmentation metadata. The data reading module is used to read the segmented data corresponding to the data to be read according to the segmented data index corresponding to the data to be read; and to assemble the segmented data corresponding to the data to be read to obtain the data to be read.

13. A data deletion device, characterized in that, include: The third-level metadata acquisition module is used to obtain the segment metadata corresponding to the data to be deleted in response to the data deletion request. The data to be deleted is stored using the method described in any one of claims 1-5; The third data segmentation module is used to determine the segmented data index corresponding to the data to be deleted based on the number of segments and the starting segment number in the segmentation metadata. The data deletion module is used to delete the data to be deleted according to the segmented data index corresponding to the data to be deleted.

14. A mobile electronic device terminal, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

15. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images