Data storage method, data query method and device of database, electronic equipment, computer readable storage medium and program product

Abstract

The application provides a data storage method and device of a database, a data query method and device, an electronic device, a computer readable storage medium and a program product. The method comprises: receiving a data write request, wherein the data write request carries to-be-written data; determining a first entry shard for storing the to-be-written data in a shard at a first level of a plurality of levels, wherein the first level is the first level in a top-down order; determining an index value of the to-be-written data in response to the type of the first entry shard being an index shard; recursively querying data shards subordinate to the first entry shard in a top-down order of the plurality of levels until a first data shard corresponding to the index value is queried; and writing the to-be-written data into the first data shard. Through the application, the space utilization and data read-write performance of data storage can be improved.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a database data storage method, data query method, apparatus, electronic device, computer-readable storage medium, and program product. Background Technology

[0002] In the field of data storage, such as scenarios involving infinitely long lists, both relational and non-relational databases face different challenges in practical use. In business scenarios with high requirements for data read / write speed and storage availability, the performance and cost considerations of storage solutions are particularly critical.

[0003] Data storage methods in related technologies often involve creating additional indexes on the stored data in relational databases to reduce resource consumption of the database storage instance. This leads to wasted storage space. When the number of data read / write requests per unit time is large, a higher number of processor cores is required for retrieval and computation, resulting in higher storage costs. Alternatively, related technologies record the location of data through index lists. When the long list of data exhibits sparse characteristics, storage space utilization is low, leading to a significant waste of unnecessary storage resources. Summary of the Invention

[0004] This application provides a database data storage method, data query method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the space utilization and data read / write performance of data storage.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a data storage method for a database. The database storage structure includes multiple layers, each layer including at least one shard. The shard types include index shards and data shards. The method includes:

[0007] Receive a data write request, wherein the data write request carries the data to be written;

[0008] In the fragments of the first level of the plurality of levels, a first entry fragment for storing the data to be written is determined, wherein the first level is the first level in the top-down order;

[0009] In response to the fact that the type of the first entry shard is the index shard, the index value of the data to be written is determined;

[0010] For data fragments belonging to the first entry fragment, recursively query according to the multiple levels in a top-down order until the first data fragment corresponding to the index value is found.

[0011] The data to be written is written to the first data fragment.

[0012] This application provides a data query method for a database. The database storage structure includes multiple layers, each layer including at least one shard. The shard types include index shards and data shards. The method includes:

[0013] Receive a data query request, wherein the data query request carries the index value to be queried;

[0014] In the first level of the multiple levels of the shards, the entry shard to which the data shard corresponding to the index value to be queried belongs is determined, wherein the first level is the first level in the top-down order;

[0015] In response to the entry shard being an index shard, the data shards belonging to the entry shard are recursively queried in a top-down order according to the multiple levels until the target data shard corresponding to the index value to be queried is found.

[0016] Extract the data to be queried from the target data fragments.

[0017] This application provides a data storage device for a database. The database storage structure includes multiple layers, each layer including at least one shard. The shard types include index shards and data shards. The device includes:

[0018] A request receiving module is used to receive a data write request, wherein the data write request carries data to be written;

[0019] A first determining module is configured to determine, among the fragments of the first level of the plurality of levels, a first entry fragment for storing the data to be written, wherein the first level is the first level in a top-down order;

[0020] The second determining module is used to determine the index value of the data to be written in response to the first entry fragment being the index fragment.

[0021] The first query module is used to recursively query the data fragments belonging to the first entry fragment in a top-down order of the multiple levels until the first data fragment corresponding to the index value is found.

[0022] The data writing module is used to write the data to be written into the first data segment.

[0023] This application provides a data storage device for a database. The database storage structure includes multiple layers, each layer including at least one shard. The shard types include index shards and data shards. The device includes:

[0024] A data receiving module is used to receive data query requests, wherein the data query request carries the index value to be queried;

[0025] The third determining module is used to determine the entry shard to which the data shard corresponding to the index value to be queried belongs in the shards of the first level of the multiple levels, wherein the first level is the first level in the top-down order;

[0026] The second query module is used to respond to the fact that the type of the entry shard is the index shard, and to recursively query the data shards belonging to the entry shard in the order of the multiple levels from top to bottom, until the target data shard corresponding to the index value to be queried is found.

[0027] The data extraction module is used to extract the data to be queried from the target data fragments.

[0028] This application provides an electronic device, the electronic device comprising:

[0029] Memory is used to store executable instructions or computer programs.

[0030] When a processor executes computer-executable instructions or computer programs stored in the memory, it implements the database data storage method provided in the embodiments of this application, or implements the database data query method provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the data storage method of the database provided in this application, or implements the data query method of the database provided in this application.

[0032] This application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, they implement the database data storage method provided in this application embodiment, or implement the database data query method provided in this application embodiment.

[0033] The embodiments of this application have the following beneficial effects:

[0034] In response to the fact that the first entry shard storing the data to be written is an index shard, the data shards belonging to the first entry shard are recursively queried, and the data to be written is written to the first data shard corresponding to the index value. Through data shards and index shards, the data in the database is managed in multiple layers, which reduces redundant storage, improves the utilization of storage space, and quickly locates the data shard through the index value, reducing the overhead of index storage, reducing the time to query the data shard, improving query efficiency, and thus reducing the latency of data writing. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the architecture of the database data storage system 100 provided in this application embodiment;

[0036] Figure 2A This is a schematic diagram of the structure of server 200-1 provided in an embodiment of this application;

[0037] Figure 2B This is a schematic diagram of the structure of server 200-2 provided in the embodiments of this application;

[0038] Figure 3A This is a first flowchart illustrating the database data storage method provided in this application embodiment;

[0039] Figure 3B This is a second flowchart illustrating the database data storage method provided in this application embodiment;

[0040] Figure 3C This is a schematic diagram of the third process of the database data storage method provided in the embodiments of this application;

[0041] Figure 3D This is a fourth flowchart illustrating the database data storage method provided in this application embodiment;

[0042] Figure 3E This is a schematic diagram of the fifth process of the database data storage method provided in the embodiments of this application;

[0043] Figure 3F This is a sixth flowchart illustrating the database data storage method provided in this application embodiment;

[0044] Figure 3G This is a seventh flowchart illustrating the database data storage method provided in this application embodiment;

[0045] Figure 3H This is the eighth flowchart of the database data storage method provided in the embodiments of this application;

[0046] Figure 3I This is a ninth flowchart illustrating the database data storage method provided in this application embodiment;

[0047] Figure 3J This is a tenth flowchart illustrating the database data storage method provided in this application embodiment;

[0048] Figure 3K This is a schematic diagram of the eleventh step of the data storage method for the database provided in this application embodiment;

[0049] Figure 4A This is a first flowchart illustrating the database data query method provided in this application embodiment;

[0050] Figure 4B This is a second flowchart illustrating the database data query method provided in the embodiments of this application;

[0051] Figure 5 This is a schematic diagram of the storage structure of the database provided in an embodiment of this application;

[0052] Figure 6A This is a schematic diagram of the process for reading long list data provided in an embodiment of this application;

[0053] Figure 6B This is a schematic diagram of the process for reading a single data item provided in an embodiment of this application;

[0054] Figure 6C This is a schematic diagram of the process for writing single data provided in an embodiment of this application;

[0055] Figure 6D This is a schematic diagram of the data rebalancing process provided in the embodiments of this application;

[0056] Figure 7 This is a schematic diagram illustrating the relationship between the proportion of different long lists and storage space utilization provided in the embodiments of this application.

[0057] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0060] In the following description, the terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0061] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0062] Unless otherwise specified, "at least one" as used below refers to one or more cases, and "multiple" can refer to two or more cases.

[0063] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0064] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0065] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0066] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0067] 2) Human-computer interaction interface, which is used to provide human-computer interaction functions / display the data to be queried.

[0068] For example, graphical user interfaces (GUIs) include augmented reality (AR) interfaces, virtual reality (VR) interfaces, voice user interfaces (VUIs), interactive projection interfaces (using projection technology to display information on a flat surface), eye-tracking interfaces (interfaces controlled by detecting the user's gaze), holographic interfaces (three-dimensional holograms formed by projecting images using holographic projection technology, allowing users to see stereoscopic images without wearing special glasses), multimodal interfaces (interfaces that combine multiple interaction methods, such as tactile, visual, and auditory interaction), and brain-machine interfaces (BMIs).

[0069] 3) Index sharding refers to a type of sharding in the database storage structure that does not directly store actual business data. Its data list records the index keys of all shards belonging to the index shard. With the help of these index keys, the subordinate data shards storing the actual data can be quickly located, which plays a role in guiding the direction of data query and storage.

[0070] 4) Data sharding refers to a type of sharding in the database storage structure used to directly store actual business data. A document consists of multiple parts, and the data of each part is stored in a data shard. The data in a data shard can be stored in the form of a list.

[0071] 5) The entry shard refers to the data shard selected in the first level of the database storage structure when a data write request is received. It is used to store the data to be written or to query the index of the data shard that stores the data to be written. It is the starting shard for the data write operation in the storage structure.

[0072] 6) An index value is a key used to identify the location of data in the database. When the first entry shard is an index shard, the value obtained after processing the data to be written using a specific calculation method (such as using a hash function) determines which specific data shard the data to be written should be routed to. The index value is equivalent to an address or pointer in the database, used to locate data.

[0073] 7) The home document refers to the document to which the data to be written is written. A database stores data from multiple documents, including the home document. Each document has a unique document identifier, used to uniquely identify the document.

[0074] 8) The "belonging section" refers to the logical partition within the document, representing a specific area or data segment within the document corresponding to the data to be written. A document can be divided into multiple sections based on its structure, each potentially including data of different types or formats. Each section has a part identifier used to locate and identify specific data segments within the document. During data writing, the part identifier of the belonging section is used to determine the specific location of the data to be written within the document, thus guiding the data to be written to the correct partition.

[0075] 9) Sparsity condition refers to the situation where the number of index shards and their subordinate data shards does not reach a preset number. Specifically, if the current number of shards of an index shard is greater than the number of fully loaded shards by a first preset multiple, then the index shard and its subordinate shards are considered to meet the sparsity condition. Meeting the sparsity condition indicates that the utilization rate of the index shard and its subordinate shards is low, and there are underutilized resources.

[0076] 10) Dense condition refers to a situation where the amount of data contained in any data shard under an index shard exceeds a length threshold. Specifically, if the length (i.e., the amount of data contained) of any data shard belonging to an index shard is greater than the length threshold, then the index shard and its subordinate shards are considered to meet the dense condition, indicating that the index shard and its subordinate shards have high utilization. This usually means that the index shard needs further partitioning or optimization to avoid excessively concentrated data impacting performance.

[0077] 11) Full load state refers to the index shard length (i.e., the number of data shards belonging to that index shard) reaching a specific length threshold. In this state, the index shard is considered to have reached its designed capacity limit.

[0078] 12) The number of fully loaded shards refers to the total number of data shards under an index shard when the length of the index shard (i.e., the number of data shards belonging to that index shard) reaches a specific length threshold. The length threshold represents the upper limit of the index shard's capacity; that is, when the length of an index shard equals the length threshold, the index shard is considered to be in a fully loaded state. In other words, the number of fully loaded shards refers to the maximum number of data shards subordinate to an index shard when it is in a fully loaded state.

[0079] Data storage methods in related technologies often involve creating additional indexes on the stored data in relational databases to reduce resource consumption of the database storage instance. This leads to wasted storage space. When the number of data read / write requests per unit time is large, a higher number of processor cores is required for retrieval and computation, resulting in higher storage costs. Alternatively, related technologies record the location of data through index lists. When the long list of data exhibits sparse characteristics, storage space utilization is low, leading to a significant waste of unnecessary storage resources.

[0080] Based on the above analysis, the applicant found that the data storage methods of related databases cannot simultaneously guarantee data read / write performance and space utilization. In response to the above problems, this application provides a database data storage method that can improve data storage space utilization and data read / write performance.

[0081] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. The following will describe exemplary applications when the electronic device is implemented as a server.

[0082] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the database data storage system 100 provided in the embodiments of this application. In order to support the data storage application of a database, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0083] When the entry shard of the data to be written is an index shard, the server 200 determines the data shard corresponding to the index value of the data to be written from the data shards belonging to that index shard, and writes the data to be written into the data shard corresponding to the index value.

[0084] Taking a document subscription scenario as an example, in response to writing data to be written into a document, the server 200 writes the data to be written into a data shard corresponding to the index value of the data to be written, and sends the new document data after writing to the terminal 400 so that the updated document can be displayed on the human-computer interaction interface 410.

[0085] Taking a social media platform scenario as an example, in response to the addition operation of user data (such as username, post, comment, etc.), the user data to be added is used as the data to be written, and the server 200 is used to write the data to be written in the data shard corresponding to the index value of the data to be written.

[0086] Taking a shopping scenario as an example, the product information to be added is used as the data to be written. Different shards are used to store information about different products, and server 200 is used to write the data to be written in the data shard corresponding to the index value of the data to be written.

[0087] Taking log data management as an example, system logs or object behavior logs are used as data to be written. Different shards are used to store the behavior logs of different objects. Server 200 is used to write the data to be written in the data shard corresponding to the index value of the data to be written.

[0088] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0089] See Figure 2A , Figure 2A This is a schematic diagram of the structure of server 200-1 provided in an embodiment of this application. Server 200-1 is one implementation of the aforementioned server 200 for storing data in a database. Figure 2A The server 200-1 shown includes at least one processor 210, memory 230, and at least one network interface 220. The various components in server 200-1 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general labeled all buses as Bus System 240.

[0090] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0091] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 230 may optionally include one or more storage devices physically located away from the processor 210.

[0092] The memory 230 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 230 described in this application embodiment is intended to include any suitable type of memory.

[0093] In some embodiments, memory 230 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0094] Operating system 231 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0095] The network communication module 232 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0096] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A A data storage device 233 for a database stored in memory 230 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a request receiving module 2331, a first determining module 2332, a second determining module 2333, a first query module 2334, and a data writing module 2335. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0097] See Figure 2B , Figure 2B This is a schematic diagram of the structure of server 200-2 provided in an embodiment of this application. Server 200-2 is one implementation of server 200 described above for querying data in a database. Figure 2BThe server 200-2 shown includes at least one processor 250, memory 270, and at least one network interface 260. The various components in server 200-2 are coupled together via a bus system 280. It is understood that the bus system 280 is used to implement communication between these components. In addition to a data bus, the bus system 280 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2B The general labeled all buses as Bus System 280.

[0098] Processor 250 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor.

[0099] The memory 270 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, and optical disk drives. The memory 270 may optionally include one or more storage devices physically located away from the processor 250.

[0100] The memory 270 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 270 described in this application embodiment is intended to include any suitable type of memory.

[0101] In some embodiments, memory 270 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0102] Operating system 271 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0103] The network communication module 272 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 260, exemplary network interfaces 260 including: Bluetooth, WiFi, and Universal Serial Bus (USB);

[0104] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2B A data query device 273 for a database stored in memory 270 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a data receiving module 2731, a third determining module 2732, a second query module 2733, and a data extraction module 2734. This module is logically structured and can therefore be further subdivided according to its implemented functions. The functions of this module will be described below.

[0105] In some embodiments, the terminal or server can implement the database data storage or database data query method provided in this application embodiment by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as document APPs or instant messaging APPs; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0106] The data storage method and data query method of the database provided in the embodiments of this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0107] See Figure 3A , Figure 3A This is a first flowchart illustrating the database data storage method provided in this application embodiment, with the server as the main component, and combining... Figure 3A The steps shown are explained.

[0108] In some embodiments, the storage structure of the database includes multiple layers, each layer including at least one shard, and the types of shards include index shards and data shards.

[0109] For example, see Figure 5 , Figure 5 This is a schematic diagram of the database storage structure provided in an embodiment of this application. Figure 5In this database, the storage structure comprises five levels, each including at least one shard. For example, the first level includes four shards, whose types from left to right are: data shard, data shard, index shard, and index shard. The index shard of a higher level points to the index shard or data shard of the next lower level, and the index shard or data shard of the next lower level is subordinate to the index shard of the higher level. Data shards are used to store data. For instance, the first index shard from left to right in the first level points to the first and second index shards from left to right in the second level; the first and second index shards from left to right in the second level are subordinate to the first index shard from left to right in the first level.

[0110] In step 101, a data write request is received, wherein the data write request carries the data to be written.

[0111] In some embodiments, the data to be written is data in a document to be written, and the data write request also includes a document identifier of the document to be written.

[0112] For example, in response to a write operation in a document, such as writing "apple", the data to be written is "apple". If the document's document identifier is document A, then the data write request is "apple + document A".

[0113] In step 102, among the fragments of the first level in multiple levels, a first entry fragment for storing the data to be written is determined, wherein the first level is the first level in the top-down order.

[0114] For example, see Figure 5 The first level in a multi-level hierarchy is the first level in the top-down order, for example... Figure 5 The first layer, the first-level fragmentation, is... Figure 5 The first layer consists of four shards, whose types from left to right are: data shard, data shard, index shard, and index shard. Among these four shards, the first entry shard used to store the data to be written is determined.

[0115] In some embodiments, the data write request also carries the document identifier of the document to which the data to be written belongs, see [link to relevant documentation]. Figure 3B , Figure 3B This is a second flowchart illustrating the database data storage method provided in this application embodiment. Figure 3A Step 102, "Determine the first entry fragment for storing the data to be written," can be done through... Figure 3B Steps 1021 to 1024 are implemented, and the details are explained below.

[0116] In step 1021, the part to which the data to be written belongs is determined. The part to which the data to be written belongs is the part of the ownership document. The ownership document includes multiple parts, and a shard is used to store one part. Different parts are assigned different part identifiers in the database.

[0117] Here, different documents are assigned different document identifiers in the database, dividing the document into multiple parts according to any of the following rules: by paragraph, by sentence, or by page number. Different parts within each document correspond to different part identifiers. In response to a write operation to a new part in the document, a corresponding part identifier is assigned to the new part, and a new shard is generated in the database. This new shard is used to store the new part.

[0118] In step 1022, the document identifier of the document to which the document belongs and the partial identifier of the part to which the document belongs are combined into a key.

[0119] For example, the database stores document A, document B, and document C. The document identifier for document A is doc A, the document identifier for document B is doc B, and the document identifier for document C is doc_C. If the document to which the data to be written belongs is document A, and document A includes three parts: part 1, part 2, and part 3, with part 1 identified by part1, part 2 by part2, and part 3 by part3, then if the data to be written belongs to part 1 in document A, the key would be doc_A+part1.

[0120] In step 1023, the value corresponding to the key is queried in the database, where different values ​​represent different shard identifiers.

[0121] In some embodiments, the database stores key-value pairs, each pair including a key and a value corresponding to the key. The key is obtained by combining a document identifier of a document with a part identifier of a portion of the document, and the value corresponding to the key represents an identifier of a different shard used to store a portion of the document in the key.

[0122] Following the example of step 1022 above, if the value corresponding to the key doc_A+part1 is shard1, the value corresponding to the key doc_A+part2 is shard2, and the value corresponding to the key doc_A+part3 is shard3, then the value retrieved is shard1.

[0123] In step 1024, the shard corresponding to the queried value is used as the first entry shard.

[0124] Following the example of step 1023 above, different values ​​correspond to different shards. For example, shard1 corresponds to shard A, shard2 corresponds to shard B, and shard3 corresponds to shard C. The shard A corresponding to the queried value shard1 is taken as the first entry shard.

[0125] This application's embodiments divide data into its owning document and its owning portion within the document, mapping these to different shards to create a clearer data storage structure. Each shard is responsible for storing a specific part of the data, facilitating management and maintenance. Combining the document identifier of the owning document and the part identifier of the owning portion into a key allows queries to quickly locate the shard containing the target data, reducing query time and improving response speed.

[0126] See also Figure 3A In step 103, in response to the first entry fragment being an index fragment, the index value of the data to be written is determined.

[0127] Here, different data shards belonging to the index shard correspond to different index values, and the data shards belonging to the index shard can be located through the index value.

[0128] In some embodiments, see Figure 3C , Figure 3C This is a schematic diagram of the third process of the database data storage method provided in the embodiments of this application. Figure 3A Step 103, "Determine the index value of the data to be written," can be done through... Figure 3C Steps 1031A to 1032A are implemented, and the details are explained below.

[0129] In step 1031A, the hash value of the identifier of the data to be written is determined by a hash function.

[0130] In some embodiments, the identifier of the data to be written can be a randomly assigned unique identifier, or a unique identifier generated based on a combination of key attributes of the data to be written.

[0131] For example, hash functions include Message-Digest Algorithm 5 (MD5), Secure Hash Algorithm 1 (SHA-1), and Secure Hash Algorithm 256 (SHA-256). Taking SHA-256 as an example, a hash calculation is performed on the identifier of the data to be written. Assuming the identifier of the data to be written is `data_id`, and the hash function is H, then the calculated hash value `hash_value = H(data_id)`. This hash value is a fixed-length binary sequence; for example, the hash value calculated by SHA-256 is 256 bits long.

[0132] In step 1032A, the length of the hash value is compressed to obtain the index value corresponding to the data to be written.

[0133] In some embodiments, the length of the hash value can be compressed using any of the following methods to obtain the index value corresponding to the data to be written: modulo compression, hash value truncation, bitwise operation compression, folding, or hash function combination.

[0134] Taking modulo compression as an example, determine the number of data fragments belonging to the first entry fragment; determine the remainder of the ratio of the hash value to the number; and use the remainder as the index value corresponding to the data to be written. If the number of data fragments belonging to the first entry fragment is n, the hash value of the identifier of the data to be written is x, the result of dividing the hash value by the number is x / n = a…b, the ratio is a, and the remainder is b, then the index value corresponding to the data to be written is b.

[0135] Taking bitwise compression as an example, bitwise operations (such as AND and OR operations) are used to reduce the number of bits in the hash value. If the hash value of the identifier of the data to be written is x, and the number of bits in the index value is k, then the index value corresponding to the data to be written can be x(2^k). k -1).

[0136] Taking the hash function combination method as an example, multiple different hash functions are used to hash the identifier of the data to be written, resulting in multiple hash values. These hash values ​​are then combined to obtain the index value. For example, hash function A is used to hash the identifier of the data to be written, resulting in hash value h1. Hash function b is used to hash the identifier of the data to be written, resulting in hash value h2. An XOR operation is performed on h1 and h2 to obtain the index value corresponding to the data to be written.

[0137] This application embodiment, by determining the index value, can more accurately locate the position of the data to be written in the index structure, thereby improving the accuracy of queries. Compressing long hash values ​​into shorter index values ​​can reduce the space required to store index values ​​in the index structure, lowering storage costs. Shorter index values ​​also speed up index lookups, improving data read and write performance.

[0138] In some embodiments, see Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the database data storage method provided in the embodiments of this application. Figure 3A Step 103, "Determine the index value of the data to be written," can also be done through... Figure 3D Steps 1031B to 1033B are implemented, and the details are explained below.

[0139] In step 1031B, the data to be written is vectorized to obtain a data vector.

[0140] In some embodiments, different vectorization algorithms are used for different data types, including numerical data, text data, and image data.

[0141] For example, numerical data is directly represented as a vector. For instance, a single numerical value x can be vectorized into a one-dimensional vector v. → =[x]; If there are multiple numerical features, for example, three numerical features x1, x2, and x3, then it can be vectorized into a three-dimensional vector v. →= [x1, x2, x3]. For text data, methods such as Bag of Words, Term Frequency-Inverse Document Frequency (TF-IDF), and Word Embedding can be used for vectorization. Taking Word Embedding as an example, the text is first segmented into words, and then each word is mapped to a corresponding vector using a pre-trained word vector model (such as Word2Vec or GloVe). Examples include Word toVector (Word2Vec) and GlobalVectors forword representation (GloVe). Finally, these word vectors are combined into a vector representation of the text using methods such as averaging or summing. For example, the text "I like apples" is segmented into "I", "like", and "apple". Assuming the vectors obtained by these words through the word vector model are w1, w2, and w3 respectively, the text vector can be represented as w1 + w2 + w3. For image data, the pixel values ​​of the image can be unfolded into a one-dimensional vector. For example, for an m×n grayscale image, expanding all its pixel values ​​by rows or columns yields a vector of length m×n. For a color image with k color channels (e.g., k=3 in an RGB image), the expanded vector will have a length of m×n×k.

[0142] In step 1032B, the data vector is reduced in dimensionality to obtain the data values.

[0143] In some embodiments, the data vector can be dimensionality reduced in any of the following ways: Principal Component Analysis (PCA), Singular Value Decomposition (SVD), or t-Distributed Stochastic Neighbor Embedding (t-SNE).

[0144] For example, taking principal component analysis (PCA) as an example, we calculate the covariance matrix of the data vector, then solve for the eigenvalues ​​and eigenvectors of the covariance matrix. We select the eigenvectors corresponding to the k largest eigenvalues ​​and project the original data vector onto the subspace spanned by these eigenvectors to obtain the dimensionality-reduced data vector. Assume the original data vector v... → It is n-dimensional, and after dimensionality reduction by PCA, it becomes a k-dimensional vector v. →_reduced(k < n). Taking singular value decomposition as an example, the data matrix (a matrix formed by combining multiple data vectors) is subjected to singular value decomposition to obtain three matrices U, Σ, and V. By retaining the larger singular values in the Σ matrix, the original data matrix is approximately represented as U k Σ k (V k ) T , where U k , Σ k and V k are the truncated matrices, obtaining the data representation after dimensionality reduction. Taking t-distributed stochastic neighbor embedding as an example, the high-dimensional data is mapped into a low-dimensional space while trying to maintain the relative distance relationship between data points. By calculating the similarity between high-dimensional data points, a probability distribution is constructed, and then a matching probability distribution is found in the low-dimensional space, thereby achieving dimensionality reduction.

[0145] In step 1033B, the data value is mapped to the index value of the data to be written.

[0146] Exemplarily, taking linear mapping as an example, if the range of the data value after dimensionality reduction is [a, b], and the expected index value range is [c, d], then the linear mapping formula index = ((d - c) / (b - a)) * ((value - a) + c) can be used, where value is the data value after dimensionality reduction and index is the mapped index value. Taking discretized mapping as an example, the data value after dimensionality reduction is divided into several intervals, and each interval corresponds to an index value. For example, the range of the data value after dimensionality reduction is [0, 1], which is divided into 10 intervals: [0, 0.1), [0.1, 0.2),......, [0.9, 1], corresponding to index values 0, 1,......, 99 respectively, and the corresponding index value is determined according to the interval where the data value after dimensionality reduction is located.

[0147] In the embodiments of the present application, different types of data are uniformly represented in vector form through vectorization, which is convenient for subsequent processing and analysis. The dimensionality reduction operation can extract the main features of the data and remove noise and redundant information. There are often a large number of dimensions in high-dimensional data that have little impact on the final result. Through dimensionality reduction, the key features of the data can be highlighted, making the data more compact and representative, which helps to improve the data processing efficiency. Mapping the data value to an index value can construct an efficient index structure and speed up the data retrieval speed.

[0148] Continue to refer to Figure 3A , in step 104, for the data shards belonging to the first input shard, recursive queries are performed in the order from top to bottom in multiple levels until the first data shard corresponding to the index value is queried.

[0149] Here, for multiple shards that belong to the index shard, the shards at the next level belong to the shards at the previous level.

[0150] In some embodiments, see Figure 3E , Figure 3E This is a schematic diagram of the fifth process of the database data storage method provided in the embodiments of this application. Figure 3A Step 104 can be achieved through Figure 3E Steps 1041 to 1042 are implemented, and the details are explained below.

[0151] Here, let m be an integer variable that increments from 2 to M, where M is the number of levels; iterate m and execute steps 1041 and 1042 below.

[0152] For example, see Figure 5 The number of layers is 5, that is, the value of M is 5. Steps 1041 and 1042 are executed iteratively starting from the 2nd layer and continuing until the 5th layer ends.

[0153] In step 1041, in the data shards belonging to the first entry shard at the m-th level, the first data shard corresponding to the index value is queried.

[0154] Here, the query is performed on the data shards belonging to the first entry shard at level m. First, the set of data shards at level m is obtained; for each data shard at level m, a matching query is performed using the index value. This can be based on the index structure within the shard or by directly traversing and matching the data stored in the shard. For example, a data shard might be a key-value pair storage structure, with the index value used as the key to find the corresponding data record.

[0155] In step 1042, if no result is found, continue querying the (m+1)th level until the first data fragment corresponding to the index value is found.

[0156] Here, if the first data shard corresponding to the index value is found at level m, the query stops, and the query ends successfully. If not found, the value of m is incremented to m+1, and it is checked whether m is greater than M. If m≤M, the query continues in the data shard at level m+1, repeating the above query steps; if m≥M, it means that the target data shard was not found in the entire hierarchical structure, and the query fails.

[0157] This application's embodiments utilize a hierarchical recursive query approach to quickly locate the target data fragment, improving query efficiency. Hierarchical iterative queries using the variable 'm' allow for adjustments to the query hierarchy based on actual conditions. If the data structure changes, such as increasing or decreasing the number of levels, only the value of 'M' needs adjustment, without requiring large-scale modifications to the overall query logic, demonstrating excellent flexibility and scalability.

[0158] In some embodiments, step 1042 "If not found, continue querying the (m+1)th level" includes the following two cases:

[0159] Case 1: If all the shards belonging to the first entry shard at level m are index shards, then it is determined that the first data shard was not found at level m, and the query continues to level m+1.

[0160] For example, see Figure 5 If the first entry shard is the third shard from left to right in the first level, which is also the first index shard from left to right, and m = 2, and the shards in the second level that belong to the first entry shard are all index shards, then it is determined that the first data shard was not found in the second level, and the query continues to the (m+1)th level.

[0161] Case 2: If the type of the fragment belonging to the first entry fragment at level m includes a data fragment, and no first data fragment corresponding to the index value is found, then it is determined that no first data fragment is found at level m, and the query continues at level m+1.

[0162] For example, see Figure 5 If the first entry shard is the third shard from left to right in the first layer, that is, the first index shard from left to right, and m=3, if the index value of the data shard in the third layer is different from the index value of the data to be written, then it is determined that the first data shard was not found in the third layer, and the query continues to the fourth layer.

[0163] This application's embodiments handle two different scenarios separately, accurately determining whether to proceed to the next level of query, avoiding unnecessary query operations and improving query efficiency. It can adapt to complex and diverse data sharding structures, enhancing versatility and adaptability. It ensures a comprehensive query of the entire hierarchical structure under any circumstances, without missing any levels of query due to different sharding types. This guarantees that the first data shard corresponding to the index value can be found as much as possible within the entire data sharding hierarchy, improving the completeness and reliability of the query.

[0164] See also Figure 3A In step 105, the data to be written is written to the first data fragment.

[0165] In some embodiments, see Figure 3F , Figure 3F This is a schematic diagram of the sixth process of the database data storage method provided in this application embodiment. During execution... Figure 3A During any of steps 101 to 105, the following steps may also be performed: Figure 3F Steps 201 to 204 are explained in detail below.

[0166] In step 201, a data query request is received, wherein the data query request carries the index value to be queried.

[0167] Here, data query requests are received via an interface, which can be a network interface such as a Representational State Transfer Application Programming Interface (RESTful API) or a Remote Procedure Call (RPC) interface. This interface is responsible for detecting data query requests. When a data query request is received, the index value to be queried carried in the request is parsed. For example, if it is a RESTful API, the request might be a Hypertext Transfer Protocol (HTTP) GET request, with the index value to be queried carried as a request parameter in the Uniform Resource Locator (URL). The index value to be queried is extracted by parsing the URL.

[0168] In step 202, among the first-level shards in multiple layers, the second entry shard to which the data shard corresponding to the index value to be queried belongs is determined.

[0169] In some embodiments, based on the index value to be queried, a specific lookup algorithm (such as hash value matching, range lookup, etc.) is used to determine the second entry shard to which the data shard corresponding to the index value to be queried belongs within the first-level shards. For example, if the first-level shards are stored using a hash table, and each shard has a hash range, the hash value of the index value to be queried is calculated to determine which shard's hash range the hash value falls into, thereby determining the corresponding second entry shard.

[0170] For example, see Figure 5 The first level in a multi-level hierarchy is the first level in the top-down order, for example... Figure 5 The first layer, the first-level fragmentation, is... Figure 5 The first layer consists of four shards, whose types from left to right are: data shard, data shard, index shard, and index shard. Among these four shards, the second entry shard to which the data shard corresponding to the index value to be queried belongs is determined.

[0171] In step 203, in response to the second entry shard being an index shard, the data shards belonging to the second entry shard are recursively queried in a top-down order according to multiple levels until the second data shard corresponding to the index value to be queried is found, and the data to be queried is extracted from the second data shard.

[0172] Here, for multiple shards that belong to the index shard, the shards at the next level belong to the shards at the previous level.

[0173] In some embodiments, see Figure 3G , Figure 3G This is a schematic diagram of the seventh process of the database data storage method provided in the embodiments of this application. Figure 3F Step 203, "For data shards belonging to the second entry shard, recursively query them in a top-down order according to multiple levels until the second data shard corresponding to the index value to be queried is found," is executed iteratively in step m. Figure 3G Steps 2031 to 2032 are explained in detail below.

[0174] In step 2031, in the data shards belonging to the second entry shard at the m-th level, the second data shard corresponding to the index value to be queried is queried.

[0175] Here, the query is performed on the data shards belonging to the second entry shard at level m. First, the set of data shards at level m is obtained; for each data shard at level m, a matching query is performed using the index value to be queried. This can be based on the index structure within the shard or by directly traversing and matching the data stored in the shard. For example, a data shard might be a key-value pair storage structure, with the index value to be queried serving as the key to find the corresponding data record.

[0176] In step 2032, if no result is found, continue querying the (m+1)th level until the second data fragment corresponding to the index value to be queried is found.

[0177] Here, "If not found, continue querying level m+1" includes the following two cases: If the type of the fragments belonging to the second entry fragment at level m is all index fragments, then it is determined that no second data fragment has been found, and the query continues to level m+1; If the type of the fragments belonging to the second entry fragment at level m includes data fragments, and no second data fragment corresponding to the index value has been found, then it is determined that no second data fragment has been found, and the query continues to level m+1.

[0178] This application's embodiments perform queries hierarchically from top to bottom, searching within a specific level of data shards each time. This progressively narrows the search scope, reduces the amount of data searched, and improves query efficiency. The iterative query method leverages the structural characteristics of data shards to quickly locate the region containing the target data based on hierarchical information, avoiding meaningless global searches and making the query process more efficient and targeted. By iteratively querying from each level of data shards, even in cases of uneven data distribution or complex shard structures, a comprehensive search of the entire hierarchical structure is ensured. This improves the success rate of finding the target data, reduces the possibility of misjudgments and erroneous queries, and guarantees the accuracy of the query results.

[0179] See also Figure 3F In step 204, in response to the second entry shard being a data shard, the data to be queried is extracted from the second entry shard.

[0180] This application's embodiments narrow down the scope of subsequent queries by first determining the second entry shard. In complex multi-level sharding structures, directly locating the entry point containing the target data avoids blind traversal of all shards, improving query efficiency and reducing query time overhead. When the entry shard is of different types, corresponding strategies can be used for flexible handling, adapting to various data sharding strategies and storage structures, improving versatility and scalability, and meeting diverse business needs. The recursive query mechanism ensures a comprehensive search for the target data throughout the entire multi-level structure. Queries are performed in a top-down order, ensuring that no level or shard that might contain the target data is missed, thereby improving the accuracy and completeness of the query and ensuring that the required data is found.

[0181] Here, when reading all the data in any shard, if the shard type is a data shard, then all the data items in that data shard are read; if the shard type is an index shard, then all the data shards belonging to that index shard are queried, and the set of data items belonging to all the data shards belonging to that index shard is taken as all the data under that index shard.

[0182] In some embodiments, see Figure 3H , Figure 3H This is a schematic diagram of the eighth process of the database data storage method provided in this application embodiment. After step 105, for each index shard in the first level, the following steps are performed: Figure 3H Steps 106 to 108 are explained in detail below.

[0183] In step 106, if the index shard and its subordinate shards meet the sparsity condition, the current number of shards included in the index shard is reduced to a preset number.

[0184] Here, the sparsity condition for the index shard and its subordinate shards indicates that the storage space utilization of the shards included in the index shard is less than a utilization threshold. The storage space utilization rate is the ratio of the used space in all shards included in the index shard to the total storable data space in all shards included in the index shard. In other words, the subordinate shards of the index shard have more than a preset proportion of unused storage space. This preset proportion can be set according to current performance and business needs.

[0185] In some embodiments, see Figure 3I , Figure 3I This is a ninth flowchart illustrating the database data storage method provided in this application embodiment. Before "reducing the current number of shards included in the index shard to a preset number", the above-mentioned confirmation process of "the index shard and its subordinate shards satisfying the sparsity condition" can be achieved through... Figure 3I Steps 301 to 302 are implemented, and the details are explained below.

[0186] In step 301, the number of fully loaded shards is determined based on the length of the index shard. The length of the index shard represents the number of data shards belonging to the index shard, and the number of fully loaded shards is the number of shards belonging to the index shard when the index shard is in a fully loaded state. The fully loaded state indicates that the length of the index shard is equal to the length threshold.

[0187] First, the length information of each index shard can be obtained by querying the metadata records of the index shards in the database. The metadata records the length of each index shard, which is the number of data shards belonging to that index shard. Then, a fixed value predetermined for performance considerations is used as a length threshold to determine whether an index shard is in a fully loaded state. The length of the index shard is compared with the length threshold. If the length of the index shard is equal to the length threshold, then the number of data shards belonging to that index shard is the number of fully loaded shards. If there are multiple index shards, the above operation is performed on each index shard separately to count the number of fully loaded shards for all index shards.

[0188] In some embodiments, see Figure 3J , Figure 3J This is a schematic diagram of the tenth process of the database data storage method provided in the embodiments of this application. Figure 3I Step 301, "Determine the number of fully loaded shards based on the length of the index shard," can be achieved through... Figure 3J Steps 3011 to 3012 are implemented, and the details are explained below.

[0189] In step 3011, the number of data shards belonging to the index shard is determined, and the number is used as the length of the index shard.

[0190] For example, see Figure 5 Taking the first index shard from left to right in the first level as an example, if the number of data shards belonging to this index shard is 2, then the length of this index shard is 2.

[0191] In step 3012, the ratio of length to length threshold is determined, and the ratio is used as the number of full-load fragments.

[0192] For example, if the length is 12 and the length threshold is 3, then the ratio is 12 / 3 = 4, which means the number of fully loaded fragments is 4.

[0193] See also Figure 3I In step 302, if the current number of shards is greater than the number of fully loaded shards by a first preset multiple, then the index shard and its subordinate shards are determined to meet the sparsity condition.

[0194] For example, if the current number of shards belonging to the index shard is 10, the first preset multiple is 2, and the number of fully loaded shards is 4, then the number of fully loaded shards is 8. Since the current number of shards is greater than the number of fully loaded shards, it is determined that the index shard and its subordinate shards meet the sparsity condition.

[0195] This application's embodiments quantify and evaluate the sparsity of shards by determining the number of fully loaded shards and judging sparsity conditions. This allows for an accurate understanding of the current sharding structure's usage, avoiding uncertainties caused by subjective judgment and providing reliable data for subsequent decisions. When shards are determined to meet sparsity conditions, the number of shards is reduced, minimizing storage space waste and improving data access efficiency.

[0196] See also Figure 3H In step 107, if the index shard and its subordinate shards meet the dense condition, the current number of shards included in the index shard is increased to a preset number.

[0197] Here, the dense condition of the index shard and its subordinate shards indicates that among the shards included in the index shard, there are shards whose storage length exceeds the length threshold, and there is a data overflow situation in the subordinate shards of the index shard.

[0198] In some embodiments, before "increasing the current number of shards included in the index shard to a preset number", the above-mentioned confirmation process of "the index shard and its subordinate shards satisfying the dense condition" can be implemented by performing the following process: for any data shard belonging to the index shard, if the length of the data shard is greater than the length threshold, it is determined that the index shard and its subordinate shards satisfy the dense condition, wherein the length of the data shard represents the number of data included in the data shard.

[0199] For example, for any data shard belonging to the index shard, if the number of data included in the data shard is 8, that is, the length of the data shard is 8, and the length threshold is 5, then the index shard and its subordinate shards are determined to meet the dense condition.

[0200] See also Figure 3H In step 108, the data in the data shards belonging to the index shards are restored according to a preset number.

[0201] Here, the preset quantity is the number of shards belonging to the index shard that is pre-set based on business needs and performance conditions. The data in the data shards belonging to the index shard will be re-stored according to the preset quantity.

[0202] In some embodiments, see Figure 3K , Figure 3K This is a schematic diagram of the eleventh step of the database data storage method provided in the embodiments of this application. Figure 3H Step 108 can be achieved through Figure 3K Steps 1081 to 1083 are implemented, and the details are explained below.

[0203] In step 1081, an index value is assigned to each data shard belonging to the index shard according to a preset number.

[0204] For example, for each data shard belonging to an index shard, a unique index value is assigned to it according to a preset number range, either sequentially or according to a specific rule. For instance, if the preset number range is from 1 to n, index values ​​1, 2, ..., n can be assigned to each data shard in sequence, or index values ​​can be assigned using more complex mapping rules based on some characteristic of the data shard (such as creation time, storage location, etc.).

[0205] In step 1082, each piece of data in each data shard is taken as data to be allocated, the hash value of the data to be allocated is determined by a hash function, and the hash value is mapped to the index value of the data to be allocated.

[0206] In some embodiments, the identifier of the data to be written can be a randomly assigned unique identifier, or a unique identifier generated based on a combination of key attributes of the data to be written. The number of data fragments belonging to the first entry fragment is determined; the remainder of the ratio of the hash value to the number of fragments is determined; and the remainder is used as the index value corresponding to the data to be written.

[0207] For example, hash functions include Message-Digest Algorithm 5 (MD5), Secure Hash Algorithm 1 (SHA-1), and Secure Hash Algorithm 256 (SHA-256). Taking SHA-256 as an example, a hash calculation is performed on the identifier of the data to be written. Assuming the identifier of the data to be written is data_id, and the hash function is H, then the calculated hash value hash_value = H(data_id). This hash value is a fixed-length binary sequence; for example, the hash value calculated by SHA-256 is 256 bits long. If the number of data fragments belonging to the first entry fragment is n, and the hash value of the identifier of the data to be written is x, the result of dividing the hash value by the number is x / n = a…b, the ratio is a, and the remainder is b. Therefore, the index value corresponding to the data to be written is b.

[0208] In step 1083, the data shard with the same index value as the data to be allocated is queried and used as the data shard to be written, and the data to be allocated is written into the data shard to be written.

[0209] Here, based on the index value of the data to be allocated, the index values ​​of all data shards are traversed, or the location is determined by the index structure, such as a hash table, where the key is the index value and the value is the corresponding data shard. Data shards with the same index value are found and used as the data shards to be written to. The data to be allocated is then written to these shards. The write operation depends on the storage structure of the data shards, such as appending records to the database or inserting new rows into the database table.

[0210] This application's embodiments map data to index values ​​using a hash function. Leveraging the characteristics of hash functions, different input data are evenly mapped into the hash space, preventing data concentration in a few fragments and improving data storage balance. Querying using index values ​​significantly reduces the search scope, improving data writing efficiency compared to traversing all data fragments to determine the write location.

[0211] See Figure 4A , Figure 4A This is a first flowchart illustrating the database data query method provided in this application embodiment, with the server as the main component, and combining... Figure 4A The steps shown are explained.

[0212] In step 401, a data query request is received, wherein the data query request carries the index value to be queried.

[0213] Here, data query requests are received via an interface, which can be a network interface such as a RESTful API or RPC interface. This interface is responsible for detecting data query requests. When a data query request is received, the index value to be queried carried in the request is parsed. For example, if it is a RESTful API, the request might be an HTTP GET request, and the index value to be queried is carried as a request parameter in the URL. The index value to be queried is extracted by parsing the URL.

[0214] In step 402, among the first-level shards in multiple levels, the entry shard to which the data shard corresponding to the index value to be queried belongs is determined, wherein the first level is the first level in the top-down order.

[0215] In some embodiments, based on the index value to be queried, a specific lookup algorithm (such as hash value matching, range lookup, etc.) is used to determine the entry shard to which the data shard corresponding to the index value to be queried belongs in the first-level shards. For example, if the first-level shards are stored using a hash table, and each shard has a hash range, the hash value of the index value to be queried is calculated to determine which shard's hash range the hash value falls into, thereby determining the corresponding entry shard.

[0216] For example, see Figure 5 The first level in a multi-level hierarchy is the first level in the top-down order, for example... Figure 5 The first layer, the first-level fragmentation, is... Figure 5 The first layer consists of four shards, whose types from left to right are: data shard, data shard, index shard, and index shard. Among these four shards, the entry shard to which the data shard corresponding to the index value to be queried belongs is determined.

[0217] In step 403, in response to the entry shard being an index shard, the data shards belonging to the entry shard are recursively queried in a top-down order according to multiple levels until the target data shard corresponding to the index value to be queried is found.

[0218] Here, for multiple shards that belong to the index shard, the shards at the next level belong to the shards at the previous level.

[0219] In some embodiments, see Figure 4B , Figure 4B This is a schematic diagram of the second process of the database data query method provided in the embodiments of this application. Figure 4A Step 403, "For data shards belonging to the entry shard, recursively query them in a top-down order across multiple levels until the target data shard corresponding to the index value to be queried is found," can be achieved through... Figure 4BSteps 4031 to 4032 are implemented, and the details are explained below.

[0220] Here, let m be an integer variable that increments from 2 to M, where M is the number of levels; iterate m and execute steps 4031 and 4032 below.

[0221] In step 4031, in the data shards belonging to the entry shard at the m-th level, the target data shard corresponding to the index value to be queried is queried.

[0222] Here, the query is performed on the data shards belonging to the entry shard at level m. First, the set of data shards at level m is obtained; for each data shard at level m, a matching query is performed using the index value to be queried. This can be based on the index structure within the shard or by directly traversing and matching the data stored in the shard. For example, a data shard might be a key-value pair storage structure, with the index value to be queried serving as the key to find the corresponding data record.

[0223] In step 4032, if no result is found, continue querying the (m+1)th level until the target data shard corresponding to the index value to be queried is found.

[0224] In some embodiments, "If not found, continue querying level m+1" includes the following two cases: If the type of the shards belonging to the entry shard at level m is all index shards, then it is determined that no target data shard has been found, and the query continues at level m+1; If the type of the shards belonging to the entry shard at level m includes data shards, and no target data shard corresponding to the index value to be queried has been found, then it is determined that no target data shard has been found, and the query continues at level m+1.

[0225] For example, see Figure 5 If the entry shard is the third shard from left to right in the first level (i.e., the first index shard from left to right), and m = 2, then all shards in the second level that belong to the entry shard are index shards. Therefore, it is determined that the target data shard was not found, and the query continues to the (m+1)th level. If m = 3, and the index value of the data shard in the third level is different from the index value to be queried, then it is determined that the target data shard was not found, and the query continues to the fourth level.

[0226] See also Figure 4A In step 404, the data to be queried is extracted from the target data fragment.

[0227] In some embodiments, data matching and extraction are performed in the target data shards based on relevant identifiers of the data to be queried (such as primary key values, specific attribute values, etc.). Different data extraction methods are used for different types of database structures to extract the data to be queried from the target data shards.

[0228] For example, in a relational database table structure, data is extracted from corresponding rows and columns using Structured Query Language (SQL) queries (such as SELECT statements) based on specified conditions (such as conditions in the WHERE clause). For key-value pair structures, the key is used to directly retrieve the corresponding value. If the key is generated based on an attribute of the data being queried, the corresponding value is directly found in the key-value pair set using that key. In document-oriented structures, the document hierarchy is traversed, and the matching data is found based on the document path (similar to a file path used to locate a specific position within the document). For instance, for a JSON-formatted user document, to retrieve the order list for user 123, it might be necessary to first find the document object with user ID 123, and then extract the order list data from the `orders` field of that object.

[0229] This application's embodiments directly locate the entry shard by index value, avoiding a full scan of all shards and improving query efficiency. Iterative queries follow a hierarchical order, searching within specific level shards each time, gradually narrowing the search space and reducing query time complexity. Precise matching of index values ​​within each level shard ensures the accurate target data shard corresponding to the query index value is found, reducing the possibility of misjudgments and erroneous queries, guaranteeing the accuracy and completeness of query results, and thus improving the success rate of retrieving target data.

[0230] The following will describe an exemplary application of the embodiments of this application in an online document APP application scenario.

[0231] The embodiments of this application are not limited to data storage or data retrieval in online document APP scenarios, but are also applicable in other scenarios, such as online document mini-program scenarios in instant messaging APPs or browser scenarios that can display and process online documents.

[0232] When a user has permission to access an online document, they can choose to subscribe to events related to changes in that document's content. When a user subscribes to an online document and its content changes, all subscribed users will receive a notification. In this scenario, a single document allows an unlimited number of users to subscribe; therefore, it's necessary to store an infinitely long list and provide high-performance list retrieval capabilities to push notifications to all subscribed members when document content changes.

[0233] Related technologies typically design the storage fields of relational database tables based on the data storage content. When new data is added, corresponding values ​​are filled in, and when data is retrieved, SQL retrieval syntax is used to query datasets that meet the conditions. In relational databases, data is stored in tabular form, thus allowing for the storage of lists of unlimited length. In business scenarios with high requirements for data read / write speed and storage availability, additional indexes (such as B-trees, B+ trees, etc.) are usually needed to ensure data read / write efficiency, resulting in higher additional storage space consumption. Furthermore, when the volume of data read / write requests per unit time is large, to ensure normal business operation, a higher number of Central Processing Unit (CPU) cores are usually required for retrieval and computation, further increasing storage costs.

[0234] Related technologies also utilize non-relational databases for data storage and retrieval. In non-relational database storage systems, the storage value corresponding to a single key typically has a space limit. Therefore, when storing unlimited long lists, the list needs to be sharded to store the index information and data details separately. Specifically, the list data consists of two parts: data shards and a list of data shard indexes. When writing new data, a hash function is first used to calculate the index corresponding to the data, then the corresponding data shard is located using the index value, and the new data is inserted into the data shard pointed to by that index. When reading new data, a hash function is used to calculate the index value, then the corresponding data shard is found based on the index value, and the data is retrieved. Because unlimited long lists in non-relational database storage need to store both index information and data shards simultaneously, with the index list recording the location of the data shards, when the long list data exhibits sparse characteristics, the storage space occupied by the index shards is relatively small. This means that the storage space utilization rate is low in this scenario, leading to a significant waste of unnecessary storage resources.

[0235] This application's embodiments utilize the non-relational database Redis to store infinitely long lists, avoiding the issue of relational databases consuming more CPU resources in high-concurrency scenarios. Simultaneously, Redis's memory access characteristics ensure high stability and efficiency for reading and writing long list data. Furthermore, through a novel sharded data structure, index data and list data can be stored simultaneously. In sparse long list storage scenarios, this avoids the space waste caused by storing additional index data for the vast majority of lists of moderate length, significantly improving storage space utilization. The sparser the long list data, the higher the space utilization of the stored data.

[0236] Refer to Table 1, which illustrates the data structure of composite shards. Each shard uses an enumeration value (shard_type) to identify whether the data stored in the shard is an index list or a data list. Refer to Table 2, which illustrates the enumeration values ​​for the shard types of composite shards. For data-type shards, their data list corresponds to all data under that key value; for index-type shards, their data list corresponds to the index key (Key) of all their sub-shards, and all data under that key value is the union of the data lists of all data-type sub-shards.

[0237] Table 1

[0238]

[0239] Table 2

[0240] name Enumeration value illustrate ShardType_1NVAILD 0 Violation of fragmentation type ShardType_DATA 1 Data Sharding ShardType_INDEX 2 Indexed sharding

[0241] See Figure 6A , Figure 6A This is a schematic diagram illustrating the process of reading long list data provided in an embodiment of this application. Figure 6A In this process, the list storage key is calculated as follows: The document identifier (ID) is obtained and prefixed to serve as the storage key. For example, in a content subscription scenario, if the document type is a table, with multiple sub-tables and multiple regions, each region has a unique ID (partial identifier), when storing the list entry shard, the document ID and region ID are concatenated to serve as the storage key for that entry shard (the key mentioned above). The entry shard is read from Redis storage, and its type is determined. If the entry shard is a data type, the list is a short list, and the Item field contains all the data items in that list. If the entry shard is an index type, the list is a long list, and the Item field contains the index key of the sub-shard. All shards under the index key are then recursively read until all corresponding data shards are found. The union of the Item storage items of all data shards represents all the data in the long list.

[0242] See Figure 6B , Figure 6BThis is a schematic diagram of the single-item data reading process provided in the embodiments of this application. When retrieving single-item data (i.e., the data to be queried), the storage key of the entry shard of the list is first calculated to obtain its entry shard (i.e., the second entry shard). If the entry shard is a data shard, the list is a short list, and the target data item can be retrieved in the Item field of the entry shard. If the entry shard is an index shard, the list is a long list. The Hash function is used to calculate the sub-shard index where the user is located, and the storage key value of the sub-shard is taken out from the Item data item of the entry shard. The data shard where the data item is located is searched recursively downwards. Specifically: if the sub-shard is still an index shard, the sub-shards of the sub-shard are queried until the sub-shard is a data shard. The target record (i.e., the data to be queried) is queried in the Item item of the final data shard (i.e., the second data shard).

[0243] See Figure 6C , Figure 6C This is a schematic diagram of the process for writing single-item data according to an embodiment of this application. When writing single-item data (i.e., data to be written), the storage key of the entry shard of the list is first calculated to obtain its entry shard (i.e., the first entry shard). If the entry shard is a data shard, the list is a short list, and the target data can be written to the Item list of the entry shard. If the entry shard is an index shard, the list is a long list. A hash function is used to calculate the sub-shard index (i.e., the index value) where the user is located, and the storage key value of the sub-shard is taken from the Item data item of the entry shard. The data shard where the data item is located is searched recursively downwards, and the target record (i.e., the data to be written) is inserted into the Item item of the final data shard (i.e., the first data shard). To ensure the stability of the read and write performance of the storage instance and reduce unnecessary CPU consumption, when inserting long list data, no shard rebalancing judgment is performed, and the data is written directly.

[0244] For each storage fragment, the balance of the storage fragment can be calculated using formulas (1) and (2):

[0245]

[0246] Balance Intense =Sum items ≥Limit (2)

[0247] In formula (1), Balance Sparse Sum indicates that the balance is sparse. items This indicates the length of the Item stored in this shard (i.e., the length of the index shard), while Limit indicates the maximum length of an Item stored in a single shard (i.e., the length threshold). This represents the ideal number of fragments when the fragment is under full load; Sumshards This indicates the current number of shards (i.e., the current quantity). Therefore, when the ideal full-load shard count is less than or equal to 1 / 2 of the current shard count, the current long list storage shards are considered to be in a sparse state (i.e., meet the sparse condition), the space utilization is less than 50%, and rebalancing is required.

[0248] In formula (2), Balance Intense Sum indicates that the balance is sparse. items This indicates the length of the Item stored in the shard, and Limit indicates the upper limit of the length of the Item stored in a single shard. When the total number of item data stored in a single shard is greater than the specified upper limit of the number of item data in a single shard, the storage shard is considered to be in a dense state (that is, it meets the dense condition) and needs to be rebalanced.

[0249] Rebalancing is performed during full list data reads (at this time, the full list data result can be reused, eliminating the need for additional full list data reads and avoiding extra overhead). During insertions or deletions, data is written directly without checking shard load. When rebalancing is deemed necessary, a new number of shards (i.e., the preset number) is allocated based on 1.5 times the number of fully loaded shards. This rebalancing method minimizes the problem of frequent shard rebalancing caused by uneven data distribution due to storage writes or deletions, thus reducing the CPU resource consumption of the storage instance. This improves the read / write throughput performance of the storage instance and enhances service stability.

[0250] See Figure 6D , Figure 6D This is a schematic diagram of the data rebalancing process provided in this application embodiment. When reading all data in the long list, it is determined whether rebalancing is needed for each of its shards. When any storage shard is determined to be a sparse shard or a dense shard, the entire long list is rebalanced. Specifically: based on the new number of sub-shards (i.e., a preset number), the key of each sub-shard is determined, for example, a doc prefix + index value (1, 2, 3...). The data of the current long list is hashed into multiple sub-shards according to a hash function, and each data is associated with the key of a sub-shard. At this point, all data in all sub-shards is filled, and all sub-shards are rewritten back to storage, completing the rebalancing process.

[0251] During the rebalancing process, the number of new fragments after rebalancing is first calculated. The number of new fragments can be calculated using the following formula (3):

[0252]

[0253] Among them, Num shard Sum represents the new number of shards. itemThis indicates the length of the Item stored in this shard, and Limit indicates the maximum length of an Item stored in a single shard. This represents the ideal number of shards under full load conditions (i.e., the number of fully loaded shards). The final number of shards is 1.5 times the ideal number of shards under full load conditions, rounded up. The 1.5 times rounding up strategy ensures that the data storage space utilization of the new sharding structure remains between 50% and 100%. This minimizes the problems of dense sharding caused by inserting new data into the list after rebalancing or sparse sharding caused by deleting data, reducing the frequency of rebalancing and improving storage read / write performance and stability.

[0254] In the document content subscription feature, each document corresponds to a subscription list. The number of subscribers varies for different documents. For some highly popular documents, there are long subscription lists. In this scenario, the data sparsity is the percentage of highly popular documents with long subscription lists out of all subscribed documents. See also Figure 7 , Figure 7 This is a schematic diagram illustrating the relationship between the proportion of different long lists and storage space utilization provided in the embodiments of this application. According to... Figure 7 It can be seen that the sparser the data in a long list, the higher the storage space utilization.

[0255] Furthermore, for unsharded lists (i.e., lists where the entry shard is the same as the data shard), the time complexity for data read and write is O(1), while for sharded long lists (i.e., lists where the entry shard is the same as the index shard), the time complexity for data read and write is O(logN). When performing long list data read and write using the database data storage method provided in this application embodiment, higher data sparsity (i.e., a smaller proportion of long lists) results in lower read and write time complexity and higher read and write performance; conversely, lower data sparsity (i.e., a larger proportion of long lists) results in higher space utilization.

[0256] This application embodiment divides data into its owning document and its owning portion within the document, mapping them to different shards to create a clearer data storage structure. Each shard is responsible for storing a specific part of the data, facilitating management and maintenance. Combining the document identifier of the owning document and the part identifier of the owning portion into a key allows queries to quickly locate the shard containing the target data, reducing query time and improving response speed. Determining the index value allows for more precise location of the data to be written within the index structure, improving query accuracy. Compressing long hash values ​​into shorter index values ​​reduces the space required to store index values ​​in the index structure, lowering storage costs. Shorter index values ​​also speed up index lookups, improving data read / write performance. Hierarchical recursive queries quickly locate the target data shard, improving query efficiency. Hierarchical iterative queries using the variable 'm' allow for adjustments to the query hierarchy based on actual conditions. If the data structure changes, such as increasing or decreasing the number of levels, only the value of 'M' needs adjustment, without requiring large-scale modifications to the overall query logic, demonstrating good flexibility and scalability.

[0257] The following description continues to illustrate the exemplary structure of the data storage device 233 of the database provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2A As shown, the software modules in the data storage device 233 storing the database in the memory 230 may include:

[0258] The request receiving module 2331 is used to receive data write requests, wherein the data write request carries the data to be written.

[0259] The first determining module 2332 is used to determine the first entry fragment for storing the data to be written in the first level fragments of multiple levels, wherein the first level is the first level in the top-down order.

[0260] The second determining module 2333 is used to determine the index value of the data to be written in response to the first entry fragment being an index fragment.

[0261] The first query module 2334 is used to recursively query the data fragments belonging to the first entry fragment in a top-down order of multiple levels until the first data fragment corresponding to the index value is found.

[0262] The data writing module 2335 is used to write the data to be written to the first data segment.

[0263] In some embodiments, the data write request also carries the document identifier of the document to which the data to be written belongs. The first determining module 2332 is further configured to determine the part to which the data to be written belongs, wherein the part to which the data to be written belongs is the part of the document to which the data to be written belongs. The document to which the data belongs includes multiple parts, and a shard is used to store a part. Different parts are assigned different part identifiers in the database. The document identifier of the document to which the data belongs and the part identifier of the part to which the data belongs are combined into a key. The value corresponding to the key is queried in the database, wherein different values ​​represent the identifiers of different shards. The shard corresponding to the queried value is used as the first entry shard.

[0264] In some embodiments, the second determining module 2333 is further configured to determine the hash value of the identifier of the data to be written by a hash function; and to compress the length of the hash value to obtain the index value corresponding to the data to be written.

[0265] In some embodiments, the second determining module 2333 is further configured to vectorize the data to be written to obtain a data vector; reduce the dimensionality of the data vector to obtain a data value; and map the data value to the index value of the data to be written.

[0266] In some embodiments, the first query module 2334 is further configured to let m be an integer variable increasing from 2 to M, where M is the number of multiple levels; iterate m and perform the following processing: in the data fragments belonging to the first entry fragment at the m-th level, query the first data fragment corresponding to the index value; if not found, continue to query the (m+1)-th level until the first data fragment corresponding to the index value is found.

[0267] In some embodiments, the first query module 2334 is further configured to: if the type of the fragments belonging to the first entry fragment at the m-th level are all index fragments, then determine that no first data fragment was found at the m-th level and continue querying the (m+1)-th level; if the type of the fragments belonging to the first entry fragment at the m-th level includes data fragments and no first data fragment corresponding to the index value is found, then determine that no first data fragment was found at the m-th level and continue querying the (m+1)-th level.

[0268] In some embodiments, the first query module 2334 is further configured to receive a data query request, wherein the data query request carries an index value to be queried; determine the second entry shard to which the data shard corresponding to the index value to be queried belongs in a first-level sharding of multiple levels; in response to the second entry shard being an index shard, recursively query the data shards belonging to the second entry shard in a top-down order of multiple levels until the second data shard corresponding to the index value to be queried is found, and extract the data to be queried from the second data shard; in response to the second entry shard being a data shard, extract the data to be queried from the second entry shard.

[0269] In some embodiments, the first query module 2334 is further configured to iterate m and perform the following process: in the data shards belonging to the second entry shard at the m-th level, query the second data shard corresponding to the index value to be queried; if not found, continue querying the (m+1)-th level until the second data shard corresponding to the index value to be queried is found.

[0270] In some embodiments, the data writing module 2335 is further configured to reduce the current number of fragments included in the index fragment to a preset number when the index fragment and its subordinate fragments meet the sparsity condition; increase the current number of fragments included in the index fragment to a preset number when the index fragment and its subordinate fragments meet the dense condition; and restore the data in the data fragments belonging to the index fragment according to the preset number.

[0271] In some embodiments, the data writing module 2335 is further configured to determine the number of fully loaded shards based on the length of the index shard, wherein the length of the index shard represents the number of data shards belonging to the index shard, the number of fully loaded shards is the number of shards belonging to the index shard when the index shard is in a fully loaded state, and the fully loaded state represents that the length of the index shard is equal to the length threshold; if the current number of shards is greater than the number of fully loaded shards by a first preset multiple, then it is determined that the index shard and its subordinate shards meet the sparsity condition.

[0272] In some embodiments, the data writing module 2335 is further configured to determine the number of data fragments belonging to the index fragment, and use the number as the length of the index fragment; determine the ratio of the length to the length threshold, and use the ratio as the number of full-load fragments.

[0273] In some embodiments, the data writing module 2335 is further configured to determine that the index shard and its subordinate shards meet the dense condition if the length of the data shard is greater than a length threshold for any data shard belonging to the index shard, wherein the length of the data shard represents the number of data included in the data shard.

[0274] In some embodiments, the data writing module 2335 is further configured to allocate an index value to each data shard belonging to the index shard according to a preset number; take each piece of data in each data shard as the data to be allocated, determine the hash value of the data to be allocated by a hash function, and map the hash value to the index value of the data to be allocated; query the data shard that has the same index value as the data to be allocated, and use it as the data shard to be written, and write the data to be allocated into the data shard to be written.

[0275] The following description continues to illustrate the exemplary structure of the database query device 273 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2AAs shown, the software module in the data query device 273 of the database stored in memory 270 may include:

[0276] The data receiving module 2731 is used to receive data query requests, wherein the data query request carries the index value to be queried.

[0277] The third determining module 2732 is used to determine the entry shard to which the data shard corresponding to the index value to be queried belongs in the first-level sharding of multiple levels, wherein the first level is the first level in the top-down order.

[0278] The second query module 2733 is used to respond to the entry shard being an index shard, and to recursively query the data shards belonging to the entry shard in a top-down order of multiple levels until the target data shard corresponding to the index value to be queried is found.

[0279] The data extraction module 2734 is used to extract the data to be queried from the target data shards.

[0280] In some embodiments, the second query module 2733 is further configured to let m be an integer variable increasing from 2 to M, where M is the number of multiple levels; iterate m and perform the following processing: in the data fragments belonging to the entry fragment at the m-th level, query the target data fragment corresponding to the index value to be queried; if not found, continue to query the (m+1)-th level until the target data fragment corresponding to the index value to be queried is found.

[0281] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the database data storage method or database data query method described in this application embodiment.

[0282] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the database data storage method or the database data query method provided in this application. For example, ... Figure 3A The database data storage methods shown are as follows: Figure 4A The database query method is shown.

[0283] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0284] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0285] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0286] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0287] In summary, through the embodiments of this application, by dividing data according to the belonging document and the belonging part within the belonging document, and mapping it to different shards, the data storage structure becomes clearer. Each shard is responsible for storing a specific part of the data, facilitating management and maintenance. Combining the document identifier of the belonging document and the part identifier of the belonging part into a key allows queries to quickly locate the shard containing the target data, reducing query time and improving response speed. By determining the index value, the position of the data to be written in the index structure can be located more accurately, improving query accuracy. Compressing long hash values ​​into shorter index values ​​reduces the space required to store index values ​​in the index structure, lowering storage costs. Shorter index values ​​also speed up index lookups, improving data read and write performance. Through hierarchical recursive queries, the target data shard can be quickly located, improving query efficiency. Hierarchical iterative queries using the variable m allow adjustment of the query hierarchy based on actual conditions. If the data structure changes, such as increasing or decreasing the number of levels, only the value of M needs to be adjusted, without requiring large-scale modifications to the overall query logic, demonstrating good flexibility and scalability.

[0288] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A database data storage method, characterized in that, The storage structure of the database includes multiple layers, and each layer includes at least one shard. The types of shards include index shards and data shards. The method includes: Receive a data write request, wherein the data write request carries the data to be written; In the fragments of the first level of the plurality of levels, a first entry fragment for storing the data to be written is determined, wherein the first level is the first level in the top-down order; In response to the fact that the type of the first entry shard is the index shard, the index value of the data to be written is determined; For data fragments belonging to the first entry fragment, recursively query according to the multiple levels in a top-down order until the first data fragment corresponding to the index value is found. The data to be written is written to the first data fragment.

2. The method according to claim 1, characterized in that, The recursive querying of data shards belonging to the first entry shard, in a top-down order according to the multiple levels, until the first data shard corresponding to the index value is found, includes: Let m be an integer variable that increases from 2 to M, where M is the number of the multiple levels; Iteration m performs the following processing: In the data shards belonging to the first entry shard at the m-th level, query the first data shard corresponding to the index value; If not found, continue querying the (m+1)th level until the first data fragment corresponding to the index value is found.

3. The method according to claim 2, characterized in that, If not found, continue querying the (m+1)th level, including: If all the fragments belonging to the first entry fragment at the m-th level are of the type of the index fragment, then it is determined that the first data fragment was not found at the m-th level, and the query continues at the (m+1)-th level. If the data shard type of the shard belonging to the first entry shard at the m-th level includes the data shard, and no first data shard corresponding to the index value is found, then it is determined that no first data shard is found at the m-th level, and the query continues at the (m+1)-th level.

4. The method according to claim 1, characterized in that, The data write request also carries the document identifier of the document to which the data to be written belongs; The determination of the first entry fragment for storing the data to be written includes: The belonging portion of the data to be written is determined, wherein the belonging portion is the portion in the belonging document where the data to be written is located, the belonging document includes multiple portions, one shard is used to store one portion, and different portions are assigned different portion identifiers in the database; Combine the document identifier of the document to which it belongs and the partial identifier of the part to which it belongs into a key; Query the value corresponding to the key in the database, wherein different values ​​represent different identifiers of the shard; The shard corresponding to the queried value is used as the first entry shard.

5. The method according to claim 1, characterized in that, Determining the index value of the data to be written includes: The hash value of the identifier of the data to be written is determined by a hash function; The length of the hash value is compressed to obtain the index value corresponding to the data to be written.

6. The method according to claim 1, characterized in that, Determining the index value of the data to be written includes: The data to be written is vectorized to obtain a data vector; The data vector is reduced in dimensionality to obtain the data values; The data value is mapped to the index value of the data to be written.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Receive a data query request, wherein the data query request carries the index value to be queried; In the first level of the multiple levels of the shards, determine the second entry shard to which the data shard corresponding to the index value to be queried belongs; In response to the second entry shard being an index shard, the data shards belonging to the second entry shard are recursively queried in a top-down order according to the multiple levels until the second data shard corresponding to the index value to be queried is found, and the data to be queried is extracted from the second data shard. In response to the second entry shard being of the type of the data shard, the data to be queried is extracted from the second entry shard.

8. The method according to claim 7, characterized in that, The process of recursively querying the data shards belonging to the second entry shard according to the multiple levels in a top-down order until the second data shard corresponding to the index value to be queried is found includes: Iteration m performs the following processing: In the data shards belonging to the second entry shard at the m-th level, query the second data shard corresponding to the index value to be queried; If no result is found, continue querying the (m+1)th level until the second data fragment corresponding to the index value to be queried is found.

9. The method according to any one of claims 1 to 6, characterized in that, After writing the data to be written to the first data fragment, the method further includes: For each index shard in the first level, the following processing is performed: If the index shard and its subordinate shards satisfy the sparsity condition, the current number of shards included in the index shard is reduced to a preset number. If the index shard and its subordinate shards meet the dense condition, the current number of shards included in the index shard is increased to the preset number; Data belonging to the index shard is restored according to the preset number.

10. The method according to claim 9, characterized in that, The method further includes: The number of fully loaded shards is determined based on the length of the index shard, wherein the length of the index shard represents the number of data shards belonging to the index shard, and the number of fully loaded shards is the number of shards belonging to the index shard when the index shard is in a fully loaded state, and the fully loaded state represents that the length of the index shard is equal to the length threshold. If the current number of shards is greater than the number of fully loaded shards by a first preset multiple, then the index shard and its subordinate shards are determined to satisfy the sparsity condition.

11. The method according to claim 10, characterized in that, The step of determining the number of fully loaded shards based on the length of the index shard includes: Determine the number of data shards belonging to the index shard, and use the number as the length of the index shard; Determine the ratio of the length to the length threshold, and use the ratio as the number of full-load fragments.

12. The method according to claim 9, characterized in that, The method further includes: For any data shard belonging to the index shard, if the length of the data shard is greater than the length threshold, then the index shard and its subordinate shards are determined to satisfy the dense condition, wherein the length of the data shard represents the number of data included in the data shard.

13. The method according to claim 9, characterized in that, The step of re-storing data belonging to the index shard according to the preset number includes: Assign an index value to each data shard belonging to the index shard according to the preset number; Each piece of data in each data segment is taken as data to be allocated. The hash value of the data to be allocated is determined by a hash function, and the hash value is mapped to the index value of the data to be allocated. Query the data shard that has the same index value as the data to be allocated, and use it as the data shard to be written. Write the data to be allocated into the data shard to be written.

14. A data query method for a database, characterized in that, The database storage structure includes multiple layers, each layer including at least one shard, and the shard types include index shards and data shards; the method includes: Receive a data query request, wherein the data query request carries the index value to be queried; In the first level of the multiple levels of the shards, the entry shard to which the data shard corresponding to the index value to be queried belongs is determined, wherein the first level is the first level in the top-down order; In response to the entry shard being an index shard, the data shards belonging to the entry shard are recursively queried in a top-down order according to the multiple levels until the target data shard corresponding to the index value to be queried is found. Extract the data to be queried from the target data fragments.

15. The method according to claim 14, characterized in that, The recursive querying of data shards belonging to the entry shard, in a top-down order according to the multiple levels, until the target data shard corresponding to the query index value is found, includes: Let m be an integer variable that increases from 2 to M, where M is the number of the multiple levels; Iteration m performs the following processing: In the data shards belonging to the entry shard at the m-th level, query the target data shard corresponding to the index value to be queried; If no result is found, continue querying the (m+1)th level until the target data fragment corresponding to the index value to be queried is found.

16. A data storage device for a database, characterized in that, The database storage structure includes multiple layers, each layer including at least one shard, and the shard types include index shards and data shards. The apparatus includes: A request receiving module is used to receive a data write request, wherein the data write request carries data to be written; A first determining module is configured to determine, among the fragments of the first level of the plurality of levels, a first entry fragment for storing the data to be written, wherein the first level is the first level in a top-down order; The second determining module is used to determine the index value of the data to be written in response to the first entry fragment being the index fragment. The first query module is used to recursively query the data fragments belonging to the first entry fragment in a top-down order of the multiple levels until the first data fragment corresponding to the index value is found. The data writing module is used to write the data to be written into the first data segment.

17. A data query device for a database, characterized in that, The database storage structure includes multiple layers, each layer including at least one shard, and the shard types include index shards and data shards; the apparatus includes: A data receiving module is used to receive data query requests, wherein the data query request carries the index value to be queried; The third determining module is used to determine the entry shard to which the data shard corresponding to the index value to be queried belongs in the shards of the first level of the multiple levels, wherein the first level is the first level in the top-down order; The second query module is used to respond to the fact that the type of the entry shard is the index shard, and to recursively query the data shards belonging to the entry shard in the order of the multiple levels from top to bottom, until the target data shard corresponding to the index value to be queried is found. The data extraction module is used to extract the data to be queried from the target data fragments.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data storage method of the database according to any one of claims 1 to 13, or implements the data query method of the database according to any one of claims 14 to 15.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data storage method of the database according to any one of claims 1 to 13, or the data query method of the database according to any one of claims 14 to 15.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data storage method of the database according to any one of claims 1 to 13, or the data query method of the database according to any one of claims 14 to 15.

Images