Data writing method and device, electronic equipment and computer storage medium
By sending data write requests directly to the target nodes of the Elasticsearch cluster from the client, the problem of high network overhead in existing technologies is solved, and more efficient data writing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING HONGTENG INTELLIGENT TECH CO LTD
- Filing Date
- 2024-10-29
- Publication Date
- 2026-05-01
AI Technical Summary
The current Elasticsearch cluster experiences numerous network traffic forwardings during data writing, resulting in significant network overhead.
The client directly obtains the primary shard number of the document to be written, determines the address of the data storage node based on the primary shard number, and sends the data write request to the target node of the Elasticsearch cluster, reducing the forwarding processing through the coordinating node.
This reduces the number of traffic forwardings and network overhead during the data writing process, thus improving data writing efficiency.
Smart Images
Figure CN121967357A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data storage technology, and in particular to a data writing method, apparatus, electronic device, and computer storage medium. Background Technology
[0002] When writing data to an Elasticsearch cluster, the primary method is to use the API (Application Programming Interface) provided by the Elasticsearch cluster to write document data, including adding new documents to the Elasticsearch cluster or updating existing documents in the Elasticsearch cluster.
[0003] Currently, the typical workflow for writing data to an Elasticsearch cluster is as follows: The client sends a data write request to a node in the Elasticsearch cluster (i.e., the coordinator node) via a Virtual IP Address (VIP). The coordinator node receives the request, parses the write information in the request (such as index name, document ID, etc.), determines which primary shard the document should be stored in, and then forwards the request to that primary shard. The primary shard receives the request, performs the write operation, and can synchronize the data to replica shards. After completing the write process, it sends an acknowledgment message to the coordinator node. After receiving the acknowledgment message from the primary shard, the coordinator node returns the final data write result to the client.
[0004] While the existing data writing scheme described above can accomplish basic write operations, it involves communication between multiple nodes in the Elasticsearch cluster, including communication from the client to the coordinating node, from the coordinating node to the node containing the primary shard, and potential data synchronization between the primary and replica shards. The entire data writing process involves numerous traffic forwardings, resulting in significant network overhead. Summary of the Invention
[0005] This application provides a data writing method, apparatus, electronic device, and computer storage medium that can reduce the number of traffic forwardings and network overhead during the data writing process.
[0006] In a first aspect, embodiments of this application provide a data writing method, applied to a client, including:
[0007] In response to a data write command, obtain the primary shard number corresponding to the document to be written;
[0008] Based on the primary shard number, determine the address of the data storage node corresponding to the document to be written;
[0009] Based on the address of the data storage node, the data write request for the document to be written is sent to the Elasticsearch cluster.
[0010] In one possible implementation, the above method also includes:
[0011] At preset time intervals, information retrieval requests are sent to the sharding information interface of the Elasticsearch cluster, so that the sharding information interface sends sharding mapping information to the client based on the information retrieval requests; the sharding mapping information contains the mapping relationship information between the address of the data storage node and the primary shard number;
[0012] Receive and store the fragmentation mapping information sent by the fragmentation information interface.
[0013] In one possible implementation, the address of the data storage node corresponding to the document to be written is determined based on the primary shard number, including:
[0014] Based on the primary fragment number and the mapping relationship information in the fragment mapping information, determine the address of the data storage node corresponding to the document to be written.
[0015] In one possible implementation, in response to a data write command, the primary shard number corresponding to the document to be written is obtained, including:
[0016] In response to a data write command, obtain the document identifier of the document to be written and the number of primary shards in the Elasticsearch cluster;
[0017] The document identifier is hashed to obtain the hash value corresponding to the document to be written;
[0018] Based on the hash value and the number of primary shards, obtain the primary shard number corresponding to the document to be written.
[0019] In one possible implementation, the data write request for the document to be written is sent to the Elasticsearch cluster based on the address of the data storage node, including:
[0020] The documents to be written are grouped according to the address of the data storage node, and the documents to be written with the same address of the data storage node are grouped into the same group.
[0021] Concurrency technology is used to send data write requests from each group of documents to be written to the Elasticsearch cluster.
[0022] In one possible implementation, the documents to be written are grouped according to the address of the data storage node, and documents with the same data storage node address are grouped into the same group, including:
[0023] The documents to be written are grouped according to the address of the data storage node, the mapping relationship between the data storage node and the documents to be written is determined, and a mapping relationship table is formed with the address of the data storage node as the key.
[0024] Documents with the same key identifier in the mapping table are grouped into the same group.
[0025] One possible implementation uses concurrency techniques to send data write requests for documents in the same group to the Elasticsearch cluster, including:
[0026] Start a concurrent execution model for each document in the same group that is to be written;
[0027] By using a concurrent execution model, data write requests from each group of documents to be written are sent to the Elasticsearch cluster.
[0028] Secondly, embodiments of this application provide a data writing device applied to a client, comprising:
[0029] The acquisition module is used to obtain the main shard number corresponding to the document to be written in response to the data write command;
[0030] The determination module is used to determine the address of the data storage node corresponding to the document to be written, based on the primary shard number.
[0031] The sending module is used to send data write requests for the documents to be written to the Elasticsearch cluster based on the address of the data storage node.
[0032] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory; wherein the memory stores a computer program, the computer program being adapted to be loaded by the processor and execute the method steps provided in the first aspect of embodiments of this application.
[0033] Fourthly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps provided in the first aspect of embodiments of this application.
[0034] The aforementioned data writing method, apparatus, electronic device, and computer storage medium, in response to a data write command, obtain the primary shard number corresponding to the document to be written; determine the address of the data storage node corresponding to the document to be written based on the primary shard number; and send the data write request for the document to be written to the Elasticsearch cluster based on the address of the data storage node. Through this scheme, the client can directly send the data write request to the actual data storage node storing the document to be written, without needing to go through the coordinating node in the Elasticsearch cluster for forwarding, thereby effectively reducing the number of traffic forwardings and network overhead. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is a schematic diagram illustrating the implementation process of a data writing method provided in related technologies;
[0037] Figure 2 A schematic diagram illustrating the implementation process of a data writing method provided for an exemplary embodiment of this application;
[0038] Figure 3 A schematic diagram illustrating the application environment of a data writing method provided in an exemplary embodiment of this application;
[0039] Figure 4 A flowchart illustrating a data writing method provided for an exemplary embodiment of this application;
[0040] Figure 5 A flowchart illustrating another data writing method provided for an exemplary embodiment of this application;
[0041] Figure 6 A schematic diagram of the structure of a data writing device provided for an exemplary embodiment of this application;
[0042] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0044] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.
[0045] An Elasticsearch cluster refers to a collection of one or more nodes running the Elasticsearch service. These nodes work together to provide search and data analytics services. While each node in an Elasticsearch cluster has its own IP address, using virtual IP addresses helps simplify client configuration, achieve load balancing, and provide failover capabilities. Therefore, users typically use virtual IP addresses to access the Elasticsearch cluster. In this case, client requests are usually routed to a coordinating node in the cluster.
[0046] In related technologies, the implementation process of a data writing method is as follows: Figure 1 As shown. Please see below. Figure 1 This illustrates the three nodes (node 1, node 2, and node 3) in an Elasticsearch cluster and the different types of shards they support. Specifically, Figure 1 It contains two primary shards, R0 and R1, and their corresponding replica shards, P0 and P1. Node 1 carries primary shard P1 and replica shard R0, node 2 carries replica shards R0 and R1, and node 3 carries primary shard P0 and replica shard R1. (The rest of the text appears to be a list of nodes and their corresponding replicas.) Figure 1 As shown, the process of writing data to an Elasticsearch cluster using relevant technologies is as follows:
[0047] 1. The client publishes the data write request to the coordinating node ( Figure 1 In the middle node 1), the coordinating node receives the request, parses the write information in the request (such as index name, document ID, etc.), and determines that the document should be stored on the primary shard P0 based on the document ID.
[0048] 2. The coordinating node forwards the request to the primary shard P0, which receives the request and performs the write operation.
[0049] 3. The primary shard P0 synchronizes the data to the corresponding replica shard R0, and the replica shard R0 completes the write operation.
[0050] Understandably, Figure 1 The existing data writing scheme shown involves communication between multiple nodes in an Elasticsearch cluster, including communication from the client to the coordinating node, from the coordinating node to the node containing the primary shard, and possible data synchronization between the primary and replica shards. The entire data writing process involves numerous traffic forwardings, resulting in significant network overhead.
[0051] To address the aforementioned issues, this application provides a data writing method, apparatus, electronic device, and computer storage medium. When a user issues a data write command, the client responds by obtaining the primary shard number corresponding to the document to be written. Based on the primary shard number, it determines the address of the data storage node corresponding to the document to be written. Using the address of the data storage node, it sends the data write request for the document to be written to the Elasticsearch cluster, thereby directly sending the write request to the data storage node (i.e., the node where the primary shard of the document to be written actually resides), without needing to forward the request through a coordinating node. This solution reduces the number of traffic forwarding attempts and network overhead.
[0052] Please see Figure 2 This is a schematic diagram illustrating the implementation process of a data writing method provided in an exemplary embodiment of this application. Figure 2 The Elasticsearch cluster shown in the image contains three nodes and the different types of shards they support, all of which are related to... Figure 1 As shown, this is consistent with the previous description and will not be repeated here. Figure 2 As shown, the implementation process of writing data to an Elasticsearch cluster using the solution provided in this application is as follows:
[0053] 1. The client sends a data write request to the primary shard P0. The primary shard P0 receives the request and performs the write operation.
[0054] 2. The primary shard P0 synchronizes the data to the corresponding replica shard R0, and the replica shard R0 completes the write operation.
[0055] contrast Figure 1 and Figure 2 As can be seen, the data writing scheme provided in this application, compared with the data writing schemes in related technologies, eliminates the first step of the coordinating node forwarding the request to the primary shard, thus reducing one traffic forwarding. The data writing scheme provided in this application can significantly reduce network overhead when there are many data writing requests.
[0056] The data writing method provided in this application embodiment can be applied to, for example... Figure 3 The application environment is shown. Client 10 communicates with server 20 running the Elasticsearch service over a network. The data storage system stores the data that server 20 needs to process. The data storage system can be integrated onto server 20 or located on a cloud or other network server. In response to a data write command, client 10 obtains the primary shard number corresponding to the document to be written; based on the primary shard number, it determines the address of the data storage node corresponding to the document to be written; and based on the address of the data storage node, it sends the data write request for the document to be written to the Elasticsearch cluster.
[0057] The client 10 can be, but is not limited to, various personal computers, laptops, tablets, etc. The server 20 can be implemented using a server cluster consisting of multiple servers.
[0058] In one embodiment, such as Figure 4 As shown, a data writing method is provided, which is applied to... Figure 3 Taking client 10 as an example, the explanation includes the following steps:
[0059] S401: In response to a data write command, obtain the primary shard number corresponding to the document to be written.
[0060] The primary shard number corresponding to the document to be written refers to the number used in the Elasticsearch cluster to identify which primary shard the document should be written to.
[0061] Understandably, in Elasticsearch, each index can contain multiple documents, and each index is divided into multiple shards. The primary shard is the main carrier of the index data (including documents), while replica shards are backups of the primary shard, used to improve data fault tolerance and availability. When a user issues a data write command to client 10 to write data to the Elasticsearch cluster, client 10 responds to the data write command by obtaining a number that identifies which primary shard the document to be written should be written to.
[0062] Specifically, in response to the user's data write command, client 10 obtains the shard number of the primary shard corresponding to the document to be written using the document ID of the document to be written and a hash algorithm (such as the MurmurHash3 algorithm). Here, the document ID of the document to be written is a user-defined document ID, the hash algorithm is used to hash the document ID of the document to be written, and the primary shard corresponding to the document to be written is the shard responsible for storing the main data of the document to be written.
[0063] S402: Determine the address of the data storage node corresponding to the document to be written based on the primary shard number.
[0064] The address of the data storage node corresponding to the document to be written refers to the network address of the node in the Elasticsearch cluster where the primary shard of the document to be written is located, such as the IP (Internet Protocol) address of the node where the primary shard of the document to be written is located.
[0065] Optionally, the client 10 pre-stores a sharding mapping table for the Elasticsearch cluster. This sharding mapping table contains mapping information between the addresses of data storage nodes and primary shard numbers, that is, the correspondence between each primary shard number and the network address of its corresponding storage node. After obtaining the primary shard number corresponding to the document to be written, the client 10 determines the address of the data storage node corresponding to the document to be written by referring to the pre-stored sharding mapping table.
[0066] S403: Based on the address of the data storage node, send the data write request for the document to be written to the Elasticsearch cluster.
[0067] The data writing requests for the document to be written include, but are not limited to, HTTP (Hypertext Transfer Protocol) requests and HTTPS (Hypertext Transfer Protocol Secure) requests.
[0068] For example, client 10 sends an HTTP request (such as a POST request) for the document to be written to the address of the data storage node corresponding to the document in the Elasticsearch cluster. This enables the primary shard on that data storage node to receive the request and perform the data write operation. Optionally, the primary shard can also send the data write request to its corresponding replica shard for data synchronization. After a replica shard successfully writes the data, it sends an acknowledgment message to the primary shard. Once the primary shard receives acknowledgments from all replica shards, it sends a final acknowledgment message to the coordinating node, indicating that the write operation has been completed. After receiving the acknowledgment message from the primary shard, the coordinating node returns the final write result to client 10.
[0069] It is worth noting that the document to be written mentioned in this embodiment can be a single document or multiple documents. For a single document, client 10 can directly send the data write request for that document to the corresponding data storage node address. For multiple documents, client 10 can group the multiple documents according to their data storage node addresses, and then use concurrency technology to send the data write requests for multiple documents to the Elasticsearch cluster in groups. The specific implementation method for sending the data write requests for multiple documents to the Elasticsearch cluster in batches will be described in the next embodiment, and will not be repeated here.
[0070] The above data writing method, in response to a data write command, obtains the primary shard number corresponding to the document to be written; based on the primary shard number, determines the address of the data storage node corresponding to the document to be written; and based on the address of the data storage node, sends the data write request for the document to be written to the Elasticsearch cluster. Through this scheme, the client can directly send the data write request to the actual data storage node storing the document to be written, without needing to go through the coordinating node in the Elasticsearch cluster for forwarding, thereby effectively reducing the number of traffic forwardings and network overhead.
[0071] In one embodiment, such as Figure 5 As shown, another data writing method is provided, which can be applied to... Figure 3 Taking client 10 as an example, the explanation includes the following steps:
[0072] S501: Sends information retrieval requests to the Elasticsearch cluster's sharding information interface at preset time intervals, so that the sharding information interface sends sharding mapping information to the client based on the information retrieval requests.
[0073] The sharding information interface refers to the ` / _cat / shards` interface of the Elasticsearch cluster, which returns sharding information for all indexes in the Elasticsearch cluster, including shard status, type, shard number, node name, and node IP. Therefore, the sharding mapping information contains the mapping relationship between the address of the data storage node and the primary shard number.
[0074] Understandably, the above fragmentation mapping information can be presented in the form of a list. For example, "my index":{"0":["192.168.1.1"],"1":["192.168.1.2"],"2":["192.168.1.3"]} means that the primary fragment with fragment number 0 is stored on the IP address 192.168.1.1, the primary fragment with fragment number 1 is stored on the IP address 192.168.1.2, and the primary fragment with fragment number 2 is stored on the IP address 192.168.1.3.
[0075] Optionally, client 10 sends information retrieval requests to the ` / _cat / shards` interface of the Elasticsearch cluster at preset time intervals (e.g., every hour). This allows the ` / _cat / shards` interface to send shard mapping information between the data storage node address and the primary shard number to client 10 based on the information retrieval requests. It should be noted that the primary shard of the Elasticsearch cluster can change; therefore, client 10 needs to send information retrieval requests to the ` / _cat / shards` interface at preset time intervals to continuously update the shard mapping information stored in client 10.
[0076] S502: Receive and store the fragmentation mapping information sent by the fragmentation information interface.
[0077] Optionally, client 10 receives sharding mapping information sent by the / _cat / shards interface and stores the sharding mapping information in a preset medium so that client 10 can subsequently determine the address of the data storage node corresponding to the document to be written based on the primary shard number and the stored sharding mapping information. It is worth noting that the preset medium can be any of memory, Redis (Remote Dictionary Server), MySQL, or Txt (Text File).
[0078] It's worth noting that each time client 10 receives sharding mapping information from the / _cat / shards interface, it prioritizes obtaining the sharding mapping information corresponding to the primary shard in the RUNNING (normal operation) state. If the primary shard is in one of the special states—STARTED, RELOCATING, or UNASSIGNED—it ignores it initially, meaning it doesn't store the sharding mapping information for these states. When client 10 subsequently obtains new sharding mapping information at preset time intervals, if the primary shard in the new sharding mapping information has transitioned from one of the aforementioned special states to a primary shard in the RUNNING state, then it stores that information.
[0079] In this embodiment, the client periodically obtains sharding mapping information sent by the Elasticsearch cluster's sharding information interface. This facilitates the subsequent determination of the address of the data storage node corresponding to the document to be written based on the primary shard number and the stored sharding mapping information. The client then directly sends the request to write the document to that data storage node address, thereby reducing the number of traffic forwardings and lowering network overhead.
[0080] S503: In response to a data write command, obtain the document identifier of the document to be written and the number of primary shards in the Elasticsearch cluster.
[0081] The document identifier of the document to be written is the document ID of the document to be written.
[0082] Optionally, in response to a data write command, client 10 obtains the document identifier of the document to be written and the number of primary shards in the Elasticsearch cluster, which are pre-configured in the Elasticsearch cluster.
[0083] S504: Hash the document identifier to obtain the hash value corresponding to the document to be written.
[0084] Optionally, client 10 applies the MurmurHash3 algorithm to the document ID to obtain the hash value corresponding to the document to be written.
[0085] S505: Obtain the primary shard number corresponding to the document to be written based on the hash value and the number of primary shards.
[0086] Optionally, a modulo operation (modulo operation) can be used to map the hash value to the primary shard number corresponding to the document to be written. Specifically, the primary shard number corresponding to the document to be written can be calculated by taking the modulo of the hash value and the number of primary shards.
[0087] In this embodiment, the client responds to the data write command by obtaining the primary shard number corresponding to the document to be written based on the document identifier of the document to be written and the number of primary shards in the Elasticsearch cluster, combined with a hash algorithm. This helps to simplify the request forwarding process in the Elasticsearch cluster and reduce network overhead while achieving load balancing.
[0088] S506: Determine the address of the data storage node corresponding to the document to be written based on the primary fragment number and the mapping relationship information in the fragment mapping information.
[0089] Understandably, since the sharding mapping information contains the mapping relationship information between the address of the data storage node and the primary shard number, the client 10 can determine the address of the data storage node corresponding to the document to be written based on the primary shard number and the mapping relationship information in the sharding mapping information.
[0090] For example, assuming the primary shard number corresponding to the document to be written is 1, the shard mapping information can be displayed in list form as: "my index":{"0":["192.168.1.1"], "1":["192.168.1.2"], "2":["192.168.1.3"]}. Then, based on the primary shard number and the mapping relationship information in the shard mapping information, client 10 can determine that the address of the data storage node corresponding to the document to be written is 192.168.1.2.
[0091] In this embodiment, the node address where the document should be written can be directly determined through the fragmentation mapping information. This allows the client to directly send the data write request to the data storage node corresponding to the document to be written, without needing to forward it through a coordinating node, thus reducing network communication overhead. Furthermore, directly sending the write request to the target node can also reduce network latency and improve data writing efficiency.
[0092] S507: Group the documents to be written according to the address of the data storage node, and divide the documents to be written with the same address of the data storage node into the same group of documents to be written.
[0093] Optionally, the client 10 groups the documents to be written according to the address of the data storage node, dividing the documents to be written with the same data storage node address into the same group of documents to be written, and dividing the documents to be written with different data storage node addresses into different groups of documents to be written.
[0094] In one embodiment, grouping documents to be written according to the address of the data storage node, and dividing documents to be written with the same address of the data storage node into the same group of documents to be written, includes: grouping documents to be written according to the address of the data storage node, determining the mapping relationship between the data storage node and the documents to be written, forming a mapping relationship table with the address of the data storage node as the key; and dividing documents to be written with the same key in the mapping relationship table into the same group of documents to be written.
[0095] Optionally, client 10 groups the documents to be written according to the addresses of data storage nodes, determines the mapping relationship between data storage nodes and documents to be written, and uses a mapping table to store the documents to be written, where the key is the address of the data storage node and the value is a list to store all documents to be written under that data storage node address. In this way, all documents to be written with the same data storage node address can be grouped together and then assembled and sent to the same data storage node.
[0096] In this embodiment, the client can group documents with the same key identifier together to achieve batch processing of multiple documents to be written. This not only reduces network communication overhead, but also combines concurrency technology to assemble and send the data write requests corresponding to multiple documents to be written to the Elasticsearch cluster, thereby improving data writing efficiency.
[0097] S508: Employs concurrency technology to send data write requests from each group of documents to be written to the Elasticsearch cluster.
[0098] Optionally, client 10 employs concurrency technology combined with multiplexing to send data write requests from each group of documents to be written to the corresponding data storage nodes in the Elasticsearch cluster. Specifically, client 10 can create multiple threads or coroutines to process write requests from different groups of documents in parallel. Each thread or coroutine can be responsible for sending a group of documents, thus achieving concurrent writing. Furthermore, client 10 can register all established network connections with the selected I / O multiplexing model. This model monitors the connection status of the network connections established between client 10 and each data storage node in the Elasticsearch cluster, and notifies client 10 to take appropriate action if any connection status is abnormal.
[0099] In one embodiment, concurrency technology is used to send data write requests for each group of documents to be written to the Elasticsearch cluster, including: starting a concurrent execution model for each group of documents to be written; and sending data write requests from each group of documents to be written to the Elasticsearch cluster through the concurrent execution model.
[0100] Optionally, client 10 starts a concurrent execution model for each group of documents to be written, such as starting a process or a coroutine for each group of documents. Through the started process or coroutine, different data write requests from each group of documents are sent in batches to the corresponding data storage nodes in the Elasticsearch cluster, thereby effectively improving data write efficiency. In actual tests under the same conditions, the data write solution provided in this application improves write speed by 0%-13% compared to related technologies; the specific improvement depends on the number of concurrent writes.
[0101] In this embodiment, client 10 processes multiple data write requests for documents in batches, sending the data write requests directly to the node to which the primary shard of the document actually resides. This reduces the number of network traffic forwardings and network communication overhead. Furthermore, by processing multiple document write requests in parallel, client 10 also reduces data write time and improves data write efficiency.
[0102] In the above data writing method, the client determines the primary shard number of the document to be written by hashing its document identifier. Then, based on the primary shard number and pre-stored shard mapping information, it determines the address of the data storage node corresponding to the document to be written. This allows the client to directly send the data write request to the target node based on the data storage node's address, thereby reducing the number of traffic forwardings and network overhead. Furthermore, by concurrently processing multiple data write requests for documents to be written, the client also reduces data write time and improves data write efficiency.
[0103] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0104] Based on the above-mentioned data writing method, such as Figure 6 As shown, this application embodiment also provides a data writing device 600 for implementing the above-described data writing method applied to a client. The data writing device 600 includes:
[0105] The acquisition module 601 is used to obtain the main shard number corresponding to the document to be written in response to the data write command;
[0106] The determination module 602 is used to determine the address of the data storage node corresponding to the document to be written based on the main shard number;
[0107] The sending module 603 is used to send the data write request of the document to be written to the Elasticsearch cluster according to the address of the data storage node.
[0108] In one possible implementation, the determining module 602 is further configured to send information retrieval requests to the sharding information interface of the Elasticsearch cluster at preset time intervals, so that the sharding information interface sends sharding mapping information to the client based on the information retrieval requests; the sharding mapping information contains the mapping relationship information between the address of the data storage node and the primary shard number; and receive and store the sharding mapping information sent by the sharding information interface.
[0109] In one possible implementation, the determining module 602 is specifically used to determine the address of the data storage node corresponding to the document to be written, based on the mapping relationship information in the primary fragment number and fragment mapping information.
[0110] In one possible implementation, the acquisition module 601 is specifically used to respond to the data write command by obtaining the document identifier of the document to be written and the number of primary shards in the Elasticsearch cluster; performing hash processing on the document identifier to obtain the hash value corresponding to the document to be written; and obtaining the primary shard number corresponding to the document to be written based on the hash value and the number of primary shards.
[0111] In one possible implementation, the sending module 603 is specifically used to group the documents to be written according to the address of the data storage node, and divide the documents to be written with the same address of the data storage node into the same group of documents to be written; using concurrency technology, the data write requests of each group of documents to be written are sent to the Elasticsearch cluster.
[0112] In one possible implementation, the sending module 603 is specifically used to group the documents to be written according to the address of the data storage node, determine the mapping relationship between the data storage node and the documents to be written, and form a mapping relationship table with the address of the data storage node as the key; and divide the documents to be written with the same key in the mapping relationship table into the same group of documents to be written.
[0113] In one possible implementation, the sending module 603 is specifically used to initiate a concurrent execution model for each group of documents to be written; through the concurrent execution model, the data write requests in each group of documents to be written are sent to the Elasticsearch cluster.
[0114] Each module in the aforementioned data writing device 600 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0115] This application also provides an electronic device, which can be a terminal, and its internal structure diagram can be as follows: Figure 7As shown, this electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data writing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.
[0116] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0117] In one possible implementation, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0118] In response to a data write command, obtain the primary shard number corresponding to the document to be written;
[0119] Based on the primary shard number, determine the address of the data storage node corresponding to the document to be written;
[0120] Based on the address of the data storage node, the data write request for the document to be written is sent to the Elasticsearch cluster.
[0121] In one possible implementation, the processor, when executing the computer program, also performs the following steps: sending information retrieval requests to the sharding information interface of the Elasticsearch cluster at preset time intervals, so that the sharding information interface sends sharding mapping information to the client based on the information retrieval requests; the sharding mapping information contains mapping relationship information between the address of the data storage node and the primary shard number; and receiving and storing the sharding mapping information sent by the sharding information interface.
[0122] In one possible implementation, the processor, when executing the computer program, also performs the following steps: determining the address of the data storage node corresponding to the document to be written, based on the primary fragment number and the mapping relationship information in the fragment mapping information.
[0123] In one possible implementation, the processor, when executing the computer program, also performs the following steps: in response to a data write instruction, obtains the document identifier of the document to be written and the number of primary shards in the Elasticsearch cluster; hashes the document identifier to obtain the hash value corresponding to the document to be written; and obtains the primary shard number corresponding to the document to be written based on the hash value and the number of primary shards.
[0124] In one possible implementation, the processor, while executing the computer program, also performs the following steps: grouping the documents to be written according to the address of the data storage node, dividing the documents to be written with the same address of the data storage node into the same group of documents to be written; and using concurrency technology, sending the data write requests of each group of documents to be written to the Elasticsearch cluster.
[0125] In one possible implementation, the processor, when executing the computer program, also performs the following steps: grouping the documents to be written according to the address of the data storage node, determining the mapping relationship between the data storage node and the documents to be written, and forming a mapping relationship table identified by the address of the data storage node; and dividing the documents to be written with the same key in the mapping relationship table into the same group of documents to be written.
[0126] In one possible implementation, the processor, while executing the computer program, also performs the following steps: initiating a concurrent execution model for each group of documents to be written; and sending data write requests from each group of documents to be written to the Elasticsearch cluster via the concurrent execution model.
[0127] This application also provides a computer storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer storage medium.
[0128] In one possible implementation, a computer storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, it performs the following steps:
[0129] In response to a data write command, obtain the primary shard number corresponding to the document to be written;
[0130] Based on the primary shard number, determine the address of the data storage node corresponding to the document to be written;
[0131] Based on the address of the data storage node, the data write request for the document to be written is sent to the Elasticsearch cluster.
[0132] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: sending information retrieval requests to the sharding information interface of the Elasticsearch cluster at preset time intervals, so that the sharding information interface sends sharding mapping information to the client based on the information retrieval requests; the sharding mapping information contains the mapping relationship information between the address of the data storage node and the primary shard number; and receiving and storing the sharding mapping information sent by the sharding information interface.
[0133] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: determining the address of the data storage node corresponding to the document to be written, based on the primary fragment number and the mapping relationship information in the fragment mapping information.
[0134] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: in response to a data write instruction, it obtains the document identifier of the document to be written and the number of primary shards in the Elasticsearch cluster; it hashes the document identifier to obtain the hash value corresponding to the document to be written; and it obtains the primary shard number corresponding to the document to be written based on the hash value and the number of primary shards.
[0135] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: grouping the documents to be written according to the address of the data storage node, dividing the documents to be written with the same address of the data storage node into the same group of documents to be written; and using concurrency technology, sending the data write requests of each group of documents to be written to the Elasticsearch cluster.
[0136] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: grouping the documents to be written according to the address of the data storage node, determining the mapping relationship between the data storage node and the documents to be written, and forming a mapping relationship table identified by the address of the data storage node; and dividing the documents to be written with the same key in the mapping relationship table into the same group of documents to be written.
[0137] In one possible implementation, the computer program, when executed by the processor, also performs the following steps: initiating a concurrent execution model for each group of documents to be written; and sending data write requests from each group of documents to be written to the Elasticsearch cluster via the concurrent execution model.
[0138] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).
[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.
[0140] The embodiments described above are merely preferred embodiments of this application and are not intended to limit the scope of this application. Any modifications and improvements made by those skilled in the art to the technical solutions of this application without departing from the spirit of this application should fall within the protection scope defined by the claims.
[0141] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
Claims
1. A data writing method, characterized in that, Applied to a client, the method includes: In response to a data write command, obtain the primary shard number corresponding to the document to be written; Based on the primary shard number, determine the address of the data storage node corresponding to the document to be written; Based on the address of the data storage node, the data write request for the document to be written is sent to the Elasticsearch cluster.
2. The method as described in claim 1, characterized in that, The method further includes: At preset time intervals, information retrieval requests are sent to the sharding information interface of the Elasticsearch cluster, so that the sharding information interface sends sharding mapping information to the client based on the information retrieval requests; the sharding mapping information includes the mapping relationship information between the address of the data storage node and the primary shard number; Receive and store the fragmentation mapping information sent by the fragmentation information interface.
3. The method as described in claim 2, characterized in that, The step of determining the address of the data storage node corresponding to the document to be written based on the primary shard number includes: The address of the data storage node corresponding to the document to be written is determined based on the primary fragment number and the mapping relationship information in the fragment mapping information.
4. The method as described in claim 1, characterized in that, The step of obtaining the primary shard number corresponding to the document to be written in response to the data write command includes: In response to a data write command, obtain the document identifier of the document to be written and the number of primary shards in the Elasticsearch cluster; The document identifier is hashed to obtain the hash value corresponding to the document to be written; Based on the hash value and the number of primary shards, obtain the primary shard number corresponding to the document to be written.
5. The method as described in claim 1, characterized in that, The step of sending the data write request for the document to be written to the Elasticsearch cluster according to the address of the data storage node includes: The documents to be written are grouped according to the address of the data storage node, and the documents to be written with the same address of the data storage node are grouped into the same group of documents to be written. Using concurrency technology, data write requests from each of the documents to be written in the same group are sent to the Elasticsearch cluster.
6. The method as described in claim 5, characterized in that, The step of grouping the documents to be written according to the address of the data storage node, and dividing the documents to be written with the same address of the data storage node into the same group of documents to be written, includes: The documents to be written are grouped according to the address of the data storage node, and the mapping relationship between the data storage node and the documents to be written is determined to form a mapping relationship table identified by the address of the data storage node; Documents with the same key identifier in the mapping table are grouped into the same group of documents to be written.
7. The method as described in claim 5, characterized in that, The method of employing concurrency technology to send data write requests for each of the documents to be written in the same group to the Elasticsearch cluster includes: A concurrent execution model is launched for each of the documents to be written in the same group; The concurrent execution model sends data write requests from each of the same group of documents to the Elasticsearch cluster.
8. A data writing device, characterized in that, Applied to a client, the device includes: The acquisition module is used to obtain the main shard number corresponding to the document to be written in response to the data write command; The determination module is used to determine the address of the data storage node corresponding to the document to be written based on the main shard number; The sending module is used to send the data write request of the document to be written to the Elasticsearch cluster according to the address of the data storage node.
9. An electronic device, characterized in that, include: A processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of claims 1-7.
10. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-7.