Data writing method, device, medium and electronic device based on designated route

Abstract

The present application relates to a data writing method, device, medium and electronic device based on a specified route, the method comprising: receiving a data writing request, and obtaining the write data in the data writing request; selecting a routing value from a pre-stored routing value table to configure the write data, the routing value table comprising a plurality of routing values; querying the corresponding shard code in the correspondence table according to the configured routing value, the correspondence table comprising the correspondence between each routing value and each shard code; determining the corresponding shard according to the queried shard code, and sending the write data to the node where the corresponding shard is located for writing. The present application enables each ES data writing operation to no longer have internal data distribution, effectively reducing the cluster I / O load and improving the stability of the cluster. It solves the long tail problem and improves the writing efficiency of ES data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data writing method, device, medium and electronic device based on a specified route. Background Art

[0002] ElasticSearch (ES) is a popular distributed, multi-user, full-text search server designed for cloud computing, enabling real-time search. Due to its distributed nature, ES can be clustered across multiple servers, further sharding the data in the cluster index. Multiple shards can be contained within a single server, improving indexing and search capabilities.

[0003] In related technologies, writing large amounts of data to ES uses a batch request method, submitting the data in batches to the ES cluster. Once a batch of data is sent to the ES client node, it distributes it according to the routing values in the data. The data to be distributed to different shards is categorized and assembled into a batch of BulkShardRequest requests, which are then sent to the data node where each shard resides. This operation consumes a considerable amount of network resources. In cluster environments with large data volumes and strict resource constraints, writing to ES can result in high I / O usage, potentially leading to I / O saturation or even ES crashes. Furthermore, sending BulkShardRequests is asynchronous, requiring each BulkRequest to wait for all BulkShardRequest responses before returning to the client. Requests to some shards in the index being written can be slowed down by memory garbage collection, failures, queues, locks, and other issues, creating the so-called "long tail" problem.

[0004] Therefore, an efficient data writing method based on a specified route is urgently needed in the related art. Summary of the Invention

[0005] In view of the above situation, the main purpose of the present invention is to overcome the high I / O occupancy and long tail problems when writing data in the existing ES cluster, and to provide a data writing method, device, medium and electronic device based on specified routing.

[0006] A data writing method based on a specified route, comprising:

[0007] receiving a data write request and obtaining write data in the data write request;

[0008] Selecting a routing value from a pre-stored routing value table and configuring it to the write data, wherein the routing value table includes a plurality of routing values;

[0009] According to the configured routing value, the corresponding fragment code is searched in the corresponding relationship table, wherein the corresponding relationship table includes the corresponding relationship between each routing value and each fragment code;

[0010] The corresponding shard is determined according to the queried shard code, and the write data is sent to the node where the corresponding shard is located for writing.

[0011] Furthermore, the above data writing method, before the step of receiving the data writing request, further comprises:

[0012] Send a query request containing at least one index alias to the ES cluster, so that the ES cluster queries all shards under the index alias based on the query request and returns the allocation code of each shard;

[0013] Determine the number of shards according to the shard code of each shard returned by the ES cluster;

[0014] Traversing the values of each positive integer within a preset value range in ascending order, and performing slice code calculation based on the currently traversed value and the number of slices to obtain the slice code corresponding to the value, and stopping the slice code calculation when the number of calculated slice codes is equal to the number of slices;

[0015] Using the numerical value corresponding to each of the fragment codes as the routing value, and establishing the routing value table according to each of the routing values;

[0016] The corresponding relationship table is established according to the corresponding relationship between the routing value and the fragment code.

[0017] Furthermore, in the above-mentioned data writing method, the step of calculating the slice code according to the currently traversed value and the number of slices to obtain the slice code corresponding to the value includes:

[0018] A modulo operation is performed based on the currently traversed value and the number of slices to obtain the slice code corresponding to the value.

[0019] Furthermore, in the above data writing method, the step of sending a query request containing at least one index alias to the ES cluster includes:

[0020] Send a query request containing at least one index alias to the ES cluster periodically or based on the message monitoring mechanism.

[0021] Furthermore, in the above data writing method, the step of selecting a routing value from a pre-stored routing value table and configuring it for the written data includes:

[0022] A routing value is selected from a pre-stored routing value table according to a random selection rule or a cyclic selection rule and configured to the write data.

[0023] Furthermore, the above data writing method, wherein, after the step of sending the written data to the node where the corresponding shard is located for writing, further comprises:

[0024] When the write data fails to be written in the node, the write data is sent to other nodes for writing.

[0025] The present invention also discloses a data writing device based on a designated route, comprising:

[0026] An acquisition module, configured to receive a data write request and acquire write data in the data write request;

[0027] A configuration module, configured to select a routing value from a pre-stored routing value table and configure it to the write data, wherein the routing value table includes a plurality of routing values;

[0028] A query module, configured to query a corresponding fragment code in a correspondence table according to a configured routing value, wherein the correspondence table includes a correspondence between each routing value and each fragment code;

[0029] The first sending module is used to determine the corresponding fragment according to the queried fragment code, and send the write data to the node where the corresponding fragment is located for writing.

[0030] Furthermore, the data writing device further includes:

[0031] The second sending module is used to send a query request containing at least one index alias to the ES cluster, so that the ES cluster queries all shards under the index alias based on the query request and returns the allocation code of each shard;

[0032] A determination module, configured to determine the number of the shards according to the shard codes of the shards returned by the ES cluster;

[0033] a calculation module, configured to sequentially traverse the values of each positive integer within a preset value range in ascending order, and perform slice code calculation based on the currently traversed value and the number of slices to obtain a slice code corresponding to the value, and stop the slice code calculation when the number of calculated slice codes equals the number of slices;

[0034] a routing table establishing module, configured to use the numerical value corresponding to each of the fragment codes as the routing value, and establish the routing value table according to each of the routing values;

[0035] The corresponding relationship establishing module is used to establish the corresponding relationship table according to the corresponding relationship between the routing value and the fragment code.

[0036] The present invention also discloses a computer-readable storage medium on which a program is stored. When the program is executed by a processor, any one of the above-mentioned methods is implemented.

[0037] The present invention also discloses an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements any one of the above-described methods when executing the program.

[0038] This invention is based on real-time write scenarios for large-scale, multi-node ES clusters. It addresses situations where cluster I / O usage is high and large cluster batch writes have a long-tail problem. By specifying a routing value for each ES write request and, based on the relationship between the routing value and the shard code, sending each batch data write request to the ES node where the shard corresponding to the routing value is located, this method eliminates internal data distribution for each ES data write operation, effectively reducing the cluster I / O load and improving cluster stability. This solves the long-tail problem and improves the efficiency of ES data writes.

[0039] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 Flowchart of the data writing method in the first embodiment of the present invention;

[0042] Figure 2 is a flow chart of a data writing method according to a second embodiment of the present invention;

[0043] Figure 3 is a flow chart of a data writing method according to a third embodiment of the present invention;

[0044] Figure 4 2 is a schematic diagram of the module structure of a data writing device in a fourth embodiment of the present invention;

[0045] Figure 5Schematic diagram of the module structure of the electronic device in an embodiment of the present invention.

[0046] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0048] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0049] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0050] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0051] Technical terms in this invention:

[0052] Distributed storage search engine system (Elastic Search, ES): ES is a distributed RESTful-style search and data analysis engine, and is also a popular enterprise-level search engine.

[0053] Bulk Request: A request for batch data operations supported by ES.

[0054] Data Node: A data node is a node in an ES cluster that is responsible for storing data and can also provide storage and query services.

[0055] Client Node: It is a role of a node in the ES cluster and can provide storage and query services in the ES cluster.

[0056] Shards: Data in Elasticsearch (ES) is organized into indexes. Each index is composed of one or more shards. Each shard is a Lucene index instance that indexes a subset of the data in the Elasticsearch cluster and processes related queries. Shards are the unit by which Elasticsearch distributes data within the cluster.

[0057] See also Figure 1, which is a data writing method based on a designated route in the first embodiment of the present invention, including steps S11 to S14.

[0058] Step S11: receiving a data write request and obtaining the write data in the data write request.

[0059] The write request is an operation requesting to transmit write data to a server in the ES cluster, and the write request includes the write data.

[0060] Step S12: assigning a routing value (routing value) to the written data from a preset routing value table, wherein the routing value table includes a plurality of routing values.

[0061] Step S13: query the corresponding fragment code in the corresponding relationship table according to the configured routing value, wherein the corresponding relationship table includes the corresponding relationship between each routing value and each fragment code.

[0062] In an embodiment of the present application, two tables may be pre-stored in the system, namely a routing value table and a correspondence table. The routing value table may contain multiple routing values, and the multiple routing values may be arbitrary values pre-set by the user. The correspondence table includes a one-to-one correspondence between each routing value and each slice code. In actual applications, the routing value table and the correspondence table may be stored in the system in the form of a table, or may be stored only as data. Of course, the routing value table and the correspondence table may also be combined into one table for storage, and this application does not impose any restrictions on this.

[0063] It should be noted that since each shard code corresponds to a shard in the ES cluster, the correspondence between each routing value and shard can be determined based on the correspondence table. When writing data, a routing value can be given to each write request to specify that all data in the write request be written to a single shard.

[0064] In specific implementation, when selecting the routing value and its corresponding request node, load balancing needs to be considered, including but not limited to assigning a routing value to the write data in a random or cyclic selection manner.

[0065] Step S14: determine the corresponding shard according to the queried shard code, and send the write data to the node where the corresponding shard is located for writing.

[0066] In an embodiment of the present application, the node may include a server in an ES cluster. Since the ES cluster is composed of multiple servers, the written data only needs to be transmitted to one of the servers. The server where the corresponding shard is located can be found based on the queried shard code. The written data is sent to the server with the difference, so that the server performs the write operation.

[0067] In an embodiment of the present application, a routing value is specified for each request to write to ES, and according to the correspondence between the shards and the routing values in the ES cluster, each data batch write request is sent to the ES node where the shard corresponding to the routing value is located, so that each ES data write operation no longer has internal data distribution, which can effectively reduce the cluster I / O load, solve the long tail problem, improve the writing efficiency of ES data, reduce the system load, and improve the stability of the cluster.

[0068] For further information, see Figure 2 In the second embodiment of the present invention, the step of receiving the data write request further includes steps S21 to S25.

[0069] Step S21: Send a query request containing at least one index alias to the ES cluster, so that the ES cluster queries all shards under the index alias based on the query request and returns the allocation code of each shard.

[0070] Step S22: Determine the number of the shards according to the shard codes of the shards returned by the ES cluster.

[0071] Step S23, traverse the values of each positive integer in the preset numerical range in order from small to large, and perform slice code calculation based on the currently traversed value and the number of slices to obtain the slice code corresponding to the value, and stop the slice code calculation when the calculated number of slice codes is equal to the number of slices.

[0072] Step S24: using the numerical value corresponding to each of the fragment codes as the routing value, and establishing the routing value table according to each of the routing values.

[0073] Step S25: establishing the corresponding relationship table according to the corresponding relationship between the routing value and the fragment code.

[0074] In the embodiment of the present application, the routing value table and the corresponding relationship table are obtained by the system after calculation according to a series of calculation processes. In specific implementation, a query request is sent to the ES cluster to query the relationship between the index shard to be written and the ES node name.

[0075] This includes but is not limited to using the official ES HTTP API: GET_cat / shards / alias? v. This request will return all shards for the index represented by the alias and their node relationships, as well as the number of shards corresponding to the index. For example, the request is GET_cat / shards / ailpha-baas-alarm? v, and the corresponding return value format is:

[0076]

[0077] Maintain information such as the relationship between the queried shard and the node it resides on, as well as the number of shards, including but not limited to storing it in the system's memory for subsequent program use. Based on the number of shards, calculate the relationship between the routing value and the shard number, and save the result in the system. The calculation method includes, but is not limited to:

[0078] Preset the value range to 0 to a sufficiently large value, and iterate over the integer values within the preset range. For each value, calculate and obtain the corresponding shard number according to the calculation method for writing routing values provided by ES (this method uses the number of shards n to ensure that the calculated shard number is in the range of 0 to n-1). Repeat this calculation until the size of the resulting shard encoding result set equals the number of shards.

[0079] For example, if the value ranges from 0 to 100, the shard code is calculated based on the current value and the number of shards. This can be done using a modulo operation. This means dividing the current value by number_of_primary_shards (the number of shards) yields the remainder, which is between 0 and number_of_primary_shards minus one (counting starts at 0, so if there are 5 shards, the range is 0 to 4). This is the shard location where the document is stored. For example, if the current value is 2, the calculated shard code is 2. Since the values are calculated in descending order, the shard codes corresponding to the values 1, 2, 3, 4, and 5 are 1, 2, 3, 4, and 0, respectively. In other words, we only need to calculate the shard codes corresponding to 1, 2, 3, 4, and 5.

[0080] The numerical value corresponding to each shard code is used as the routing value. For example, in the example above, 1, 2, 3, 4, and 5 are routing values. These routing values can be stored in a routing value table. Furthermore, the relationship between the calculated routing value and the index code is maintained, including but not limited to storing it in the system's memory for subsequent program use. The system has already established the relationship between the routing value and the shard code, and between the shard code and the node where the shard resides, and thus the relationship between the routing value and the corresponding node.

[0081] It should be noted that maintaining ES-related information in the system includes but is not limited to:

[0082] Query in advance the number of shards for the specific index corresponding to the index alias being written.

[0083] Calculate the correspondence between routing values and shard codes under the current number of shards.

[0084] Query the relationship between the shard and the node name.

[0085] Finally, the relationship between routing value -> node name can be obtained and saved in the system.

[0086] See also Figure 3 , which is a data writing method based on a designated route in a third embodiment of the present invention, including steps S31 to S39.

[0087] In step S31, the ES query and standby client sends a query request including at least one index alias to the ES cluster, so that the ES cluster queries all shards under the index alias based on the query request and returns the allocation code of each shard.

[0088] In step S32, the ES query and standby client determines the number of the shards according to the shard codes of the shards returned by the ES cluster.

[0089] In step S33, the ES query and the backup client sequentially traverse the values of each positive integer within the preset value range in order from small to large, and perform fragment code calculation based on the currently traversed value and the number of fragments to obtain the fragment code corresponding to the value, and stop the fragment code calculation when the number of calculated fragment codes is equal to the number of fragments.

[0090] In step S34, the ES query and standby client uses the numerical value corresponding to each of the fragment codes as the routing value, establishes a routing value table according to each of the routing values, and sends the routing value table to the ES client.

[0091] In the embodiment of the present application, a query and backup client is pre-established in the system, which is mainly responsible for querying the status of the ES cluster and retrying the write request after failing to send a write request to the specified ES node using the routing value.

[0092] By using the query and the standby client, a query request is sent to the ES cluster to find out the relationship between the index shard to be written and the name of the ES node where it is located.

[0093] Maintain information such as the relationship between the queried shard and the ES node where it resides, as well as the number of shards n, including but not limited to saving it in the system's memory for subsequent program use.

[0094] Based on the obtained shard quantity information, the relationship between the routing value and the shard code is calculated and the result is saved in the system. The calculation method is for example, but not limited to:

[0095] Preset the routing value to a range of 0 to a sufficiently large integer. Iterate through the integers within the preset range and calculate the corresponding shard code for each routing value using the calculation method provided by ES for writing routing values (this method uses the number of shards n obtained in step S32 to ensure that the calculated shard code is in the range of 0 to n-1). Repeat this calculation until the resulting set of shard codes is equal to the total number of shards.

[0096] Maintain the relationship between the calculated routing value and the fragment code, including but not limited to storing it in the system memory for use by subsequent programs.

[0097] In step S35, the ES query and standby client establish a corresponding relationship table according to the corresponding relationship between the routing value and the fragment code, and send the corresponding relationship table to the ES client.

[0098] Step S36: The ES client receives the data write request and obtains the write data in the data write request.

[0099] In step S37, the ES client selects a routing value from a pre-stored routing value table and configures it for the write data, wherein the routing value table includes a plurality of routing values.

[0100] In step S38, the ES client searches for the corresponding fragment code in the corresponding relationship table according to the configured routing value, and the corresponding relationship table includes the corresponding relationship between each routing value and each fragment code.

[0101] In step S39, the ES client determines the corresponding shard according to the queried shard code, and sends the write data to the node where the corresponding shard is located for writing.

[0102] The system has already obtained the routing value and shard code, and the relationship between the shard code and the node where the shard is located is also known, that is, the relationship between the routing value and the corresponding node is determined.

[0103] An ES client is also pre-established in the system. This client is mainly responsible for initiating write requests to the ES cluster with a specified route and a specified node. For each write request, a routing value is given to specify that all data in the request is written to a shard. Based on the relationship between the routing value and the corresponding node, a batch data write request can be initiated for the specified ES node. Since the routing value set in the request is always related to the ES node to which the request is sent, the final storage node of all data in the write request is the same as the object node that initiated the request. After receiving the request, the ES cluster will no longer distribute and occupy I / O resources on its own, which also fundamentally solves the long-tail problem of data writing.

[0104] The data writing method of the present invention is described below with a specific example, which includes the following steps:

[0105] 1. Establish an ES query and backup client: Establish a query and backup client in the system. This client is mainly responsible for querying the status of the ES cluster and retrying write requests after a write request fails to be sent to the specified ES node using the routing value.

[0106] 2. Query and record the shard distribution of the ES index: Using the query and backup client created in step 1, send a query request to the ES cluster to find the relationship between the shards and the ES node names for the index you want to write to. This can include, but is not limited to, using the official ES HTTP API: GET_cat / shards / alias? v. This request will return all shards for the index represented by the alias and their node relationships, as well as the current number of shards for the index.

[0107] Maintain information such as the relationship between the queried shard and the node it is on, and the number of shards n, including but not limited to storing it in the system's memory for use by subsequent programs.

[0108] 3. Calculate the relationship between routing value and shard code: Based on the shard number information obtained in step 2, calculate the relationship between routing value and shard code. The result is saved in the system. The calculation method includes, but is not limited to:

[0109] Preset the routing value to a range of 0 to a sufficiently large integer. Iterate through the integers within the preset range and calculate the corresponding shard code for each routing value using the calculation method provided by ES for writing routing values (this method uses the number of shards n obtained in step S32 to ensure that the calculated shard code is in the range of 0 to n-1). Repeat this calculation until the resulting set of shard codes is equal to the total number of shards.

[0110] Maintain the relationship between the calculated routing value and the fragment code, including but not limited to storing it in the system memory for use by subsequent programs.

[0111] At this point, through steps 2 and 3, the system has obtained the relationship between the routing value and the shard code, and the relationship between the shard code and the node where the shard is located, that is, the relationship between the routing value and the corresponding node.

[0112] 4. Establish an ES client capable of initiating requests to a specific ES node: Establish an ES client in the system. This client is primarily responsible for initiating write requests to the ES cluster for a specified route and node. When creating this client, write and implement a custom node selector for this client. This allows it to specify the node address based on the passed parameters and initiate requests to the specified node name, rather than randomly or in rotation. This ensures that there is an ES client in the system capable of initiating requests to specific ES nodes.

[0113] 5. Specify the routing value and node and initiate a write request: Using the ES client from step 4, initiate a batch data write request to the specified ES node based on the relationship between the routing value and the corresponding node calculated by the query and the backup client. The routing value set in this request is always related to the ES node to which the request is sent. This ensures that all data in the write request is ultimately stored on the same node as the node to which the request was initiated. This prevents the ES cluster from automatically distributing and occupying I / O resources after receiving the request, fundamentally solving the long-tail problem of data writes.

[0114] Furthermore, based on the ES client described in step 1, the system will update the shard, routing, and node information in ES as needed. This includes, but is not limited to, executing steps 2 and 3 on a scheduled basis or based on message monitoring to maintain accurate cluster-related information.

[0115] Furthermore, this system also incorporates a fault-tolerance mechanism. If the write request in step 5 fails due to a specified node being offline or failing, the system will retry the data batch using the ES query and backup client established in step 1. The node selector implementation in this client includes, but is not limited to, the default node selector implementation in the official ES client. In this case, writes no longer specify routing values or nodes, and the write method remains consistent with the basic ES write method, ensuring data security and integrity.

[0116] Furthermore, based on the ES client described in step 4, when selecting the routing value and its corresponding request node, load balancing needs to be considered, including but not limited to random, round-robin selection, and the like.

[0117] See also Figure 4 , a data writing device based on a designated route in a fourth embodiment of the present invention, comprising:

[0118] The acquisition module 41 is configured to receive a data write request and acquire the write data in the data write request;

[0119] A configuration module 42 is configured to select a routing value from a pre-stored routing value table and configure it for the write data, wherein the routing value table includes a plurality of routing values;

[0120] A query module 43 is configured to query a corresponding fragment code in a correspondence table according to a configured routing value, wherein the correspondence table includes a correspondence between each routing value and each fragment code;

[0121] The first sending module 44 is configured to determine a corresponding shard according to the queried shard code, and send the write data to the node where the corresponding shard is located for writing.

[0122] Furthermore, the data writing device further includes:

[0123] A second sending module is configured to send a query request containing at least one index alias to the ES cluster, so that the ES cluster queries all shards under the index alias based on the query request and returns the allocation code of each shard;

[0124] A determination module, configured to determine the number of the shards according to the shard codes of the shards returned by the ES cluster;

[0125] a calculation module, configured to sequentially traverse the values of each positive integer within a preset value range in ascending order, and perform slice code calculation based on the currently traversed value and the number of slices to obtain a slice code corresponding to the value, and stop the slice code calculation when the number of calculated slice codes equals the number of slices;

[0126] a routing table establishing module, configured to use the numerical value corresponding to each of the fragment codes as the routing value, and establish the routing value table according to each of the routing values;

[0127] The corresponding relationship establishing module is used to establish the corresponding relationship table according to the corresponding relationship between the routing value and the fragment code.

[0128] The data writing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding content in the aforementioned method embodiment.

[0129] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0130] In addition, combined Figures 1 to 2 The data writing method described in the embodiments of the present application is mainly implemented by an electronic device, such as a computer device, a server, etc.

[0131] Figure 5 FIG. 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. The electronic device may include a processor 10 and a memory 20 storing a computer program 30 .

[0132] Specifically, the processor 10 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0133] Among them, the memory 20 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 20 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 20 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 20 may be inside or outside the data processing device. In a specific embodiment, the memory 20 is a non-volatile memory. In a specific embodiment, the memory 20 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0134] The memory 20 may be used to store or cache various data files required for processing and / or communication, as well as possible computer programs 30 executed by the processor 10 .

[0135] The processor 10 implements the data writing method in the above embodiment by reading and executing the computer program 30 stored in the memory 20 .

[0136] In some embodiments, the data providing terminal and platform may further include a communication interface and a bus. Figure 5 As shown, the processor 10, the memory 20, and the communication interface are connected via a bus and communicate with each other.

[0137] The communication interface is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of the present application. The communication interface can also enable data communication with other components such as external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0138] A bus consists of hardware, software, or both, and couples components of corresponding devices together. Buses include, but are not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. By way of example and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of the above. Where appropriate, a bus may include one or more buses. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0139] In addition, in conjunction with the data writing method in the above embodiments, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores a computer program 30; when the computer program 30 is executed by a processor, it implements any of the data writing methods in the above embodiments.

[0140] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data writing method based on a specified route, characterized in that: include: receiving a data write request and obtaining write data in the data write request; Selecting a routing value from a pre-stored routing value table and configuring it to the write data, wherein the routing value table includes a plurality of routing values; According to the configured routing value, the corresponding fragment code is searched in the corresponding relationship table, wherein the corresponding relationship table includes the corresponding relationship between each routing value and each fragment code; Determine the corresponding shard according to the queried shard code, and send the write data to the node where the corresponding shard is located for writing; The step of receiving a data write request also includes: Send a query request containing at least one index alias to the ES cluster, so that the ES cluster queries all shards under the index alias based on the query request and returns the allocation code of each shard; Determine the number of shards according to the shard code of each shard returned by the ES cluster; Traversing the values of each positive integer within a preset value range in ascending order, and performing slice code calculation based on the currently traversed value and the number of slices to obtain the slice code corresponding to the value, and stopping the slice code calculation when the number of calculated slice codes is equal to the number of slices; Using the numerical value corresponding to each of the fragment codes as the routing value, and establishing the routing value table according to each of the routing values; The corresponding relationship table is established according to the corresponding relationship between the routing value and the fragment code.

2. The data writing method according to claim 1, wherein: The step of performing slice code calculation according to the currently traversed value and the number of slices to obtain the slice code corresponding to the value includes: A modulo operation is performed based on the currently traversed value and the number of slices to obtain the slice code corresponding to the value.

3. The data writing method according to claim 1, wherein: The step of sending a query request including at least one index alias to the ES cluster includes: Send a query request containing at least one index alias to the ES cluster periodically or based on the message monitoring mechanism.

4. The data writing method according to claim 1, wherein: The step of selecting a routing value from a pre-stored routing value table and configuring it for writing data includes: A routing value is selected from a pre-stored routing value table according to a random selection rule or a cyclic selection rule and configured to the write data.

5. The data writing method according to claim 1, wherein: After the step of sending the write data to the node where the corresponding shard is located for writing, the step further includes: When the write data fails to be written in the node, the write data is sent to other nodes for writing.

6. A data writing device based on a specified route, characterized in that: include: An acquisition module, configured to receive a data write request and acquire write data in the data write request; A configuration module, configured to select a routing value from a pre-stored routing value table and configure it to the write data, wherein the routing value table includes a plurality of routing values; A query module, configured to query a corresponding fragment code in a correspondence table according to a configured routing value, wherein the correspondence table includes a correspondence between each routing value and each fragment code; A first sending module is used to determine the corresponding shard according to the queried shard code, and send the write data to the node where the corresponding shard is located for writing; The step of receiving a data write request also includes: Send a query request containing at least one index alias to the ES cluster, so that the ES cluster queries all shards under the index alias based on the query request and returns the allocation code of each shard; Determine the number of shards according to the shard code of each shard returned by the ES cluster; Traversing the values of each positive integer within a preset value range in ascending order, and performing slice code calculation based on the currently traversed value and the number of slices to obtain the slice code corresponding to the value, and stopping the slice code calculation when the number of calculated slice codes is equal to the number of slices; Using the numerical value corresponding to each of the fragment codes as the routing value, and establishing the routing value table according to each of the routing values; The corresponding relationship table is established according to the corresponding relationship between the routing value and the fragment code.

7. The data writing device according to claim 6, wherein: Also includes: The second sending module is used to send a query request containing at least one index alias to the ES cluster, so that the ES cluster queries all shards under the index alias based on the query request and returns the allocation code of each shard; A determination module, configured to determine the number of the shards according to the shard codes of the shards returned by the ES cluster; a calculation module, configured to sequentially traverse the values of each positive integer within a preset value range in ascending order, and perform slice code calculation based on the currently traversed value and the number of slices to obtain a slice code corresponding to the value, and stop the slice code calculation when the number of calculated slice codes equals the number of slices; a routing table establishing module, configured to use the numerical value corresponding to each of the fragment codes as the routing value, and establish the routing value table according to each of the routing values; The corresponding relationship establishing module is used to establish the corresponding relationship table according to the corresponding relationship between the routing value and the fragment code.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

Images