Data writing method, device and terminal

Abstract

The application discloses a data writing method, device and terminal, and the method comprises the following steps: a terminal acquires or generates at least one index request; the terminal fills each index request into a target request block; the terminal stores each target request block into a to-be-processed queue; and the terminal acquires a bulk request from the to-be-processed queue and submits the bulk request to an ES cluster. The technical scheme of the application can realize that a coordination node forwards a bulk request to a target node based on one-time forwarding at the ES cluster end, and in the realization process, any configuration change is not needed for the ES cluster, and only a small amount of adjustment of the original writing program is needed to complete the implementation, so that the application is more suitable for writing a large amount of data into an ES index in quasi-real time, and the writing efficiency can be obviously improved when data needs to be written into multiple different indexes of an ES cluster at the same time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of database, and particularly relates to a data writing method, device and terminal. BACKGROUND

[0002] At present, when a large amount of data is written into Elasticsearch (ES for short), the Bulk mode is recommended, and the ES also provides a Bulk Processor tool for developers to solve the problem that the number of bulk requests cannot be controlled. Specifically, the ES cluster processes a bulk request based on the following mode. The terminal submits a bulk request to a certain node (randomly) in the ES cluster, the node groups all requests according to the target index of each request in the bulk request, and then determines the target shard of the index in each group. The idea of determining the shard is to take the hash value according to the random identity document (id for short) field if the root (Routing) field is not specified, and determine the index master shard sequence number (0 starts) by taking the modulus of the index shard number. Figure 1 As shown in the figure, a bulk request is finally forwarded to multiple nodes in the cluster for separate processing, and the nodes that receive the forwarded request need to return a response to the node that initially accepted the request after completing the processing, and then return a response to the terminal program. At this point, the ES completes a bulk request.

[0003] However, the prior art has the problem that some indexes cannot be written into the ES cluster in time due to the large log flow, the log is accumulated in Kafka after analysis, the writing rate is low, and the delay is large. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, one object of the present application is to provide a data writing method, device and terminal.

[0005] In order to solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0006] A data writing method, comprising:

[0007] The terminal acquires or generates at least one index request;

[0008] The terminal fills each index request into a target request block;

[0009] The terminal stores each target request block into a to-be-processed queue;

[0010] The terminal acquires a bulk request from the to-be-processed queue and submits it to the ES cluster.

[0011] Optionally, the terminal fills each of the index requests into a target request block, including:

[0012] The terminal determines the target request block matched with each of the index requests based on the target index of the index request and the number of master shards of the target index, and fills the index request into the target request block.

[0013] The size of each of the target request blocks is consistent.

[0014] Optionally, the terminal determines the target request block matched with each of the index requests based on the target index of the index request and the number of master shards of the target index, including:

[0015] When the number of master shards of the target index is equal to a threshold value, the target request block is determined according to the node name corresponding to the master shard; or when the number of master shards of the target index is greater than the threshold value, the request block to be filled is determined according to the target index name, and the routing information of the index request is set according to the routing information of the determined request block.

[0016] Optionally, after the terminal fills each of the index requests into a target request block, the terminal further includes:

[0017] When the number of master shards of the target index is equal to a threshold value, if one of the target request blocks is filled, the terminal creates a new request block.

[0018] Optionally, after the terminal fills each of the index requests into a target request block, the terminal further includes:

[0019] When the number of master shards of the target index is greater than the threshold value, if one of the target request blocks is filled, the terminal creates a new request block, and determines the routing information of the request block according to the number of master shards of the target index.

[0020] Optionally, the terminal stores each of the target request blocks into a to-be-processed queue, including:

[0021] The terminal determines a target node matched with each of the target request blocks based on the current state of the ES cluster.

[0022] The terminal determines the to-be-processed queue matched with each of the target request blocks based on the target index and the master shard of each of the target request blocks.

[0023] The target request block is stored into the to-be-processed queue of the target node.

[0024] Optionally, the terminal stores each of the target request blocks into a to-be-processed queue, and the method further comprises:

[0025] When the state of the target request block is not full, the target request block is not cached, and the filling of the target request block is continued until the state of the target request block is full, and then the target request block is cached.

[0026] Optionally, the terminal obtains a bulk request from the to-be-processed queue and submits the bulk request to an ES cluster, and the method further comprises:

[0027] A preset number of the target request blocks are obtained from the to-be-processed queue each time;

[0028] The preset number of the target request blocks are assembled to form a bulk request;

[0029] The bulk request is submitted to the ES cluster;

[0030] The preset number is determined based on the size of the bulk request.

[0031] Optionally, the assembling of the preset number of the target request blocks to form a bulk request comprises:

[0032] The preset number of the target request blocks are assembled to obtain an assembly block;

[0033] The size of the assembly block is compared with a preset value of the size of the bulk request;

[0034] If the size of the assembly block is smaller than the preset value, a plurality of target request blocks of other to-be-processed queues are obtained, and the plurality of target request blocks of other to-be-processed queues are assembled into the assembly block until the size of the assembly block is equal to the preset value.

[0035] Optionally, after the bulk request is submitted to the ES cluster, the method further comprises:

[0036] The ES cluster determines a node that submits the bulk request as a coordination node, and the coordination node is used for processing the bulk request;

[0037] The coordination node determines a target node of the bulk request based on the target index and routing information of each index request in the bulk request;

[0038] The coordination node forwards the bulk request to the target node for writing based on the routing information through one forwarding.

[0039] The embodiment of the present application further provides a data writing device, comprising:

[0040] A triggering module is configured to acquire or generate at least one index request by the terminal;

[0041] A filling module is configured to fill each of the index requests into a target request block by the terminal;

[0042] A storage module is configured to store each of the target request blocks into a to-be-processed queue by the terminal;

[0043] A submission module is configured to acquire a bulk request from the to-be-processed queue by the terminal and submit the bulk request to an ES cluster.

[0044] The embodiment of the present application further provides a data writing terminal, comprising:

[0045] A cluster state synchronizer is configured to initialize node information and routing state information of an ES cluster;

[0046] A request aggregation container is configured to fill each of the acquired index requests into a target request block according to a target index of the index request and a primary shard of the target index;

[0047] A request submission thread is configured to submit a bulk request assembled based on the target request blocks in the request aggregation container to the ES cluster.

[0048] The embodiment of the present application further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the method as described above when executing the computer program.

[0049] The embodiment of the present application further provides a computer readable storage medium, comprising a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the method as described above when the computer program runs.

[0050] The embodiment of the present application has the following technical effects:

[0051] The above technical solution of the present application can realize that the coordination node forwards a bulk request to a target node based on one-time forwarding at the ES cluster end, and in the implementation process, no configuration change is needed for the ES cluster, and only a small amount of adjustment of the original writing program is needed to complete the implementation, which is more suitable for writing large amounts of data into an ES index in quasi-real time, and can significantly improve the writing efficiency, especially when data needs to be written into multiple different indexes of an ES cluster at the same time.

[0052] 2) The state table information and other important cluster meta information of the target ES cluster are periodically synchronized by the cluster state synchronizer (RequestEmitter) to guide the packaging target request block process of the index request.

[0053] 3) The efficiency of writing log data into ES is significantly improved, and the problem of log accumulation in Kafka after parsing due to large log traffic of part of the index and unable to be written into ES in time is solved.

[0054] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is the principle diagram of the existing ES processing bulk request;

[0056] Figure 2 is the principle diagram of the ES processing bulk request provided by the embodiment of the present application;

[0057] Figure 3 is the flow diagram of the data writing method provided by the embodiment of the present application;

[0058] Figure 4 is the schematic diagram of the existing writing throughput (TPS);

[0059] Figure 5 is the schematic diagram of the writing throughput (TPS) provided by the embodiment of the present application;

[0060] Figure 6 is the structural diagram of the data writing device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0061] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0062] First, in order to facilitate the understanding of the embodiments by those skilled in the art, some terms are explained:

[0063] (1) Elasticsearch search engine

[0064] The search engine is a retrieval technology of searching information from the Internet according to user demand and a certain algorithm and using a specific strategy to feed back the information to the user. The search engine relies on various technologies, such as retrieval and sorting technology, big data processing technology, natural language processing technology, to provide fast and highly relevant information services for information retrieval users. ES is a relatively mature distributed search engine in the open source field, and has a wide range of use scenarios in information retrieval requirements; for example, real-time query analysis of log data.

[0065] (2) Message middleware

[0066] Message middleware is a supporting software system that provides synchronous or asynchronous and reliable message transmission for application systems in a network environment based on queue and message passing technology, has functions such as peak shaving, asynchronous decoupling, and generally has a message retry function. For example: Kafka, etc.

[0067] (3) Flink

[0068] An open source stream processing framework, the core of which is a distributed stream data flow engine written. Flink executes any stream data program in a data-parallel and pipelined manner, and the pipelined runtime system of Flink can execute batch processing and stream processing programs.

[0069] (4) Polling

[0070] It is a processing flow of the base station allocating bandwidth for the terminal. Such allocation can be for a single terminal or a group of terminals. Allocating bandwidth for a single terminal and a group of terminal connections actually defines a bandwidth request competition mechanism. Such allocation is not a single message, but a series of allocation mechanisms contained in the uplink mapping message.

[0071] Secondly, based on the corresponding application scenarios of the present application, the embodiments of the present application are also understood.

[0072] As shown in Figure 2 , the embodiments of the present application provide a data writing terminal, comprising:

[0073] A cluster state synchronizer (RequestEmitter) is configured to initialize node information and routing state information of an ES cluster;

[0074] A request aggregation container (RequestAccumulator) is configured to fill the obtained index request to the target request block according to a target index of the index request and a master shard of the target index;

[0075] A request submission thread (RequestEmitter) is configured to assemble the target request blocks in the request aggregation container based on the request into a bulk request and submit the bulk request to the ES cluster.

[0076] As shown in the actual application scenario, when the terminal is initialized, the cluster state synchronizer (RequestEmitter) acquires ES cluster meta information such as a routing table of the ES cluster, and periodically synchronizes the state table information of the target ES cluster to guide the packaging target request block process of the index request. Figure 2

[0077] Specifically, for example, the ES cluster can include five nodes, namely, a first node, a second node, a third node, a fourth node, and a fifth node.

[0078] The request submission thread (RequestEmitter) acquires a preset number of request blocks from the request aggregation container (RequestAccumulator), assembles the request blocks into a first bulk request, and then submits the first bulk request to the ES cluster.

[0079] According to the routing information, the first node is taken as a target node to receive the first bulk request forwarded by the coordination node once, and the first bulk request is written and then a first response is sent to the Bulk Processor.

[0080] According to the routing information, the fifth node can also be taken as a target node to receive the second bulk request forwarded by the coordination node once, and the first bulk request is written and then a second response is sent to the Bulk Processor.

[0081] Further, the log data in the log service is saved in the ES, the original log is temporarily stored in the Kafka after being parsed and converted after being reported, and the parsed log data is consumed by the Flink program, and the terminal writes the log belonging to the application information into the target ES index for user search and analysis.

[0082] The embodiments of the present application are used to improve the efficiency of data writing to the ES cluster, solve the problem of large delay of application log writing to the ES cluster without increasing the resources of the ES cluster.

[0083] As shown in the actual application scenario, when the terminal is initialized, the cluster state synchronizer (RequestEmitter) acquires ES cluster meta information such as a routing table of the ES cluster, and periodically synchronizes the state table information of the target ES cluster to guide the packaging target request block process of the index request. Figure 3

[0084] Step S1: The terminal acquires or generates at least one index request.

[0085] ​​Specifically, the user triggers an index request based on the terminal, and then the terminal obtains the user's index request or assembles and generates an index request.

[0086] Further, the terminal can be any intelligent device with the above functions.

[0087] For example, the terminal obtains a large amount of log data based on the user, such as various insurance data generated by the platform.

[0088] Step S2: The terminal fills each index request into a target request block.

[0089] Specifically, the terminal fills each index request into a target request block, including:

[0090] The terminal determines the target request block matched with each index request based on the target index of the index request and the number of primary shards of the target index, and fills the index request into the target request block.

[0091] Wherein, the size of each target request block is consistent.

[0092] It should be noted that the size of the target request block of the embodiment of the present application is a fixed size; the size of the target request block can be determined based on a preset parameter.

[0093] Further, the terminal determines the target request block matched with each index request based on the target index of the index request and the number of primary shards of the target index, including:

[0094] When the number of primary shards of the target index is equal to a threshold value, the target request block is determined according to the node name corresponding to the primary shard; or when the number of primary shards of the target index is greater than the threshold value, the request block to be filled is determined according to the target index name, and the routing information of the index request is set according to the routing information of the determined request block.

[0095] Wherein, in the embodiment of the present application, the threshold value is equal to 1, that is, before determining the target request block to be filled, the number of primary shards corresponding to the target index of the index request is compared with the threshold value (1).

[0096] In an actual application scenario, when the number of master shards of the target index is 1, the target request block to be filled is determined according to the node name of the node where the No. 0 master shard is located, and then the index request is filled into the target request block; when the number of master shards of the target index is greater than 1 (for example, 5), the master shard to which the current index request belongs is determined in a polling manner, that is, the target master shard, and then the target request block to be filled is determined according to the node name of the node where the target master shard is located, and the current index request is filled into the target request block.

[0097] Further, after the terminal fills each index request into a target request block, the terminal further includes:

[0098] When the number of master shards of the target index is equal to a threshold value, if one target request block is filled, the terminal creates a new request block.

[0099] Further, after the terminal fills each index request into a target request block, the terminal further includes:

[0100] When the number of master shards of the target index is greater than a threshold value, if one target request block is filled, the terminal creates a new request block, and determines the routing information of the request block according to the number of master shards of the target index.

[0101] Step S3: The terminal stores each target request block into a to-be-processed queue.

[0102] Specifically, the terminal stores each target request block into a to-be-processed queue, including:

[0103] The terminal determines a target node matched with the target index or the master shard of each target request block according to the current state of the ES cluster.

[0104] The terminal determines the to-be-processed queue matched with the target request block according to the target index and the master shard of each target request block.

[0105] The target request block is stored into the to-be-processed queue of the target node.

[0106] The terminal selects a root field, that is, a Routing field, for each target request block, and the value of the root field can ensure that the node where the shard is located is the current target node when the ES processes the index request according to the value of the root field.

[0107] Further, the terminal stores each target request block into a to-be-processed queue, and further includes:

[0108] When the state of the target request block is not filled, the target request block is not cached, and the filling of the target request block is continued until the state of the target request block is filled, and then the target request block is cached.

[0109] Step S4: the terminal obtains a bulk request from the to-be-processed queue and submits it to the ES cluster.

[0110] The embodiment of the application can realize that the coordination node forwards a bulk request to a target node based on one-time forwarding at the ES cluster end. In the implementation process, no configuration change is needed for the ES cluster, and only a small amount of adjustment of the original write program is needed to complete the implementation. The embodiment is suitable for writing large amounts of data into an ES index in real time, and can significantly improve the writing efficiency, especially when data needs to be written into multiple different indexes in an ES cluster at the same time.

[0111] Specifically, the terminal obtains a bulk request from the to-be-processed queue and submits it to the ES cluster, including:

[0112] A preset number of target request blocks are obtained from the to-be-processed queue each time;

[0113] The preset number of target request blocks are assembled to form a bulk request;

[0114] The bulk request is submitted to the ES cluster;

[0115] The preset number is determined based on the size of the bulk request.

[0116] Specifically, the request submission thread obtains a bulk request from the to-be-processed queue of the target node, wherein the request submission thread can determine the order of selecting the target node according to the priority of data transmission and the like.

[0117] In an actual application scenario, after a target node is determined, the request submission thread assembles the target request blocks of the target node, and then forms a bulk request.

[0118] Further, the assembling of the preset number of target request blocks to form a bulk request includes:

[0119] The preset number of target request blocks are assembled to obtain an assembly block;

[0120] The size of the assembly block is compared with a preset value of the size of the bulk request;

[0121] If the size of the assembly block is less than the preset value, a plurality of target request blocks of other to-be-processed queues are obtained, and the plurality of target request blocks of other to-be-processed queues are assembled into the assembly block until the size of the assembly block is equal to the preset value.

[0122] In an actual application scenario, the terminal can determine the size of the target request block and the size of the preset value of the bulk request and the Bulk Processor concurrency degree and other parameters through pressure testing.

[0123] After the above parameters are determined, since the size of each target request block is fixed, the number of target request blocks required for assembling a bulk request is also fixed, and therefore, the preset value can be the actual size of a bulk request or the number of target request blocks (for example, 1000).

[0124] Specifically, the embodiment of the application determines the size of the preset value based on the actual size of the bulk request.

[0125] After the request submission thread completes the first assembly of the current bulk request, the size of the assembly block is compared with the preset value, and if the size of the assembly block is greater than or equal to the preset value.

[0126] If the size of the assembly block is less than or equal to the preset value, the request submission thread obtains a plurality of target request blocks in other to-be-processed queues of the target node, and assembles the plurality of target request blocks in other to-be-processed queues into the assembly block, and the above process of increasing the size of the assembly block is repeated until the size of the loop block is equal to the preset value.

[0127] Further, after the bulk request is submitted to the ES cluster, the following steps are included.

[0128] The ES cluster determines a node that submits the bulk request as a coordination node, and the coordination node is used to process the bulk request.

[0129] The coordination node determines a target node of the bulk request based on a target index and routing information of each index request in the bulk request.

[0130] The coordination node forwards the bulk request to the target node for writing based on the routing information through one forwarding.

[0131] As shown in the embodiment of the application, as shown in Figure 4 and Figure 5 , wherein the abscissa represents time, the ordinate represents the index rate, the ES cluster scale is a cluster of 5 nodes (16 cores and 64G memory disks are solid state disks (SSD)), and specifically, as shown in Figure 4The existing write TPS is shown, and the write TPS average is 100000 / s, as shown in the figure. Figure 5 The write TPS of the embodiment of the application is shown, and the write TPS average is 150000 / s. It can be seen that the write TPS of the embodiment of the application increases by about 50% compared with the existing write TPS. Therefore, the embodiment of the application significantly improves the efficiency of writing log data to ES, and solves the problem that the parsed log is accumulated in Kafka and cannot be written to ES in time due to the large log flow of part of the index.

[0132] In an optional embodiment of the application, the size of the ES cluster index cache is adjusted, the transaction log Translog is asynchronously flushed to the disk, the update refresh time period of the index is appropriately adjusted to a larger value (the default is 1s), and the like, so that the performance of the ES cluster index reaches a higher performance.

[0133] As shown in the figure, Figure 6 The embodiment of the application further provides a data writing device, which comprises:

[0134] The triggering module 601 is configured to acquire or generate at least one index request by the terminal.

[0135] The filling module 602 is configured to fill each index request to a target request block by the terminal.

[0136] The storage module 603 is configured to store each target request block to a to-be-processed queue by the terminal.

[0137] The submission module 604 is configured to acquire a bulk request from the to-be-processed queue by the terminal, and submit the bulk request to an ES cluster.

[0138] Optionally, the terminal fills each index request to a target request block, which comprises:

[0139] The terminal determines the target request block matched with each index request based on the target index of the index request and the number of primary shards of the target index, and fills the index request to the target request block.

[0140] The size of each request block is consistent.

[0141] Optionally, the terminal determines the target request block matched with each index request based on the target index of the index request and the number of primary shards of the target index, which comprises:

[0142] When the number of the master shards of the target index is equal to the threshold value, the terminal determines the target request block according to the node name corresponding to the master shard; or when the number of the master shards of the target index is greater than the threshold value, the terminal determines the request block to be filled according to the target index name, and sets the routing information of the index request according to the routing information of the determined request block.

[0143] Optionally, after the terminal fills each index request into a target request block, the terminal further comprises:

[0144] When the number of the master shards of the target index is equal to the threshold value, if one target request block is filled, the terminal newly creates a request block.

[0145] Optionally, after the terminal fills each index request into a target request block, the terminal further comprises:

[0146] When the number of the master shards of the target index is greater than the threshold value, if one target request block is filled, the terminal newly creates a request block, and determines the routing information of the request block according to the number of the master shards of the target index.

[0147] Optionally, the terminal stores each target request block into a to-be-processed queue, comprising:

[0148] The terminal determines a target node matched with the target index or the master shard of each target request block according to the current state of the ES cluster.

[0149] The terminal determines the to-be-processed queue matched with the target request block according to the target index and the master shard of each target request block.

[0150] The terminal stores the target request block into the to-be-processed queue of the target node.

[0151] Optionally, the terminal stores each target request block into a to-be-processed queue, further comprising:

[0152] When the state of the target request block is not filled, the terminal does not cache the target request block, and continues to fill the target request block, until the state of the target request block is filled, and then the terminal caches the target request block.

[0153] Optionally, the terminal obtains a bulk request from the to-be-processed queue at a time, and submits the bulk request to the ES cluster, comprising:

[0154] Each time, a preset number of target request blocks are obtained from the to-be-processed queue.

[0155] Assemble the preset number of the target request blocks to form a bulk request;

[0156] Submit the bulk request to the ES cluster;

[0157] The preset number is determined based on the size of the bulk request.

[0158] Optionally, the assembling the preset number of the target request blocks to form a bulk request comprises:

[0159] Assemble the preset number of the target request blocks to obtain an assembled block;

[0160] Compare the size of the assembled block with a preset value of the size of the bulk request;

[0161] If the size of the assembled block is smaller than the preset value, obtain a plurality of target request blocks of other to-be-processed queues, and assemble the plurality of target request blocks of other to-be-processed queues to the assembled block until the size of the assembled block is equal to the preset value.

[0162] Optionally, after the bulk request is submitted to the ES cluster, the method further comprises:

[0163] The ES cluster determines a node that submits the bulk request as a coordination node; wherein the coordination node is used for processing the bulk request;

[0164] The coordination node determines a target node of the bulk request based on a target index and routing information of each index request in the bulk request;

[0165] The coordination node forwards the bulk request to the target node for writing based on the routing information through one forwarding.

[0166] Embodiments of the present application also provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the method as described above when executing the computer program.

[0167] Embodiments of the present application also provide a computer readable storage medium, comprising a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the method as described above when the computer program runs.

[0168] In addition, other configurations and functions of the device of the embodiments of the present application are known to those skilled in the art, and to reduce redundancy, they are not described here.

[0169] It should be noted that the logical and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a product of the manufacturing and / or processing, and / or a combination of such products. The computer-readable medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via the optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in the computer memory.

[0170] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0171] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0172] In the description of the application, it should be understood that the orientation or positional relationship indicated by terms such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the purpose of facilitating the description of the application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the application.

[0173] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly specified and limited.

[0174] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0175] In the present application, unless otherwise explicitly specified and limited, the first feature "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature "above", "over" and "on" the second feature can be that the first feature is directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "under" and "under" the second feature can be that the first feature is directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0176] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A data write method, characterized by, The method comprises: a terminal acquires or generates at least one index request; the terminal fills each index request into a target request block; the terminal stores each target request block into a pending queue; the terminal acquires a bulk request from the pending queue and submits it to an ES cluster; the terminal fills each index request into a target request block, comprising: the terminal determines the target request block matched with each index request based on the target index of the index request and the number of primary shards of the target index, and fills the index request into the target request block; wherein the size of each target request block is consistent; the terminal determines the target request block matched with each index request based on the target index of the index request and the number of primary shards of the target index, comprising: when the number of primary shards of the target index is equal to a threshold, the target request block is determined according to the node name corresponding to the primary shard; or when the number of primary shards of the target index is greater than the threshold, the request block to be filled is determined according to the target index name, and the routing information of the index request is set according to the routing information of the determined request block; the terminal stores each target request block into a pending queue, comprising: the terminal determines the target node matched with the target index or the primary shard of each target request block according to the current state of the ES cluster; the terminal determines the pending queue matched with the target request block according to the target index and the primary shard of each target request block; the target request block is stored into the pending queue of the target node; the terminal acquires a bulk request from the pending queue and submits it to an ES cluster, comprising: each time, a preset number of target request blocks are acquired from the pending queue; the preset number of target request blocks are assembled to form a bulk request; the bulk request is submitted to the ES cluster; wherein the preset number is determined based on the size of the bulk request; the preset number of target request blocks are assembled to form a bulk request, comprising: the preset number of target request blocks are assembled to obtain an assembly block; the size of the assembly block is compared with a preset value of the size of the bulk request; if the size of the assembly block is less than the preset value, then a plurality of target request blocks of other pending queues are acquired and assembled into the assembly block until the size of the assembly block is equal to the preset value.

2. The method of claim 1, wherein, after the terminal fills each index request into a target request block, it further comprises: when the number of primary shards of the target index is equal to a threshold, if a target request block is filled, the terminal creates a new request block.

3. The method of claim 1, wherein, after the terminal fills each index request into a target request block, it further comprises: When the number of the master shards of the target index is greater than a threshold, if one of the target request blocks is filled, the terminal creates a new request block, and determines routing information of the request block according to the number of the master shards of the target index.

4. The method of claim 1, wherein, The terminal stores each of the target request blocks into a to-be-processed queue, and the method further includes: When the target request block is not filled, the terminal does not cache the target request block, and continues to fill the target request block until the target request block is filled, and then caches the target request block.

5. The method of claim 1, wherein, After the bulk request is submitted to the ES cluster, the method further includes: The ES cluster determines a node that submits the bulk request as a coordination node, wherein the coordination node is used to process the bulk request, and the coordination node determines a target node of the bulk request based on the target index and the routing information of each index request in the bulk request; The coordination node forwards the bulk request to the target node for writing based on the routing information through one forwarding.

6. A data writing apparatus characterized by comprising: The method includes: A triggering module is configured to acquire or generate at least one index request by a terminal; A filling module is configured to fill each of the index requests into a target request block by the terminal; A storage module is configured to store each of the target request blocks into a to-be-processed queue by the terminal; A submission module is configured to acquire a bulk request from the to-be-processed queue by the terminal, and submit the bulk request to an ES cluster; The filling module is further configured to determine the target request block matched with each of the index requests based on the target index of the index request and the number of master shards of the target index, and fill the index request into the target request block, wherein the size of each of the target request blocks is consistent; When the number of the master shards of the target index is equal to a threshold, the filling module is further configured to determine the target request block according to a node name corresponding to the master shard, or when the number of the master shards of the target index is greater than a threshold, the filling module is further configured to determine the request block to be filled according to the target index name, and set the routing information of the index request according to the routing information of the determined request block; The storage module is further configured to determine a target node matched with the target index or the master shard of each of the target request blocks according to a current state of the ES cluster, determine the to-be-processed queue matched with the target request block according to the target index and the master shard of each of the target request blocks, and store the target request block into the to-be-processed queue of the target node; A preset number of the target request blocks are acquired from the to-be-processed queue each time; The preset number of the target request blocks are assembled to form a bulk request, and the bulk request is submitted to the ES cluster, wherein the preset number is determined based on the size of the bulk request; and The coordination node forwards the bulk request to the target node for writing based on the routing information through one forwarding. The submission module is further configured to: assemble the preset number of target request blocks to obtain an assembly block; compare the size of the assembly block with a preset value of the size of the bulk request; if the size of the assembly block is smaller than the preset value, obtain a plurality of target request blocks of other to-be-processed queues, and assemble the plurality of target request blocks of other to-be-processed queues to the assembly block until the size of the assembly block is equal to the preset value.

7. A data writing terminal, characterized by comprising: Comprise: A cluster state synchronizer, configured to perform initialization processing on node information and routing state information of an ES cluster; A request aggregation container, configured to fill each index request obtained to a target request block according to a target index of the index request and a primary shard of the target index; A request submission thread, configured to submit a bulk request assembled based on the target request blocks in the request aggregation container to the ES cluster; The request aggregation container is further configured to: determine the target request block matched with each index request based on the target index of the index request and the number of primary shards of the target index, and fill the index request to the target request block; wherein the size of each target request block is consistent; The request aggregation container is further configured to: when the number of primary shards of the target index is equal to a threshold value, determine the target request block according to the node name corresponding to the primary shard; or when the number of primary shards of the target index is greater than a threshold value, determine the request block to be filled according to the target index name, and set the routing information of the index request according to the routing information of the determined request block; The request aggregation container is further configured to determine a target node matched with the target index or the primary shard of each target request block according to the current state of the ES cluster; determine a to-be-processed queue matched with the target request block according to the target index and the primary shard of each target request block; and store the target request block to the to-be-processed queue of the target node; The request submission thread is further configured to: obtain a preset number of target request blocks from the to-be-processed queue each time; assemble the preset number of target request blocks to form a bulk request; and submit the bulk request to the ES cluster; wherein the preset number is determined based on the size of the bulk request; The request submission thread is further configured to: assemble the preset number of target request blocks to obtain an assembly block; compare the size of the assembly block with a preset value of the size of the bulk request; if the size of the assembly block is smaller than the preset value, obtain a plurality of target request blocks of other to-be-processed queues, and assemble the plurality of target request blocks of other to-be-processed queues to the assembly block until the size of the assembly block is equal to the preset value.

8. An electronic device, comprising: A computer program product comprising a computer readable storage medium having stored thereon computer program means, wherein said computer program means, when executed by a processor, cause the processor to carry out the method according to any of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer program, when executed, controls a device in which the computer readable storage medium is located to carry out the method according to any of claims 1 to 5.

Images