Data writing method and device based on specified route, medium and electronic equipment
A data writing and routing technology, applied in the computer field, can solve problems such as high I/O occupancy, full I/O, ES hangs, etc., to reduce the load, improve stability, and solve the long tail problem.
Pending Publication Date: 2022-07-26
HANGZHOU ANHENG INFORMATION TECH CO LTD
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
In a cluster environment with a large amount of data and strict resource constraints, writing to the ES may occupy a high I / O, which may lead to full I / O, or even cause the ES to hang up.
In addition, sending BulkShardRequest requests is asynchronous, but BulkRequest requests need to wait for all BulkShardRequest responses before returning to the client
However, some shard requests in the index for writing will slow down the response due to memory garbage collection, faults, queues, locks, etc., resulting in the so-called long tail problem.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0047] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0048] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, the present application can also be applied to the present application according to these drawings without any creative effort. other similar situations. In addition, it will also be appreciated t...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a data write-in method and device based on a specified route, a medium and electronic equipment, and the method comprises the steps: receiving a data write-in request, and obtaining the write-in data in the data write-in request; selecting a routing value from a pre-stored routing value table and configuring the routing value to the write-in data, wherein the routing value table comprises a plurality of routing values; querying a corresponding fragment code in a corresponding relation table according to the configured routing value, wherein the corresponding relation table comprises a corresponding relation between each routing value and each fragment code; and determining a corresponding fragment according to the queried fragment code, and sending the write-in data to a node where the corresponding fragment is located to perform write-in. According to the method and the device, internal data distribution is no longer available in each ES data writing operation, the cluster I / O load is effectively reduced, and the stability of the cluster is improved. The long tail problem is solved, and the ES data writing efficiency is improved.
Description
technical field [0001] The present invention relates to the field of computer technology, and in particular, to a method, device, medium and electronic device for writing data based on a specified route. Background technique [0002] Distributed storage search engine system (ElasticSearch, ES) is a popular full-text search server with distributed multi-user capability. It is designed to be used in cloud computing and can achieve real-time search. Due to the distributed characteristics of the system, multiple servers can be used to form an ES cluster to further shard the data in the cluster index. One server can include multiple shards to jointly improve the indexing and search functions. [0003] In the related art, a bulk request method is used to write a large amount of data to the ES, and the data is submitted to the ES cluster in batches. After a batch of data is sent to the client node (Client Node) of the ES, the client node of the ES will distribute the batch of data...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/245
CPCG06F16/2282G06F16/2228G06F16/245
Inventor 滕大正范渊刘博
Owner HANGZHOU ANHENG INFORMATION TECH CO LTD