File read-write opening method and device

Abstract

The embodiment of the invention provides a file read-write opening method and device. The read-write opening method comprises: acquiring a request for a read-in flow of a target index file; judging whether a read-in stream of the target index file exists in the started read-in streams or not; if the read-in flow of the target index file does not exist and it is determined that the total number ofthe started read-in flow and the started write-out flow reaches a first preset threshold value, closing the file flow according to a preset flow closing strategy, wherein the file flow is a read-in flow or a write-out flow; starting a new read-in flow as a target read-in flow of the target index file; and marking the target read-in flow as being occupied, and opening the target index file according to the reference read of the target read-in flow. By applying the embodiment of the invention, the distribution of the handles is ensured, and the possibility of unhealthy service states of the ES caused by unreasonable use of the handles is reduced.

Description

technical field [0001] The present application relates to the technical field of data processing, in particular to a method and device for reading, writing and opening a file. Background technique [0002] ElasticSearch (hereinafter referred to as ES) is a distributed full-text search system based on Lucene, including multiple node devices. Lucene is a full-text search library. Its internal file-based architecture establishes inverted indexes for the texts to be retrieved. These indexes are stored in the form of different files as Lucene indexes. Lucene's index consists of multiple index files, and the index files are divided into metadata files and index data files. Such as figure 1 The ES shown includes multiple ES indexes such as figure 1 ES Index 1, ES Index 2, and ES Index 3 are shown. Each ES index includes multiple shards, such as figure 1 The ES index 3 shown includes Shard1, Shard2 and Shard3. A shard is an index unit, corresponding to a Lucene index, and a Lu...

AI Technical Summary

Problems solved by technology

Among them, reading and opening a file requires opening a new read-in stream, which consumes a handle, and writing and opening a file requires opening a new write-out stream, which consumes a handle

This causes a large number of handles to be consumed during the running of ES

[0004] With the increase of storage index files in ES, the number of shards created in ES and the number of ES indexes will continue to increase, and the number of handles consumed by ES to open files will continue to increase, but the number of handles is limited, which makes ES likely to reach the handle critical point

When the critical point of the number of handles is reached, if you continue to operate on the index file in ES, such as requesting to read in or write out the file, ES will not be able to allocate new handles, which will put ES in an unhealthy service state. Unable to provide normal service

Images