ES distribution perception method and device in a remote multi-center network environment

Abstract

The present invention discloses an ES allocation perception method and device in a remote multi-center network environment. A plurality of log collection processors are configured on a device log processing platform. Each log collection processor corresponds to a shard of at least one ES node. The device to be collected scores the log collection processor according to the load information of the log collection processor and the network delay from the device to the log collection processor. The log collection processor with the best network and the smallest node load is preferably allocated, which effectively reduces the time required for the device to transmit logs and reduces the disk IO occupancy time, thereby effectively improving the utilization rate of the node.

Description

Technical Field

[0001] The present invention relates to the field of ES distribution perception, and in particular to an ES distribution perception method and device in a remote multi-center network environment. Background Art

[0002] With the development of computer technology, more and more enterprises and relevant functional departments have hardware equipment to support services. In order to improve fault tolerance and high availability, large data centers will set up computer rooms in multiple locations to deal with emergencies. In the process of monitoring and managing hardware equipment, it is usually necessary to collect and analyze the log files of the equipment in order to timely discover the status changes of the equipment and understand the operating rules of the equipment. Through the processing and aggregation of data by the application platform, the investment of human resources can be reduced, and data support can be provided for the formulation of scientific management plans in the future. Due to the large amount of logs generated when the equipment is running, it brings great challenges to the storage and query of these log information. In the face of massive equipment, in order to facilitate query, Elasticsearch (ES for short) is usually used to store log information. Elasticsearch provides efficient query methods through indexing, sharding and other mechanisms, and can be easily expanded. However, by default, the writing of Elasticsearch data and the allocation of shards in the cluster are randomly allocated. In the case of multi-centers in different locations, if the data is written without considering the network delay, the program response time will be extended in some cases, which will reduce the user experience.

[0003] like Figure 1 The figure shows the current device log processing platform framework diagram. There are multiple devices that need to collect logs. The devices report the original logs to the log collection processor. The log collection processor, such as logstash, formats the text information of the logs and stores the information in the ES cluster. The log information of different devices will be randomly stored in the shards of the ES node. In a multi-center network environment in different locations, devices 1, 2...N, N+1, N+2...2N, 2N+1, 2N+2...3N may be located in different network centers. When they are located in different network centers, the network consumption of a large number of devices transmitting logs will have a great impact on the real-time performance of the data. At the same time, due to network delays and bandwidth limitations, the system load will increase and the hardware utilization rate will be reduced. Summary of the invention

[0004] To solve the above problems, the present invention provides an ES allocation perception method and device in a remote multi-center network environment, which preferentially allocates log collection processors with optimal network and minimum node load, effectively reduces the time required for device log transmission, reduces disk io occupancy time, and thus effectively improves node utilization.

[0005] In the first aspect, the technical solution of the present invention provides an ES allocation perception method in a remote multi-center network environment, configuring multiple log collection processors, each log collection processor corresponding to a shard of at least one ES node;

[0006] When allocating a log collection processor to a device to be collected, perform the following steps:

[0007] The device to be collected obtains the load information of each log collection processor, and scores it according to the load information to obtain the load score of each log collection processor;

[0008] The device to be collected obtains the network delay time from itself to each log collection processor, and scores it according to the network delay time to obtain the network delay score of each log collection processor;

[0009] Multiply the load score and the network delay score to obtain the weight value of the corresponding log collection processor;

[0010] The log collection processor with the largest weight value is selected, and the devices to be collected are allocated to the log collection processor.

[0011] Furthermore, the load information of the log collection processor obtained by the collection device includes four indicator items of data, namely CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy.

[0012] Furthermore, the load information is scored to obtain the load score of each log collection processor, including:

[0013] Step 1: Score each indicator item of the load information using the following formula;

[0014]

[0015] Among them, x is the obtained indicator item data, and α is the corresponding indicator item threshold;

[0016] Step 2: Multiply the scores of each indicator item to obtain the load score of the corresponding log collection processor.

[0017] Furthermore, the network delay time is scored to obtain the network delay score of each log collection processor. Specifically, the network delay score is calculated by the following formula:

[0018]

[0019] Wherein, y is the obtained network delay time, and β is the preset threshold.

[0020] Furthermore, the method further comprises the following steps:

[0021] Determine whether all the obtained weight values ​​are the same;

[0022] If they are the same, a log collection processor is randomly selected to allocate the device to be collected;

[0023] Otherwise, all weight values ​​are sorted, and the log collection processor with the largest weight value is selected, and the device to be collected is assigned to the log collection processor.

[0024] Furthermore, the shards of the ES node corresponding to each log collection processor are shards of the same ES node.

[0025] In a second aspect, the technical solution of the present invention provides an ES distribution perception device in a remote multi-center network environment, including a device to be collected and multiple log collection processors;

[0026] Each log collection processor corresponds to at least one shard of an ES node;

[0027] The device to be collected is configured with a distribution perception program, which includes:

[0028] Load scoring module: obtains the load information of each log collection processor, and scores it according to the load information to obtain the load score of each log collection processor;

[0029] Network delay scoring module: obtains the network delay time from itself to each log collection processor, and scores it according to the network delay time to obtain the network delay score of each log collection processor;

[0030] Weight value calculation module: multiply the load score and the network delay score to obtain the weight value of the corresponding log collection processor;

[0031] Log collection processor selection module: selects the log collection processor with the largest weight value and assigns the device to be collected to the log collection processor.

[0032] Furthermore, the load information of the log collection processor obtained by the load scoring module includes four indicator items of data, namely, CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy;

[0033] The load scoring module scores the load information and obtains the load score of each log collection processor, including:

[0034] Step 1: Score each indicator item of the load information using the following formula;

[0035]

[0036] Among them, x is the obtained indicator item data, and α is the corresponding indicator item threshold;

[0037] Step 2: Multiply the scores of each indicator item to obtain the load score of the corresponding log collection processor.

[0038] Furthermore, the network delay scoring module scores the network delay time to obtain the network delay score of each log collection processor. Specifically, the network delay score is calculated by the following formula:

[0039]

[0040] Wherein, y is the obtained network delay time, and β is the preset threshold.

[0041] Furthermore, the allocation perception program of the device configuration to be collected also includes:

[0042] Weight value judgment module: judge whether all the obtained weight values ​​are the same;

[0043] The log collection processor selection module is also used to randomly select a log collection processor to allocate the device to be collected when all the weight values ​​obtained are the same; otherwise, all weight values ​​are sorted, the log collection processor with the largest weight value is selected, and the device to be collected is allocated to the log collection processor.

[0044] The ES allocation perception method and device in a remote multi-center network environment provided by the present invention have the following beneficial effects compared with the prior art: multiple log collection processors are configured on the device log processing platform, each log collection processor corresponds to a shard of at least one ES node, and the device to be collected scores the log collection processor according to the load information of the log collection processor and the network delay from itself to the log collection processor, and preferably allocates the log collection processor with the best network and the smallest node load, effectively reducing the time required for the device to transmit logs and reducing the disk io occupancy time, thereby effectively improving the utilization rate of the node. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is a schematic diagram of the current device log processing platform framework;

[0047] Figure 2 This is a schematic diagram of the device log processing platform framework of the present invention;

[0048] Figure 3A schematic diagram of a process flow of allocating log collection processors to a device to be collected provided in the first embodiment of the present invention;

[0049] Figure 4 A schematic block diagram of the structure of the allocation perception program provided in the second embodiment of the present invention;

[0050] Figure 5 A schematic diagram of the structure of a terminal provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0051] The terms used in the present invention are explained below.

[0052] Elasticsearch: (ES for short) is an open source search engine based on Apache Lucene(TM). Whether in the open source or proprietary field, Lucene can be considered to be the most advanced, best performing, and most comprehensive search engine library to date.

[0053] Allocation awareness: It is a method provided by Elasticsearch that allows users to use custom parameters to configure the allocation of shards.

[0054] Logstash: is a free and open server-side data processing pipeline that can collect data from multiple sources, transform the data, and then send the data to a "repository".

[0055] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0056] like Figure 1 The figure shows the current device log processing platform framework diagram. There are multiple devices that need to collect logs. The devices report the original logs to the log collection processor. The log collection processor, such as logstash, formats the text information of the logs and stores the information in the ES cluster. The log information of different devices will be randomly stored in the shards of the ES node. In a multi-center network environment in different locations, devices 1, 2...N, N+1, N+2...2N, 2N+1, 2N+2...3N may be located in different network centers. When they are located in different network centers, the network consumption of a large number of devices transmitting logs will have a great impact on the real-time performance of the data. At the same time, due to network delays and bandwidth limitations, the system load will increase and the hardware utilization rate will be reduced.

[0057] Therefore, the present invention provides an ES allocation perception method and device in a remote multi-center network environment, which preferentially allocates the log collection processor with the best network and the smallest node load, effectively reduces the time required for the device to transmit logs, reduces the disk io occupancy time, and thus effectively improves the utilization rate of the node.

[0058] Embodiment 1

[0059] The first embodiment of the present invention provides an ES allocation perception method in a remote multi-center network environment. Different from the prior art that uses a general log collection processor, the device log processing platform framework of the present invention configures multiple log collection processors, each log collection processor corresponds to at least one ES node shard, so that the device to be collected can select the log collection processor with the best network and the smallest node load for allocation. Figure 2 The figure shows a schematic diagram of the device log processing platform framework of the present invention, in which devices 1 to 3N are devices to be collected, and the devices to be collected refer to devices whose log information needs to be collected.

[0060] like Figure 3 The figure shows a flow chart of the execution process of allocating log collection processors to the devices to be collected, which includes the following steps to achieve the optimization of the log collection processors.

[0061] S101, the device to be collected obtains the load information of each log collection processor, and scores according to the load information to obtain the load score of each log collection processor;

[0062] S102, the device to be collected obtains the network delay time from itself to each log collection processor, and scores according to the network delay time to obtain the network delay score of each log collection processor;

[0063] S103, multiplying the load score and the network delay score to obtain a weight value of the corresponding log collection processor;

[0064] S104: Select a log collection processor with the largest weight value, and assign the device to be collected to the log collection processor.

[0065] The first embodiment of the present invention provides an ES allocation perception method in a remote multi-center network environment. Multiple log collection processors are configured on the device log processing platform. Each log collection processor corresponds to a shard of at least one ES node. The device to be collected scores the log collection processor according to the load information of the log collection processor and the network delay from itself to the log collection processor. The log collection processor with the best network and the smallest node load is preferably allocated, which effectively reduces the time required for the device to transmit logs and reduces the disk io occupancy time, thereby effectively improving the utilization rate of the node.

[0066] Considering that the hardware equipment of the log collection processor, such as CPU, memory, hard disk and bandwidth, has an impact on the load of log collection processing, the ES allocation perception method in the remote multi-center network environment provided in the first embodiment includes four indicator items of data when the device to be collected obtains the load information of the log collection processor, namely, CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy. That is, the device to be collected first obtains the CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy of each log collection processor, and then calculates the load score of the log collection processor according to the CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy of the log collection processor.

[0067] Specifically, the ES distribution perception method in the remote multi-center network environment provided in the first embodiment of the present invention is to score the collection device according to the load information to obtain the load score of each log collection processor, which specifically includes the following steps:

[0068] Step 1: Score each indicator item of the load information using the following formula;

[0069]

[0070] Wherein, x is the obtained indicator item data (including CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy), and α is the corresponding indicator item threshold;

[0071] Step 2: Multiply the scores of each indicator item to obtain the load score of the corresponding log collection processor.

[0072] For example, if the CPU occupancy rate of a log collection processor is 60% and the CPU index item threshold is 70%, the CPU index item score of the log collection processor is 1; if the CPU occupancy rate of the log collection processor is 80%, the CPU index item score of the log collection processor is 0.2. The scores of other index items are similar to those of the CPU index item. It should be noted that the index item threshold corresponding to each index item can be different and can be set according to the specific application environment and specific needs.

[0073] If the final scores of the CPU, memory, hard disk, and bandwidth indicators of a log collection processor are 1, 0.3, 0.2, and 1 respectively, the load score of the log collection processor is 1*0.3*0.2*1=0.6.

[0074] After the collection device obtains the load score of the log collection processor, it also needs to score the network, that is, score the network delay time from the collection device itself to the log collection processor. Then, the weight value of the log collection processor is obtained based on the load score and the network delay score. The preferred log collection processor is considered from the two perspectives of load and network delay.

[0075] In the ES allocation perception method in a remote multi-center network environment provided in the first embodiment, when the device to be collected is scored according to the network delay time, and the network delay score of each log collection processor is obtained, the network delay score is calculated by the following formula:

[0076]

[0077] Wherein, y is the obtained network delay time, and β is the preset threshold.

[0078] For example, β is 20ms. For a log collection processor, if the network delay time from the device to be collected to the log collection processor is 10ms, the network delay score of the log collection processor is 1; if the network delay time from the device to be collected to the log collection processor is 30ms, the network delay score of the log collection processor is 0.7; if the network delay time from the device to be collected to the log collection processor is 110ms, the network delay score of the log collection processor is 0.

[0079] If the load score of a log collection processor is 0.6 and the network delay score is 0.7, the weight value of the log collection processor is 0.6*0.7=0.42. If the load score of another log collection processor is 1 and the network delay score is 1, the weight value of the log collection processor is 1*1=1.

[0080] It should be noted that the device to be collected can obtain the corresponding network delay time by sending a command to obtain the network delay to the log collection processor.

[0081] Through the above calculation process, it can be seen that the weight values ​​of all log collection processors for a certain device to be collected may be the same, for example, all 1 or all 0, which means that for the device to be collected, the processing capabilities of each log collection processor are equivalent, so a log collection processor can be randomly selected. Specifically, when allocating log collection processors to the device to be collected, the following steps are also included:

[0082] Determine whether all the obtained weight values ​​are the same;

[0083] If they are the same, a log collection processor is randomly selected to allocate the device to be collected;

[0084] Otherwise, all weight values ​​are sorted, and the log collection processor with the largest weight value is selected, and the device to be collected is assigned to the log collection processor.

[0085] It should be noted that each log collection processor corresponds to at least one shard of an ES node, an ES cluster has multiple ES nodes, and each ES node has multiple shards. In this embodiment, the shard of the ES node corresponding to each log collection processor is the shard of the same ES node. For example, a log collection processor corresponds to shard 1 and shard 2 of ES node 1, and the log information of the device to be collected assigned to the log collection processor will be assigned to shard 1 and shard 2 of ES node 1.

[0086] To further explain the present invention, a specific embodiment is provided below. In this specific embodiment, the following steps are included when allocating a log collection processor to a device to be collected.

[0087] Step 1: The collection device obtains four indicator items of the load information of each log collection processor, namely CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy.

[0088] Step 2: Score each indicator item of the load information using the following formula;

[0089]

[0090] Among them, x is the obtained indicator item data, and α is the corresponding indicator item threshold.

[0091] Step 3: Multiply the scores of each indicator item to obtain the load score of the corresponding log collection processor.

[0092] Step 4: The collection device obtains the network delay time from itself to each log collection processor.

[0093] Step 5: Calculate the network delay score using the following formula:

[0094]

[0095] Wherein, y is the obtained network delay time, and β is the preset threshold.

[0096] Step 6: Multiply the load score and the network delay score to obtain the weight value of the corresponding log collection processor.

[0097] Step seven, determine whether all the obtained weight values ​​are the same.

[0098] Step 8: If they are the same, randomly select a log collection processor to allocate the device to be collected; otherwise, sort all weight values, select the log collection processor with the largest weight value, and allocate the device to be collected to the log collection processor.

[0099] Embodiment 2

[0100] The second embodiment of the present invention provides an ES distribution perception device in a remote multi-center network environment, which is used to implement the aforementioned ES distribution perception method in the remote multi-center network environment.

[0101] The second embodiment provides an ES distribution perception device in a remote multi-center network environment, including a device to be collected and multiple log collection processors, such as Figure 2 The schematic diagram of the device log processing platform framework of the present invention is shown.

[0102] Each log collection processor corresponds to at least one shard of an ES node. In specific implementation, the shard of the ES node corresponding to each log collection processor is a shard in the same ES node.

[0103] The device to be collected is configured with a distribution awareness program, and the distribution awareness program is executed to achieve the optimization of the log collection processor. Figure 4 Shown is a schematic block diagram of the allocation perception program structure, which includes the following functional modules.

[0104] Load scoring module: obtains the load information of each log collection processor, and scores it according to the load information to obtain the load score of each log collection processor;

[0105] Network delay scoring module: obtains the network delay time from itself to each log collection processor, and scores it according to the network delay time to obtain the network delay score of each log collection processor;

[0106] Weight value calculation module: multiply the load score and the network delay score to obtain the weight value of the corresponding log collection processor;

[0107] Log collection processor selection module: selects the log collection processor with the largest weight value and assigns the device to be collected to the log collection processor.

[0108] The load information of the log collection processor obtained by the load scoring module includes four indicator items, namely CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy.

[0109] The load scoring module scores the load information and obtains the load score of each log collection processor, including:

[0110] Step 1: Score each indicator item of the load information using the following formula;

[0111]

[0112] Among them, x is the obtained indicator item data, and α is the corresponding indicator item threshold;

[0113] Step 2: Multiply the scores of each indicator item to obtain the load score of the corresponding log collection processor.

[0114] The network delay scoring module scores the network delay time to obtain the network delay score of each log collection processor. Specifically, the network delay score is calculated using the following formula:

[0115]

[0116] Wherein, y is the obtained network delay time, and β is the preset threshold.

[0117] It can be seen from the above calculation process that the weight values ​​of a certain device to be collected may be the same for all log collection processors. Therefore, the allocation perception program of the device to be collected configuration also includes the following functional modules. When the weight values ​​are the same, a log collection processor is randomly selected to allocate the device to be collected:

[0118] Weight value judgment module: judge whether all the obtained weight values ​​are the same.

[0119] Correspondingly, the log collection processor selection module is also used to randomly select a log collection processor to allocate the device to be collected when all the weight values ​​obtained are the same; otherwise, all weight values ​​are sorted, the log collection processor with the largest weight value is selected, and the device to be collected is allocated to the log collection processor.

[0120] The ES distribution perception device in a remote multi-center network environment of this embodiment is used to implement the aforementioned ES distribution perception method in a remote multi-center network environment. Therefore, the specific implementation method of the device can be seen in the implementation example part of the ES distribution perception method in a remote multi-center network environment in the previous text. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part, which will not be elaborated here.

[0121] In addition, since the ES distribution perception device in the remote multi-center network environment of this embodiment is used to implement the aforementioned ES distribution perception method in the remote multi-center network environment, its function corresponds to that of the above-mentioned method and will not be repeated here.

[0122] Embodiment 3

[0123] Figure 5 The present invention provides a schematic diagram of the structure of a terminal device 500 provided in an embodiment of the present invention. The terminal device 500 can be used to perform the method for allocating a log collection processor to a device to be collected provided in an embodiment of the present invention, specifically performing the following steps:

[0124] The device to be collected obtains the load information of each log collection processor, and scores it according to the load information to obtain the load score of each log collection processor;

[0125] The device to be collected obtains the network delay time from itself to each log collection processor, and scores it according to the network delay time to obtain the network delay score of each log collection processor;

[0126] Multiply the load score and the network delay score to obtain the weight value of the corresponding log collection processor;

[0127] The log collection processor with the largest weight value is selected, and the devices to be collected are allocated to the log collection processor.

[0128] The terminal device 500 may include: a processor 510, a memory 520 and a communication unit 530. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0129] The memory 520 can be used to store the execution instructions of the processor 510. The memory 520 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 520 are executed by the processor 510, the terminal 500 can execute some or all of the steps in the following method embodiments.

[0130] The processor 510 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 520, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 510 can include only a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0131] The communication unit 530 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.

[0132] Embodiment 4

[0133] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the device to be collected is assigned a log collection processor, specifically comprising the following steps:

[0134] The device to be collected obtains the load information of each log collection processor, and scores it according to the load information to obtain the load score of each log collection processor;

[0135] The device to be collected obtains the network delay time from itself to each log collection processor, and scores it according to the network delay time to obtain the network delay score of each log collection processor;

[0136] Multiply the load score and the network delay score to obtain the weight value of the corresponding log collection processor;

[0137] The log collection processor with the largest weight value is selected, and the devices to be collected are allocated to the log collection processor.

[0138] The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0139] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention.

[0140] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0141] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0144] The above disclosure is only a preferred embodiment of the present invention, but the present invention is not limited thereto. Any non-creative changes that can be thought of by a person skilled in the art, as well as several improvements and modifications made without departing from the principle of the present invention, should fall within the protection scope of the present invention.

Claims

1. A method for ES distribution perception in a remote multi-center network environment, characterized in that: Configure multiple log collection processors, each of which corresponds to at least one shard of an ES node; When allocating a log collection processor to a device to be collected, perform the following steps: The device to be collected obtains the load information of each log collection processor, and scores it according to the load information to obtain the load score of each log collection processor; The device to be collected obtains the network delay time from itself to each log collection processor, and scores it according to the network delay time to obtain the network delay score of each log collection processor; Multiply the load score and the network delay score to obtain the weight value of the corresponding log collection processor; Select the log collection processor with the largest weight value, and assign the device to be collected to the log collection processor; The load information of the log collection processor obtained by the device to be collected includes four indicator items, namely CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy; Scoring is performed based on the load information to obtain the load score of each log collection processor, including: Step 1: Score each indicator item of the load information using the following formula; Among them, x is the obtained indicator item data, and α is the corresponding indicator item threshold; Step 2: multiply the scores of each indicator item to obtain the load score of the corresponding log collection processor; Scoring is performed based on the network delay time to obtain the network delay score of each log collection processor. Specifically, the network delay score is calculated using the following formula: Wherein, y is the obtained network delay time, and β is the preset threshold.

2. The ES allocation perception method in a remote multi-center network environment according to any one of claim 1, characterized in that: When allocating a log collection processor to a device to be collected, the following steps are also included: Determine whether all the obtained weight values ​​are the same; If they are the same, a log collection processor is randomly selected to allocate the device to be collected; Otherwise, all weight values ​​are sorted, and the log collection processor with the largest weight value is selected, and the device to be collected is assigned to the log collection processor.

3. The ES allocation perception method in a remote multi-center network environment according to claim 2 is characterized in that: The shards of the ES node corresponding to each log collection processor are shards of the same ES node.

4. An ES distribution sensing device in a remote multi-center network environment, characterized in that: It includes devices to be collected and multiple log collection processors; Each log collection processor corresponds to at least one shard of an ES node; The device to be collected is configured with a distribution perception program, which includes: Load scoring module: obtains the load information of each log collection processor, and scores it according to the load information to obtain the load score of each log collection processor; Network delay scoring module: obtains the network delay time from itself to each log collection processor, and scores it according to the network delay time to obtain the network delay score of each log collection processor; Weight value calculation module: multiply the load score and the network delay score to obtain the weight value of the corresponding log collection processor; Log collection processor selection module: selects the log collection processor with the largest weight value and assigns the device to be collected to the log collection processor; The load information of the log collection processor obtained by the load scoring module includes four indicator items: CPU occupancy, memory occupancy, hard disk occupancy and bandwidth occupancy; The load scoring module scores the load information and obtains the load score of each log collection processor, including: Step 1: Score each indicator item of the load information using the following formula; Among them, x is the obtained indicator item data, and α is the corresponding indicator item threshold; Step 2: multiply the scores of each indicator item to obtain the load score of the corresponding log collection processor; The network delay scoring module scores the network delay time to obtain the network delay score of each log collection processor. Specifically, the network delay score is calculated using the following formula: Wherein, y is the obtained network delay time, and β is the preset threshold.

5. The ES distribution sensing device in a remote multi-center network environment according to claim 4 is characterized in that: The allocation perception program of the device configuration to be collected also includes, Weight value judgment module: judge whether all the obtained weight values ​​are the same; The log collection processor selection module is also used to randomly select a log collection processor to allocate the device to be collected when all the weight values ​​obtained are the same; otherwise, all weight values ​​are sorted, the log collection processor with the largest weight value is selected, and the device to be collected is allocated to the log collection processor.

Images