Configuration file processing method and device for acquisition agent

Abstract

The invention provides a configuration file processing method and device for an acquisition agent, relates to the technical field of data acquisition, is applied to a cloud native scene, and comprises the following steps: constructing a configuration center for managing a server configuration file corresponding to the cloud native scene through an open source configuration center component Nacos; when a proxy server corresponding to the acquisition proxy is started, establishing a monitoring link with the configuration center, and pulling an initial configuration file required by the client from the configuration center; establishing a link between the proxy server and a log acquisition server to acquire initialization configuration data corresponding to the initialization configuration file; and when the server configuration file is updated, the configuration center returns an updated target configuration file required by the client to the proxy server through the monitoring link so as to collect target configuration data corresponding to the target configuration file. Therefore, a new configuration center is developed based on Nacos collection, the configuration file of the collection agent in the industry is managed, the function that the collection agent should have in the cloud native scene is achieved, and the data collection cost is saved.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to a method and device for processing configuration files of a collection agent. Background Art

[0002] In cloud-native scenarios, updating the configuration file of the collection agent is a complex operation. In related technologies, configuration file plug-ins are required, or resource objects such as ConfigMap are used, which are difficult to manage and operate. The log information corresponding to the collection agent is not clear enough and cannot meet the log collection scenarios of industry applications. Therefore, a more accurate and reliable method for processing the configuration file of the collection agent is needed. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose a configuration file processing method for a collection agent, develop a new configuration center based on Nacos collection, manage the configuration files of collection agents in the industry, achieve the functions that the collection agent should have in the cloud native scenario, and save data collection costs.

[0005] The second objective of the present invention is to provide a configuration file processing device for a collection agent.

[0006] A third objective of the present invention is to provide an electronic device.

[0007] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0008] A fifth object of the present invention is to provide a computer program product.

[0009] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a configuration file processing method of a collection agent, which is applied to a cloud native scenario. The method includes:

[0010] Create an initial configuration through the web interface or open application programming interface of Nacos, an open source configuration center component deployed in cluster mode, and serve as a configuration center for managing server-side configuration files corresponding to cloud native scenarios;

[0011] When the proxy server corresponding to the acquisition agent is started by the preset listener, a listening link is established between the proxy server and the configuration center according to the address, user name, password and namespace parameters of the configuration center, so as to pull out the initialization configuration file required by the client from the configuration center, wherein the namespace parameters are the business logic definition of the configuration center;

[0012] Establish a link between the proxy server and the log collection server, and read the initialization configuration file based on the unique identifier attached to each initialization configuration file, so as to collect the corresponding initialization configuration data through the log collection server;

[0013] When the server configuration file is updated, the configuration center returns the updated target configuration file required by the client to the proxy server through the monitoring link, reads the target configuration file based on the unique identifier of the target configuration file, and collects the corresponding target configuration data.

[0014] To achieve the above-mentioned purpose, a second embodiment of the present invention proposes a configuration file processing device of a collection agent, which is applied to a cloud native scenario, and the device includes:

[0015] Building modules, used to create an initial configuration through the web interface or open application programming interface of Nacos, an open source configuration center component deployed in cluster mode, and serve as a configuration center for managing server-side configuration files corresponding to cloud native scenarios;

[0016] The pulling module is used to monitor the startup of the proxy server corresponding to the collection agent through a preset listener, and build a listening link between the proxy server and the configuration center according to the address, user name, password and namespace parameters of the configuration center to pull the initialization configuration file required by the client from the configuration center, wherein the namespace parameters are the business logic definition of the configuration center;

[0017] The first collection module is used to establish a link between the proxy server and the log collection server, and read the initialization configuration file based on the unique identifier attached to each initialization configuration file to collect the corresponding initialization configuration data through the log collection server;

[0018] The second collection module is used to return the updated target configuration file required by the client to the proxy server through the monitoring link when the server configuration file is updated, read the target configuration file based on the unique identifier of the target configuration file, and collect the corresponding target configuration data.

[0019] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0020] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0021] In order to achieve the above-mentioned purpose, a fifth aspect of the present invention provides a computer program product, and when the computer program is executed by a processor, it implements the method described in the first aspect.

[0022] The configuration file processing method, device, electronic device, storage medium and computer program product of the acquisition agent provided by the embodiment of the present invention are applied to cloud native scenarios, and a configuration center for managing the server-side configuration files corresponding to cloud native scenarios is constructed through the open source configuration center component Nacos; when the acquisition agent corresponding to the proxy server is started, a listening link is established with the configuration center, and the initialization configuration file required by the client is pulled out from the configuration center; a link is established between the proxy server and the log acquisition server to collect the initialization configuration data corresponding to the initialization configuration file; when the server-side configuration file is updated, the configuration center returns the updated target configuration file required by the client to the proxy server through the listening link to collect the target configuration data corresponding to the target configuration file. Therefore, a new configuration center is developed based on Nacos acquisition to manage the configuration files of acquisition agents in the industry, so as to achieve the functions that the acquisition agent should have in the cloud native scenario and save data acquisition costs.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a configuration file processing method of a collection agent provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of an update flow of another method for processing a configuration file of a collection agent provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the operation flow of a collection agent provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of the structure of a configuration file processing device for a collection agent provided by an embodiment of the present invention;

[0029] Figure 5The figure is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0031] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the present invention are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0032] The acquisition, transmission, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of relevant laws and regulations.

[0033] It should be noted that in the embodiments of the present disclosure, certain software, components, models, etc. existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present invention, but it does not mean that the inventor has or will necessarily use the solution.

[0034] The following describes a configuration file processing method and device for a collection agent according to an embodiment of the present invention with reference to the accompanying drawings.

[0035] Figure 1 The present invention provides a flow chart of a method for processing a configuration file of a collection agent.

[0036] like Figure 1 As shown, applied to cloud native scenarios, the method includes the following steps:

[0037] Step 101, create an initial configuration through the network interface or open application programming interface of Nacos, an open source configuration center component deployed in cluster mode, and serve as a configuration center for building and managing server-side configuration files corresponding to cloud native scenarios.

[0038] In some embodiments, in cloud-native scenarios, when the acquisition agent does not have an integrated configuration center, updating the configuration file of the acquisition agent is a complex operation, which requires external configuration files or the use of resource objects such as ConfigMap, which is difficult to manage and operate. Therefore, the configuration center for managing server-side configuration files corresponding to cloud-native scenarios is built through the open source configuration center component Nacos to solve the usage problems in cloud-native scenarios. Among them, the configuration center contains the address, user name, password and namespace parameters, and the namespace parameters are the business logic definitions of the configuration center. The parameters of the configuration file can be easily adjusted to achieve optimization and adaptation in different application scenarios.

[0039] ConfigMap is a resource object in the Kubernetes application software that is used to store non-sensitive configuration data. It allows configuration information to be decoupled from the container image, so that the configuration file can be managed and updated independently of the application, making it easier to modify and deploy the application configuration file.

[0040] Furthermore, namespaces are defined for business logic, similar to database namespaces, that is, they are used to classify and distinguish different code functions, avoiding conflicts caused by the same variable names between different codes when different code snippets (usually used by different people to collaborate or call existing code snippets) are used at the same time.

[0041] Optionally, Nacos is a dynamic service discovery, configuration management and service management platform designed to help build cloud-native application scenarios. Configuration management is used to provide configuration file creation, deletion, import and export functions, and supports hot update and version management of configuration files.

[0042] Optionally, the cloud-native scenario may be an industry-native scenario corresponding to the bank, but is not limited thereto.

[0043] Optional low, Nacos, an open source configuration center component deployed in cluster mode. If some nodes (configuration center) in the cluster go down, Nacos will automatically transfer the service to other healthy nodes to ensure high availability of the service.

[0044] Step 102, when the preset listener monitors the startup of the proxy server corresponding to the collection agent, a listening link is established between the proxy server and the configuration center according to the address, user name, password and namespace parameters of the configuration center to pull out the initialization configuration file required by the client from the configuration center, wherein the namespace parameters are the business logic definition of the configuration center.

[0045] In some embodiments, when the acquisition agent corresponding to the proxy server is started by the preset listener, a listening link between the proxy server and the configuration center is constructed according to the address, user name, password and namespace parameters of the configuration center, so as to pull out the initialization configuration file required by the client from the configuration center. One implementation method can be that when the acquisition agent corresponding to the proxy server is started by the preset listener, a TCP long link between the network interface or open application programming interface of Nacos and the client interface is constructed according to the address, user name, password and namespace parameters of the configuration center, and used as a listening link between the proxy server and the configuration center; the initialization configuration file required by the client is pulled out from the configuration center using the client interface. Thus, the automatic pulling of the initialization configuration file is realized through the listening link.

[0046] The open application programming interface may be Nacos Open API, but is not limited thereto.

[0047] Optionally, a collection proxy refers to a method of making network requests through a proxy server, which is mainly used for data collection and accessing blocked or restricted websites. The proxy server acts as a middleman, sending requests to the target website (server) on behalf of the user and returning the results, thereby hiding the user's real Internet Protocol Address (IP), avoiding IP blocking, and improving the success rate of data collection.

[0048] Optionally, monitoring the connection is the process of monitoring and managing the connection between the database (proxy server and configuration center) through a listener. When the client application needs to communicate with the configuration center, it first needs to establish a connection between the client and the proxy server and the configuration center. This process is called monitoring the connection.

[0049] In some embodiments, because the configuration file updated during the operation of the process pod will not be persisted by default, the configuration file in the pod becomes unreliable after the restart, and some configuration file changes may be lost. It is necessary to pull a configuration file from the server. Therefore, when the corresponding proxy server of the collection agent is started, a listening link is established with the configuration center, and the initialization configuration file required by the client is pulled from the configuration center to ensure the real-time and reliability of the configuration file during the pod operation process.

[0050] Among them, Pod is the basic unit of application software Kubernetes scheduling, and a Pod represents a process running on the cluster. Kubernetes manages Pods instead of containers, and a Pod may have one or more containers. Each Pod has a special root container (Pause). The image corresponding to the Pause container is part of the Kubernetes platform. In addition to the Pause container, each Pod also contains one or more closely related user business containers. In the case where the cloud native scenario is the industry native scenario corresponding to the bank, the user business container can be a financial business container, but it is not limited to this.

[0051] Kubernetes is an abbreviation formed by replacing the eight characters in the middle of the name "ubernete". It is an open source containerized application management system for multiple hosts in a cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance.

[0052] Step 103: Establish a link between the proxy server and the log collection server to collect initialization configuration data corresponding to the initialization configuration file through the log collection server.

[0053] Optionally, the proxy server may establish a link with the log collection server via a network, but is not limited thereto.

[0054] In some embodiments, the initialization configuration file is accompanied by a unique identifier (data-Id), which is a combination of the client's system + deployment unit + host name, so that the initialization configuration file required by the client can be obtained from the configuration center based on the unique identifier. Therefore, the unique identifier (data-Id) is used as the main logical key corresponding to the configuration file, which is globally unique to ensure its uniqueness. This key is used to update the program. If the server updates the configuration file information later, it can be directly pushed to the collection agent (proxy server), and the collection agent is updated accordingly based on the received updated configuration file information.

[0055] Furthermore, after establishing a link between the proxy server and the log collection server, the initialization configuration file required by the client is obtained from the configuration center according to the unique identifier of the initialization configuration file, and based on the information of the initialization configuration file, the initialization configuration data is quickly collected.

[0056] Step 104, when the server configuration file is updated, the configuration center returns the updated target configuration file required by the client to the proxy server through the monitoring link, reads the target configuration file based on the unique identifier of the target configuration file, and collects the corresponding target configuration data.

[0057] Optionally, the target configuration file is accompanied by a unique identifier, which is a combination of the client's system + deployment unit + host name, so that the target configuration file required by the client can be obtained from the configuration center according to the unique identifier. Thus, based on the information of the target configuration file, the target configuration data can be quickly collected.

[0058] Specifically, the configuration center adopts a two-site three-center deployment mode. According to the unique identifier, the target configuration center is selected from the configuration centers of the two sites and three centers, and the initialization configuration file or target configuration file required by the client is obtained from the target configuration center, so as to achieve accurate matching of different configuration files and different clients.

[0059] In some embodiments, the server configuration file update includes two update methods, wherein: the first update method of the server configuration file is to actively request the server to pull the updated target configuration file based on the logic of the http request short link; the second update method of the server configuration file is a monitoring method, which pushes the target configuration file updated by the server through the TCP long link to achieve continuous collection of target configuration data.

[0060] In addition, after the initialization configuration data corresponding to the initialization configuration file or the target configuration data corresponding to the target configuration file is collected, the initialization configuration data or the target configuration data is sent to the log collection server for storage, thereby efficiently collecting, aggregating and managing massive configuration data.

[0061] Furthermore, in the cloud-native scenario, the log collection server is the log collection system Flume, and the basic working unit Agent of Flume is deployed in daemonset mode. Since the daemonset mode cannot use short-link two-way communication, the update method of the server configuration file is the listening method, which can achieve long-term continuous collection of the log collection server. Therefore, the collection agent based on Flume can be configured with multiple centers and multiple server addresses. When the current server is unavailable, it can automatically switch to a random client in the address list, and even modify the configuration file to switch the data center.

[0062] Among them, the Agent of the log collection system Flume is a core component. It is a Java process that runs on the log collection node. Flume is a real-time log collection system developed by Cloudera, mainly used to collect, aggregate and transmit massive log data. Agent is the basic working unit of Flume, responsible for the collection and transmission of log data.

[0063] The configuration file processing method of the acquisition agent of the embodiment of the present invention is applied to the cloud native scenario, and the configuration center for managing the server-side configuration files corresponding to the cloud native scenario is constructed through the open source configuration center component Nacos; when the acquisition agent corresponding to the proxy server is started, a listening link is established with the configuration center, and the initialization configuration file required by the client is pulled out from the configuration center; a link is established between the proxy server and the log acquisition server to collect the initialization configuration data corresponding to the initialization configuration file; when the server-side configuration file is updated, the configuration center returns the updated target configuration file required by the client to the proxy server through the listening link to collect the target configuration data corresponding to the target configuration file. Therefore, a new configuration center is developed based on Nacos acquisition to manage the configuration files of the acquisition agents in the industry, so as to achieve the functions that the acquisition agent should have in the cloud native scenario and save data acquisition costs.

[0064] In order to clearly explain the above embodiment, this embodiment also provides a schematic diagram of an update flow of a method for processing a configuration file of a collection agent, such as Figure 2 shown.

[0065] like Figure 2 As shown, the update process of the configuration file processing method of the acquisition agent may include the following steps:

[0066] Step 201, create an initial configuration through the network interface or open application programming interface of Nacos, an open source configuration center component deployed in cluster mode, and serve as a configuration center for managing server-side configuration files corresponding to cloud native scenarios.

[0067] Step 202, when the preset listener monitors the startup of the proxy server corresponding to the collection agent, a listening link is established between the proxy server and the configuration center according to the address, user name, password and namespace parameters of the configuration center to pull out the initialization configuration file required by the client from the configuration center, wherein the namespace parameters are the business logic definition of the configuration center.

[0068] Step 203: establish a link between the proxy server and the log collection server, and read the initialization configuration file based on the unique identifier attached to each initialization configuration file, so as to collect the corresponding initialization configuration data through the log collection server.

[0069] For detailed description of steps 201 to 203 , reference may be made to the relevant description in the above embodiment, and the embodiment of the present invention will not be elaborated here.

[0070] Step 204, the log collection server is the log collection system Flume, and when the basic working unit Agent of Flume is deployed in daemonset mode, the update mode of the server configuration file is the listening mode. The configuration center returns the updated target configuration file required by the client to the proxy server through the listening link to collect the target configuration data corresponding to the target configuration file.

[0071] In some embodiments, a listening mode is implemented through a TCP long link to build a persistent network connection between the server and the client. Through the TCP long link, when the server configuration file is updated, the updated target configuration file is automatically pushed and returned to the proxy server. The collection proxy server performs corresponding updates based on the received content, thereby achieving the functions that the collection agent should have under cloud native.

[0072] Step 205: after the initialization configuration data corresponding to the initialization configuration file or the target configuration data corresponding to the target configuration file is collected, the initialization configuration data or the target configuration data is sent to the log collection server for storage.

[0073] In some embodiments, taking the banking cloud native scenario as an example, by initializing the storage of configuration data or target configuration data through the log collection server, it is possible to determine how many rows of configuration data records have been collected and how many configuration data records have been sent, which facilitates subsequent troubleshooting and meets the application log collection scenarios within the industry.

[0074] In addition, the data collector iLogTail plus the server mode can be used to replace the configuration center, but the collection agent function needs to be redeveloped or the corresponding plug-ins need to be added to meet the monitoring requirements within the industry.

[0075] The configuration file processing method of the acquisition agent of the embodiment of the present invention is applied to the cloud native scenario, and the configuration center for managing the server-side configuration files corresponding to the cloud native scenario is constructed through the open source configuration center component Nacos; when the acquisition agent corresponding to the proxy server is started, a listening link is established with the configuration center, and the initialization configuration file required by the client is pulled out from the configuration center; a link is established between the proxy server and the log acquisition server to collect the initialization configuration data corresponding to the initialization configuration file; when the server-side configuration file is updated, the configuration center returns the updated target configuration file required by the client to the proxy server through the listening link to collect the target configuration data corresponding to the target configuration file. Therefore, a new configuration center is developed based on Nacos acquisition to manage the configuration files of the acquisition agents in the industry, so as to achieve the functions that the acquisition agent should have in the cloud native scenario and save data acquisition costs.

[0076] In summary, the present invention also proposes a schematic diagram of the operation flow of a collection agent, such as Figure 3 As shown, taking the cloud native scenario of banks as an example, the collection agent that monitors most of the bank's transactions and application-type collection scenarios is integrated into the configuration center through the open source configuration center component Nacos; when the corresponding proxy server of the collection agent is started, a listening link is established with the configuration center, and the initialization configuration file required by the client is pulled out from the configuration center; then a link is established between the proxy server and the Flume server to collect the initialization configuration data corresponding to the initialization configuration file through the Flume Agent; and when monitoring the update of the server configuration file, the configuration center returns the updated target configuration file required by the client to the proxy server through the listening connection to collect the target configuration data corresponding to the target configuration file, and finally sends the initialization configuration data or target configuration data (collection data) to the log collection server for storage. Therefore, by managing the configuration files of the collection agents in the industry through the configuration center, there is no need to manually remove the external configuration files or modify the ConfigMap management configuration files, so as to achieve the functions that the collection agent should have under cloud native.

[0077] In order to implement the above embodiment, the present invention also provides a configuration file processing device for a collection agent.

[0078] Figure 4 A schematic diagram of the structure of a configuration file processing device for a collection agent provided by an embodiment of the present invention.

[0079] like Figure 4 As shown, the configuration file processing device 40 of the acquisition agent includes: a construction module 41, a pulling module 42, a first acquisition module 43 and a second acquisition module 44.

[0080] Building module 41, for creating an initial configuration through a web interface or an open application programming interface of Nacos, an open source configuration center component deployed in cluster mode, and serving as a configuration center for managing server-side configuration files corresponding to cloud native scenarios;

[0081] The pulling module 42 is used to monitor the startup of the proxy server corresponding to the collection agent through a preset listener, and build a listening link between the proxy server and the configuration center according to the address, user name, password and namespace parameters of the configuration center to pull the initialization configuration file required by the client from the configuration center, wherein the namespace parameters are the business logic definition of the configuration center;

[0082] The first collection module 43 is used to establish a link between the proxy server and the log collection server, and read the initialization configuration file based on the unique identifier attached to each initialization configuration file to collect the corresponding initialization configuration data through the log collection server;

[0083] The second collection module 44 is used for, when the server configuration file is updated, the configuration center returns the updated target configuration file required by the client to the proxy server through the monitoring link, reads the target configuration file based on the unique identifier of the target configuration file, and collects the corresponding target configuration data.

[0084] Furthermore, in a possible implementation of the embodiment of the present invention, the pulling module 42 is specifically configured to:

[0085] When the proxy server corresponding to the acquisition agent is started by the preset listener, a TCP long link between the network interface or open application programming interface of Nacos and the client interface is constructed according to the address, user name, password and namespace parameters of the configuration center, and serves as a listening link between the proxy server and the configuration center;

[0086] Use the client interface to pull out the initialization configuration file required by the client from the configuration center.

[0087] Furthermore, in a possible implementation of the embodiment of the present invention, the server-side configuration file update includes two update methods, wherein:

[0088] The first update method of the server configuration file is to actively request the server to pull the updated target configuration file based on the logic of the http request short link;

[0089] The second update mode of the server configuration file is a monitoring mode, which pushes the target configuration file updated by the server through a TCP long link.

[0090] Furthermore, in a possible implementation of the embodiment of the present invention, the device further includes:

[0091] The sending module is used to send the initialization configuration data or the target configuration data to the log collection server for storage after the initialization configuration data corresponding to the initialization configuration file or the target configuration data corresponding to the target configuration file is collected.

[0092] Furthermore, in a possible implementation of an embodiment of the present invention, in a cloud-native scenario, the log collection server is the log collection system Flume, and the basic working unit Agent of Flume is deployed in daemonset mode, and the update method of the server configuration file is the listening method.

[0093] Furthermore, in a possible implementation of an embodiment of the present invention, the unique identifier is a combination of the client's system + deployment unit + host name, so as to obtain the initialization configuration file or target configuration file required by the client from the configuration center based on the unique identifier.

[0094] Further, in a possible implementation of the embodiment of the present invention, the configuration center adopts a deployment mode of two locations and three centers, and obtaining the initialization configuration file or the target configuration file required by the client from the configuration center according to the unique identifier includes:

[0095] A target configuration center is selected from the configuration centers in two locations and three centers according to the unique identifier, and an initialization configuration file or a target configuration file required by the client is obtained from the target configuration center.

[0096] The configuration file processing device of the acquisition agent of the embodiment of the present invention is applied to the cloud native scenario, and the configuration center for managing the server-side configuration files corresponding to the cloud native scenario is constructed through the open source configuration center component Nacos; when the acquisition agent corresponding to the proxy server is started, a listening link is established with the configuration center, and the initialization configuration file required by the client is pulled out from the configuration center; a link is established between the proxy server and the log acquisition server to collect the initialization configuration data corresponding to the initialization configuration file; when the server-side configuration file is updated, the configuration center returns the updated target configuration file required by the client to the proxy server through the listening link to collect the target configuration data corresponding to the target configuration file. Therefore, a new configuration center is developed based on Nacos acquisition to manage the configuration files of the acquisition agents in the industry, so as to achieve the functions that the acquisition agent should have in the cloud native scenario and save data acquisition costs.

[0097] In order to implement the above embodiment, Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment.

[0098] like Figure 5 As shown, the electronic device 500 includes:

[0099] Processing component 502 , memory 504 , power component 506 , multimedia component 508 , audio component 510 , input / output (I / O) interface 512 , sensor component 514 , and communication component 516 .

[0100] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0101] The memory 504 is configured to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0102] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0103] The multimedia component 508 includes a touch display screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the touch display screen may include a liquid crystal display (LCD) and a touch panel (TP). The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0104] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516.

[0105] In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0106] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0107] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0108] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0109] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned data preheating cache method.

[0110] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the above instructions can be executed by a processor 520 of the electronic device 500 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0111] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the above method.

[0112] In order to implement the above embodiments, the present invention further provides a computer program product, including a computer program, wherein the computer program implements the above method when executed by a processor.

[0113] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0114] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0115] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0116] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0117] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0118] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0119] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0120] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for processing a configuration file of a collection agent, characterized in that: Applied to cloud native scenarios, the method includes: Create an initial configuration through the web interface or open application programming interface of Nacos, an open source configuration center component deployed in cluster mode, and serve as a configuration center for managing server-side configuration files corresponding to cloud native scenarios; When the proxy server corresponding to the acquisition agent is started by the preset listener, a listening link is established between the proxy server and the configuration center according to the address, user name, password and namespace parameters of the configuration center, so as to pull out the initialization configuration file required by the client from the configuration center, wherein the namespace parameters are the business logic definition of the configuration center; Establish a link between the proxy server and the log collection server, and read the initialization configuration file based on the unique identifier attached to each initialization configuration file, so as to collect the corresponding initialization configuration data through the log collection server; When the server configuration file is updated, the configuration center returns the updated target configuration file required by the client to the proxy server through the monitoring link, reads the target configuration file based on the unique identifier of the target configuration file, and collects the corresponding target configuration data.

2. The method according to claim 1, characterized in that: When the preset listener monitors the startup of the proxy server corresponding to the acquisition agent, a listening link is established between the proxy server and the configuration center according to the address, user name, password and namespace parameters of the configuration center to pull out the initialization configuration file required by the client from the configuration center, including: When the proxy server corresponding to the acquisition agent is started by the preset listener, a TCP long link between the network interface or open application programming interface of Nacos and the client interface is constructed according to the address, user name, password and namespace parameters of the configuration center, and serves as a listening link between the proxy server and the configuration center; Use the client interface to pull out the initialization configuration file required by the client from the configuration center.

3. The method according to claim 1, characterized in that The server configuration file update includes two update methods, wherein: The first update method of the server configuration file is to actively request the server to pull the updated target configuration file based on the logic of the http request short link; The second update mode of the server configuration file is a monitoring mode, which pushes the target configuration file updated by the server through a TCP long link.

4. The method according to claim 1, characterized in that: The method further comprises: After the initialization configuration data corresponding to the initialization configuration file is collected or the target configuration data corresponding to the target configuration file is collected, the initialization configuration data or the target configuration data is sent to the log collection server for storage.

5. The method according to claims 3 and 4, characterized in that In the cloud-native scenario, the log collection server is the log collection system Flume. When the basic working unit Agent of Flume is deployed in daemonset mode, the update mode of the server configuration file is the listening mode.

6. The method according to claim 1, characterized in that The unique identifier is a combination of the client's system+deployment unit+host name, so that the initialization configuration file or target configuration file required by the client can be obtained from the configuration center according to the unique identifier.

7. The method according to claim 6, characterized in that The configuration center adopts a deployment mode of two locations and three centers, and the initialization configuration file or target configuration file required by the client is obtained from the configuration center according to the unique identifier, including: A target configuration center is selected from the configuration centers in two locations and three centers according to the unique identifier, and an initialization configuration file or a target configuration file required by the client is obtained from the target configuration center.

8. A configuration file processing device for a collection agent, characterized in that: Applied to cloud native scenarios, the device includes: Building modules, used to create an initial configuration through the web interface or open application programming interface of Nacos, an open source configuration center component deployed in cluster mode, and serve as a configuration center for managing server-side configuration files corresponding to cloud native scenarios; The pulling module is used to monitor the startup of the proxy server corresponding to the collection agent through a preset listener, and build a listening link between the proxy server and the configuration center according to the address, user name, password and namespace parameters of the configuration center to pull the initialization configuration file required by the client from the configuration center, wherein the namespace parameters are the business logic definition of the configuration center; The first collection module is used to establish a link between the proxy server and the log collection server, and read the initialization configuration file based on the unique identifier attached to each initialization configuration file to collect the corresponding initialization configuration data through the log collection server; The second collection module is used to return the updated target configuration file required by the client to the proxy server through the monitoring link when the server configuration file is updated, read the target configuration file based on the unique identifier of the target configuration file, and collect the corresponding target configuration data.

9. The device according to claim 8, characterized in that The pulling module is specifically used for: When the proxy server corresponding to the acquisition agent is started by the preset listener, a TCP long link between the network interface or open application programming interface of Nacos and the client interface is constructed according to the address, user name, password and namespace parameters of the configuration center, and serves as a listening link between the proxy server and the configuration center; Use the client interface to pull out the initialization configuration file required by the client from the configuration center.

10. The device according to claim 8, characterized in that The server configuration file update includes two update methods, wherein: The first update method of the server configuration file is to actively request the server to pull the updated target configuration file based on the logic of the http request short link; The second update mode of the server configuration file is a monitoring mode, which pushes the target configuration file updated by the server through a TCP long link.

11. The device according to claim 8, characterized in that The device further comprises: The sending module is used to send the initialization configuration data or the target configuration data to the log collection server for storage after the initialization configuration data corresponding to the initialization configuration file or the target configuration data corresponding to the target configuration file is collected.

12. The device according to claims 10 and 11, characterized in that In the cloud-native scenario, the log collection server is the log collection system Flume. When the basic working unit Agent of Flume is deployed in daemonset mode, the update mode of the server configuration file is the listening mode.

13. The device according to claim 8, characterized in that The unique identifier is a combination of the client's system+deployment unit+host name, so that the initialization configuration file or target configuration file required by the client can be obtained from the configuration center according to the unique identifier.

14. The device according to claim 13, characterized in that The configuration center adopts a deployment mode of two locations and three centers, and the initialization configuration file or target configuration file required by the client is obtained from the configuration center according to the unique identifier, including: A target configuration center is selected from the configuration centers in two locations and three centers according to the unique identifier, and an initialization configuration file or a target configuration file required by the client is obtained from the target configuration center.

15. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images