Collection plug-in management method and device, equipment, storage medium and program product

Abstract

The invention provides a collection plug-in management method and device, equipment, a storage medium and a program product, and relates to the technical field of big data, the method comprises the following steps: receiving an upgrade request sent by a client, and starting an upgrade process; obtaining a version file of each collection plug-in, and determining a to-be-upgraded collection plug-in based on the version file; the acquisition plug-ins are integrated in a plug-in full package; downloading an upgrade package of the to-be-upgraded collection plug-in based on the predefined interface, and backing up the to-be-upgraded collection plug-in of the previous version; the to-be-upgraded collection plug-in is upgraded based on the upgrade package to obtain an upgrade result, and after it is determined that upgrade of the to-be-upgraded collection plug-in fails based on the upgrade result, the to-be-upgraded collection plug-in is returned to the collection plug-in corresponding to the previous version, so that the centralized client and the plug-in have automatic upgrade capability, automatic upgrade is realized based on a uniform interface, and the upgrade efficiency is improved. User participation is not needed, the efficiency is improved, stable operation of the client is ensured, and the problems of scattered distribution of the collection plug-ins and difficulty in upgrading operation and maintenance are solved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a collection plug-in management method, device, equipment, storage medium and program product. Background Art

[0002] In the field of big data, data collection is a key technology and the first link in the life cycle of big data. It is a prerequisite for technologies such as big data storage, big data analysis, and big data processing. However, there are many data sources faced by big data collection, such as directly connectable data sources and non-directly connectable data sources. The corresponding collected data include streaming data and batch data.

[0003] In the prior art, for the collection of streaming data, Flume streaming collection method is usually adopted to collect data from corresponding data sources. For the collection of batch data, batch job data collection is usually completed based on Cos SDK (Cloud Object Storage Software Development Kit).

[0004] However, Flume streaming collection and Cos SDK use different collection plug-ins for different data sources, and the collection plug-ins are decentralized and need to be manually upgraded, making upgrade and operation more difficult. Summary of the invention

[0005] The present application provides a collection plug-in management method, device, equipment, storage medium and program product to solve the problems in the prior art that the collection plug-ins are decentralized, need to be manually upgraded, and the collection plug-in upgrade and maintenance are relatively difficult.

[0006] In a first aspect, the present application provides a collection plug-in management method, the method comprising:

[0007] Receiving an upgrade request sent by a client, and starting an upgrade process based on the upgrade request;

[0008] After starting the upgrade process, obtaining the version file of each acquisition plug-in, and determining the acquisition plug-in to be upgraded from the acquisition plug-ins based on the version file; the acquisition plug-in is integrated into the full plug-in package;

[0009] Downloading the upgrade package corresponding to the acquisition plug-in to be upgraded based on a predefined interface, and backing up the previous version of the acquisition plug-in to be upgraded;

[0010] The acquisition plug-in to be upgraded is upgraded based on the upgrade package to obtain an upgrade result, and after determining that the acquisition plug-in to be upgraded fails to be upgraded based on the upgrade result, the acquisition plug-in to be upgraded is rolled back to the acquisition plug-in corresponding to the previous version.

[0011] In a possible design, the step of upgrading the acquisition plug-in to be upgraded based on the upgrade package to obtain an upgrade result includes:

[0012] The preset files of the acquisition plug-in to be upgraded are upgraded based on the upgrade package, and the upgrade result of each preset file is obtained; the preset files include: executable files, dependent packages and configuration files.

[0013] In one possible design, the method further includes:

[0014] After determining that the acquisition plug-in to be upgraded is upgraded successfully based on the upgrade result, closing the upgrade process, and updating the full plug-in package based on the acquisition plug-in to be upgraded that is upgraded successfully;

[0015] Back up the collection plug-in in the full plug-in package and restart the client.

[0016] In a possible design, each collection plug-in is independent of each other and is used to complete the collection task of the data source; the method further includes:

[0017] Based on a predefined interface, monitoring whether the cache occupancy of the acquisition plug-in is greater than a preset threshold; the acquisition plug-in includes a streaming acquisition plug-in and a batch acquisition plug-in;

[0018] If so, control the acquisition plug-in to stop executing the acquisition task for the data source until the cache occupancy is greater than or equal to a preset threshold, and then execute the acquisition task for the data source;

[0019] If not, the collection transmission efficiency and / or the collection data sending result of the execution link of the collection task are monitored to determine whether the collection task is executed abnormally.

[0020] In one possible design, the method further includes:

[0021] If it is determined based on the acquisition transmission efficiency and / or the acquisition data sending result that the acquisition task is executed abnormally, the data collected by the acquisition plug-in is resent.

[0022] In one possible design, the method further includes:

[0023] The log information generated by the collection plug-in is managed to the client based on a predefined interface, and is visually displayed on the client so that the user can view the log information.

[0024] In a second aspect, the present application provides a collection plug-in management device, the device comprising:

[0025] A receiving module, used to receive an upgrade request sent by a client, and start an upgrade process based on the upgrade request;

[0026] A determination module, used for obtaining a version file of each acquisition plug-in after starting the upgrade process, and determining the acquisition plug-in to be upgraded from the acquisition plug-ins based on the version file; the acquisition plug-in is integrated in the full plug-in package;

[0027] A download module, used to download the upgrade package corresponding to the acquisition plug-in to be upgraded based on a predefined interface, and back up the previous version of the acquisition plug-in to be upgraded;

[0028] The upgrade module is used to upgrade the acquisition plug-in to be upgraded based on the upgrade package to obtain an upgrade result, and after determining that the upgrade of the acquisition plug-in to be upgraded fails based on the upgrade result, roll back the acquisition plug-in to be upgraded to the acquisition plug-in corresponding to the previous version.

[0029] In a third aspect, the present application provides an electronic device comprising: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the collection plug-in management method described in the first aspect and various possible designs of the first aspect.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the collection plug-in management method described in the first aspect and various possible designs of the first aspect is implemented.

[0031] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the collection plug-in management method as described in the first aspect and various possible designs of the first aspect.

[0032] The collection plug-in management method, device, equipment, storage medium and program product provided by the present application integrate various types of collection plug-ins into a full package of plug-ins and provide a unified interface for stream and batch collection to perform unified integrated management of each collection plug-in, simplify the management of collection plug-ins for different data sources, and then when upgrading the collection plug-ins corresponding to different data sources, the version information of each collection plug-in can be obtained first, so as to perform fine-grained upgrades or full upgrades for the collection plug-ins according to the version information. This method can ensure that only necessary plug-ins are upgraded, reducing unnecessary workload, and then download the upgrade package corresponding to the collection plug-in based on the unified interface, and upgrade based on the upgrade package. If it is found that the upgraded version of the collection plug-in is abnormal during production operation, because the previous version of the collection plug-in has been backed up, the collection plug-in can be rolled back to the previous version to ensure the normal operation of the collection plug-in. In this way, the centralized client and plug-in have automatic upgrade capabilities, and can implement automatic upgrades based on the unified interface without user participation, thereby improving efficiency and ensuring stable operation of the client, solving the problems of scattered distribution of collection plug-ins and difficult upgrade and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 An application scenario diagram of a collection plug-in management method provided in an embodiment of the present application;

[0035] Figure 2 A flowchart of a collection plug-in management method provided in an embodiment of the present application;

[0036] Figure 3 A system architecture diagram corresponding to a collection plug-in management method provided in an embodiment of the present application;

[0037] Figure 4 An architectural diagram of each component module in a client provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of the structure of a file in a client provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of the structure of the internal file directory of the full plug-in package provided in an embodiment of the present application;

[0040] Figure 7 A flowchart of a collection plug-in upgrade method provided in an embodiment of the present application;

[0041] Figure 8A schematic diagram of the structure of a collection plug-in management device provided in an embodiment of the present application;

[0042] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0043] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0045] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0046] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0047] There are many data sources for big data collection, such as message middleware, databases, files, object storage, etc. These data sources include directly connectable data sources and non-directly connectable data sources, and accordingly, they include structured data and unstructured data, or stream data and batch data. Therefore, a unified collection method is needed to meet the collection scenarios of these different data sources.

[0048] In a possible implementation, for the collection of streaming data, Flume streaming collection method is usually adopted to collect data from corresponding data sources, and for the collection of batch data, batch job data collection is usually completed based on Cos SDK.

[0049] Among them, Flume streaming collection method includes three major components: Source, Channel, and Sink. The Source component is used to collect data, the Channel component is used to cache data, and the Sink is used to send data to the downstream. During the development process, different Sources need to collect data from the corresponding data sources.

[0050] However, Flume streaming collection and Cos SDK use different collection plug-ins for different data sources, and the collection plug-ins are decentralized and need to be manually upgraded, making upgrade and operation more difficult.

[0051] It should be noted that Flume streaming collection and Cos SDK collection methods do not support automatic upgrade capabilities, and the job configuration files need to be manually configured.

[0052] In response to the above problems, the present application provides a collection plug-in management method, which integrates various types of collection plug-ins into a full package of plug-ins and provides a unified interface for stream and batch collection to uniformly integrate and manage each collection plug-in, simplify the management of collection plug-ins for different data sources, and then when upgrading the collection plug-ins corresponding to different data sources, the version information of each collection plug-in can be obtained first, so as to perform fine-grained upgrades or full upgrades for the collection plug-ins according to the version information. This method can ensure that only necessary plug-ins are upgraded, reducing unnecessary workload, and then download the upgrade package corresponding to the collection plug-in based on the unified interface, and upgrade based on the upgrade package. If it is found that the upgraded version of the collection plug-in is abnormal during production operation, because the previous version of the collection plug-in has been backed up, the collection plug-in can be rolled back to the previous version to ensure the normal operation of the collection plug-in. In this way, the centralized client and plug-in have automatic upgrade capabilities, and can implement automatic upgrades based on the unified interface without user participation, thereby improving efficiency, and ensuring stable operation of the client, solving the problems of scattered distribution of collection plug-ins and difficult upgrade and maintenance.

[0053] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0054] For example, Figure 1 This is an application scenario diagram of a collection plug-in management method provided in an embodiment of the present application. The application scenario can be applied to scenarios of big data collection, such as Figure 1As shown, the application scenario includes: a client 101, a first data source 102 and a second data source 103. The embodiment of the present application does not specifically limit the type of data source, which can be a directly connectable data source or a non-directly connectable data source.

[0055] When the client 101 collects data from the first data source 102 and the second data source 103, the collection plug-ins corresponding to the first data source 102 and the second data source 103 can be called based on the unified interface, and the data in the first data source 102 and the second data source 103 can be obtained based on the respective collection plug-ins. However, when the collection plug-ins corresponding to the first data source 102 and the second data source 103 need to be upgraded, an upgrade request sent by the client 101 can be received, and the upgrade process can be started based on the upgrade request, and then the upgrade package corresponding to the corresponding collection plug-in can be downloaded based on the unified interface, and the previous version of the collection plug-in can be backed up at the same time. In this way, after the collection plug-in is upgraded based on the upgrade package, if it is found that the upgraded version of the collection plug-in has an abnormality in the production operation, the collection plug-in can be rolled back to the previous version to ensure the normal operation of the collection plug-in, and then based on the upgrade request and the unified interface, the automatic upgrade of the collection plug-in can be realized, and the problem of upgrade abnormality can also be automatically dealt with.

[0056] It should be noted that the embodiment of the present application does not specifically limit the number of collection plug-ins corresponding to the data source, and when the collection plug-in is upgraded, the collection plug-in that needs to be upgraded can be determined based on the version file of the collection plug-in, and then a fine-grained upgrade can be performed on the specific plug-in, or a full upgrade can be performed to achieve full automation of the entire upgrade process of the collection plug-in without manual participation, thereby solving the problem of difficult upgrade and operation and maintenance.

[0057] For example, Figure 2 A flow chart of a collection plug-in management method provided in an embodiment of the present application is as follows: Figure 2 As shown, the execution subject of the acquisition plug-in management method can be a client or a system with acquisition plug-in management function, which is not specifically limited in the embodiment of the present application. Figure 3 A system architecture diagram corresponding to a collection plug-in management method provided in an embodiment of the present application, such as Figure 3As shown, the system includes: a presentation layer, an API (Application Programming Interface) layer, a control layer and a plug-in layer; wherein the presentation layer is used to provide a shell user interface, a client UI (User Interface) or receive requests from external applications for users to manage acquisition plug-ins; the API layer is used to provide an HTTP (Hyper Text Transfer Protocol) interface; the control layer includes an executor, an upgrade component and a monitoring component; the plug-in layer includes plug-ins corresponding to data sources such as databases, middleware, and files.

[0058] Among them, since there are many data sources, the collection plug-ins corresponding to each data source are not fixed. In order to facilitate the unified management of the collection plug-ins, a unified collection interface is provided, such as the HTTP interface. The executor can be driven according to the task configuration to complete the data collection task, the monitoring component can monitor the collection plug-in collection task, and the upgrade component can upgrade the collection plug-in, etc., to achieve the full life cycle management of the collection plug-in.

[0059] It should be noted that based on Figure 3 The system architecture shown enables the acquisition plug-in to be upgraded, monitored, and capable of executing jobs, and can be started and stopped remotely, such as upgrading the acquisition plug-in based on an upgrade request sent by an external application.

[0060] For example, Figure 4 The following is an architecture diagram of each component module in a client provided in an embodiment of the present application, such as Figure 4 As shown in the figure, the client includes API, task executor (Task Manager), upgrade component (Upgrader), monitoring component (Monitor) and server-side communication basic component (Server Client); among them, API is a unified interface for external access; the task executor is used to complete task distribution processing, internal service management and other operations; the upgrade component is used to complete the upgrade operation of the client to the collection plug-in; the monitoring component is used to complete the reporting of the client status, monitor the server upgrade and remotely start and stop streaming task instructions and other operations.

[0061] Accordingly, Figure 5 A schematic diagram of the structure of a file in a client provided in an embodiment of the present application, such as Figure 5 As shown, the files in the client include the executable file directory (bin), dependency package directory (lib), log directory (logs), configuration directory (config), runtime data directory (runtime), full plug-in package (dg_full_tool) and text file (README).

[0062] The full plug-in package refers to a collection package that contains all relevant acquisition plug-ins, which is convenient for users to download and install the required plug-ins at one time. Figure 6 A schematic diagram of the structure of the internal file directory of the plug-in full package provided in the embodiment of the present application, such as Figure 6 As shown, the full package of plug-ins can integrate streaming collection plug-ins, collection plug-ins in Linux tools, client's own collection plug-ins and other plug-ins, etc. The embodiment of the present application does not specifically limit the type and quantity of collection plug-ins that can be integrated in the full package of plug-ins. For example, the full package of plug-ins can integrate streaming collection Flume plug-ins, batch collection Cos plug-ins, or other more efficient plug-ins can be integrated according to needs. Among them, the collection plug-ins in the full package of plug-ins can be flexibly adapted, and changing the type and quantity of collection plug-ins in the full package of plug-ins will not affect the overall architecture of the system.

[0063] like Figure 6 As shown, the streaming acquisition plug-in may include a data synchronization directory (canal), a dependency package directory, a process ID directory (pid), a start process directory (start.sh), a stop process directory (stop.sh), an execution file directory, a storage location directory (position), a checkpoint directory (checkpoint), a library file directory (usrlib), a discarded data directory (discard), a plugin file directory (plugins.d), a configuration directory, a log directory, and version information (Fversion), etc.; the Linux tool may include a configuration directory, a log directory, an upload client file (uploadFileClient), and version information (Leversion), etc.; the client itself may include a configuration directory, an execution file directory, a dependency package directory, and text files, etc.

[0064] It should be noted that canal is used for data synchronization or transmission; pid (Process ID) represents the process identifier, which is used to store the ID (Identification) of the running process; start.sh is a Shell script file used to start an application or process. stop.sh is a Shell script file used to stop an application or process; position is used to store certain position or status information, such as the location of data synchronization; checkpoint is used to store checkpoint information so that the system can be restored to a known state when a failure occurs; discard is used to store discarded data or logs; usrlib is a directory for storing user-level library files; plugins.d is used to store plug-in files or extension modules. Fversion is used to indicate the version information of a component or project; README is a text file containing basic project information, installation guide, instructions for use, etc.; uploadFileClient is a client tool for uploading files; Leversion is used to indicate the version information of a component or project, similar to Fversion.

[0065] like Figure 2 As shown, the acquisition plug-in management method is based on the above Figure 3-Figure 6 The client structure shown in the figure is implemented, and the collection plug-in management method includes the following steps:

[0066] S201: Receive an upgrade request sent by a client, and start an upgrade process based on the upgrade request.

[0067] In this step, when the client is upgraded, an upgrade request may be generated, and then the task executor may start the upgrade process and close the management process of the collection plug-in to prevent the operation of other management processes from affecting the upgrade of the collection plug-in.

[0068] It is understandable that after the client is upgraded and the upgrade process is started, the management process of the acquisition plug-in may not be exited, as long as other management processes do not affect the upgrade of the acquisition plug-in. The embodiment of the present application does not make specific limitations on this.

[0069] S202: After starting the upgrade process, obtain the version file of each acquisition plug-in, and determine the acquisition plug-in to be upgraded from the acquisition plug-ins based on the version file; the acquisition plug-in is integrated into the full plug-in package.

[0070] In an embodiment of the present application, each collection plug-in corresponds to a version file. During an upgrade, the process can compare the version file to determine whether to upgrade the collection plug-in. The version file is used to indicate whether the collection plug-in is the latest version or whether it is a version that meets user requirements. The embodiment of the present application does not limit the specific content corresponding to the version file. For example, the version file may include the version number of the plug-in.

[0071] It is understandable that, since different types of acquisition plug-ins are integrated into the full plug-in package, each acquisition plug-in can be called based on a unified interface.

[0072] In this step, when the client is managing each collection plug-in, after obtaining the version file of each collection plug-in, it determines whether to upgrade each collection plug-in by comparing the version file, and then determines the upgrade of some collection plug-ins to be upgraded, or determines that all collection plug-ins need to be upgraded, so that targeted and fine-grained collection plug-in upgrades can be performed, or all collection plug-ins can be upgraded.

[0073] S203: Downloading an upgrade package corresponding to the acquisition plug-in to be upgraded based on a predefined interface, and backing up the previous version of the acquisition plug-in to be upgraded.

[0074] In the embodiment of the present application, the predefined interface is a unified interface defined in advance for calling and managing the collection plug-in. Based on the predefined interface, the collection plug-in can be called, monitored, and upgraded. The embodiment of the present application does not limit the specific type corresponding to the predefined interface, such as it can be an HTTP interface.

[0075] S204: upgrading the acquisition plug-in to be upgraded based on the upgrade package to obtain an upgrade result, and after determining based on the upgrade result that the acquisition plug-in to be upgraded has failed to be upgraded, rolling back the acquisition plug-in to be upgraded to a collection plug-in corresponding to a previous version.

[0076] For example, Figure 7 A flowchart of a collection plug-in upgrade method provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the plug-in upgrade method includes the following steps:

[0077] Step A: After receiving the upgrade request sent by the client, the management process starts the upgrade process based on the upgrade request and controls its own management process to exit.

[0078] Step B: The upgrade process is based on the upgrade package download interface, that is, the predefined interface to download the upgrade package corresponding to the acquisition plug-in to be upgraded, and back up the previous version of the acquisition plug-in to be upgraded, and then upgrade the acquisition plug-in to be upgraded based on the upgrade package, determine whether the upgrade package is installed successfully, and obtain the upgrade result.

[0079] Step C: If the upgrade result is successful, the upgrade result can be fed back to the server; if the upgrade result is failed, the acquisition plug-in to be upgraded will be rolled back to the acquisition plug-in corresponding to the previous version; after the upgrade is completed, the management process is started and the upgrade process is closed so that the acquisition plug-in can perform other tasks.

[0080] Therefore, an acquisition plug-in management method provided in an embodiment of the present application receives an upgrade request sent by a client and starts the upgrade process, so that the entire upgrade process can be automated, reducing manual intervention and improving efficiency; by obtaining the version file of each acquisition plug-in, it is possible to accurately determine which plug-ins need to be upgraded, avoiding unnecessary repeated upgrades, saving time and resources; downloading the upgrade package based on a predefined interface ensures that the required upgrade packages can be downloaded in a unified manner, simplifying the upgrade operation; by verifying the upgrade results, problems arising during the upgrade process can be discovered and handled in a timely manner, and after an upgrade fails, it can automatically roll back to the previous version, ensuring the stability and continuity of the system, and avoiding system unavailability caused by upgrade failure; wherein, the acquisition plug-in is integrated in the full plug-in package, which is convenient for unified management and distribution, simplifies the maintenance and update process of the plug-in, and can adapt to different acquisition plug-ins and different upgrade requirements, with high flexibility and scalability.

[0081] In a possible design, the step of upgrading the acquisition plug-in to be upgraded based on the upgrade package to obtain an upgrade result includes:

[0082] The preset files of the acquisition plug-in to be upgraded are upgraded based on the upgrade package, and the upgrade result of each preset file is obtained; the preset files include: executable files, dependent packages and configuration files.

[0083] It should be noted that during the upgrade of the collection plug-in, the data files generated during runtime will not be affected. Only preset files such as executable files, dependent packages, and configuration files will be updated.

[0084] The preset files are defined as necessary files that only need to be updated. Since only part of the files are updated, the upgrade process is simplified and the time required for the upgrade process is reduced.

[0085] Since the data files generated during runtime will not be affected by the upgrade process, only the executable files, dependent packages and configuration files are updated. Therefore, the integrity and security of the data can be ensured, data loss or damage can be avoided, and the risks that may be introduced during the upgrade process can be reduced, ensuring the stability of the system and business continuity, so that users can continue to access and use existing data without interrupting business operations due to the upgrade. If the upgrade fails, since the data files have not been modified, it is possible to roll back to the previous version more quickly, reducing system downtime.

[0086] In one possible design, the method further includes:

[0087] After determining that the acquisition plug-in to be upgraded is upgraded successfully based on the upgrade result, closing the upgrade process, and updating the full plug-in package based on the acquisition plug-in to be upgraded that is upgraded successfully;

[0088] Back up the collection plug-in in the full plug-in package and restart the client.

[0089] Among them, closing the upgrade process after a successful upgrade can release system resources, avoid resource waste, and improve system performance. Restarting the client after a successful upgrade can ensure that the new version of the plug-in and configuration takes effect and avoid the impact of residual problems of the old version on system stability.

[0090] In this step, if it is determined that the acquisition plug-in upgrade is successful, the client can back up the full plug-in package of the previous version and restart the client.

[0091] In this way, by updating the full package of the plug-in, the integrity and consistency of the system can be ensured, which is helpful for subsequent maintenance and management. The backup mechanism provides the possibility of rollback. When problems occur, you can quickly roll back to the previous stable version to ensure the security of the system, so that you can quickly recover when a failure occurs, reducing system downtime and the risk of business interruption.

[0092] It should be noted that since Flume streaming collection does not support job monitoring capabilities and collection volume statistics capabilities, and the Channel component in Flume does not support the back pressure mechanism when under pressure, the collection rate of the Source component is reduced.

[0093] In a possible design, each collection plug-in is independent of each other and is used to complete the collection task of the data source; the method further includes:

[0094] Based on a predefined interface, monitoring whether the cache occupancy of the acquisition plug-in is greater than a preset threshold; the acquisition plug-in includes a streaming acquisition plug-in and a batch acquisition plug-in;

[0095] If so, control the acquisition plug-in to stop executing the acquisition task for the data source until the cache occupancy is greater than or equal to a preset threshold, and then execute the acquisition task for the data source;

[0096] If not, the collection transmission efficiency and / or the collection data sending result of the execution link of the collection task are monitored to determine whether the collection task is executed abnormally.

[0097] In the embodiment of the present application, the preset threshold is the maximum threshold that the collection plug-in can withstand. If the cache occupancy of the collection plug-in is greater than the preset threshold, it means that the collection plug-in cannot withstand the pressure. If the cache occupancy of the collection plug-in is less than or equal to the preset threshold, it means that the collection plug-in can withstand the pressure and can continue to execute the task. The embodiment of the present application does not specifically limit the size of the preset threshold, which can be determined based on the type of collection plug-in.

[0098] Exemplarily, taking the collection plug-in as a streaming collection plug-in, it is possible to monitor whether the Channel component in the streaming collection plug-in is under pressure based on a predefined interface. If the Channel component is under pressure, the collection task of the Source component is suspended, waiting for the pressure of the Channel component to be released and the Source component to be automatically restored before executing the collection task.

[0099] Among them, the collection tasks can be uniformly issued based on the predefined interface.

[0100] Optionally, the time consumption statistics of each link during the execution of the collection task by the streaming collection plug-in can be monitored based on a predefined interface to grasp the collection transmission efficiency of each link, and then judge whether the collection task is executed abnormally based on the collection transmission efficiency and / or the collection data sending results.

[0101] Optionally, if the collection transmission efficiency is less than the transmission threshold, and / or the collection data fails to be sent, it is determined that the collection task is executed abnormally.

[0102] Since the monitoring component provided in this application can support job monitoring capabilities and collection volume statistics, the client can maintain the job metadata information, job status, and collection volume statistics of the current day's flow jobs, and perform persistent processing of job information in a timely manner.

[0103] It is understandable that the batch acquisition plug-in can also monitor the acquisition transmission efficiency and / or the acquisition data sending results based on the monitoring component. The process is similar to the execution process of the streaming acquisition plug-in and will not be repeated here.

[0104] Therefore, the embodiments of the present application monitor the cache occupancy and dynamically adjust the execution of the collection task according to the cache occupancy, so as to avoid system crash or performance degradation caused by cache overload, and suspend the collection task when the cache occupancy is high, so as to avoid resource overload and improve the utilization rate of the overall system resources; by monitoring the execution link and data sending results of the collection task, the abnormal situation in the transmission is discovered and handled in time, so as to improve the reliability of data transmission; wherein, each collection plug-in is independent of each other, and is easy to expand and maintain, so as to flexibly respond to different data collection tasks, and is monitored and controlled through a predefined interface, so as to have high flexibility and scalability, and adapt to different data sources and collection requirements.

[0105] It should be noted that Flume streaming collection does not support automatic retransmission, so data loss may occur in the event of network anomalies.

[0106] In one possible design, the method further includes:

[0107] If it is determined based on the acquisition transmission efficiency and / or the acquisition data sending result that the acquisition task is executed abnormally, the data collected by the acquisition plug-in is resent.

[0108] In an embodiment of the present application, if it is determined that the collected data has failed to be sent, the collection plug-in is supported to automatically resend the data to avoid data loss.

[0109] Exemplarily, taking the collection plug-in as a streaming collection plug-in as an example, when the Sink component sends at least one group of collected data to the downstream, if there is a group of data that is not sent successfully, the data of that group can be sent again. Optionally, at least all of the collected data can be sent again. The embodiment of the present application does not make any specific limitations on this.

[0110] Optionally, if the collection transmission efficiency is less than the transmission threshold, it means that there may be an abnormality in the transmission process, such as resource overload, then the data transmission can be suspended, and after waiting for a period of time, the data collected by the collection plug-in can be resent. The embodiment of the present application does not specifically limit the abnormal situation, and the resending timing for different abnormal situations is different.

[0111] It should be noted that if the collection transmission efficiency is less than the transmission threshold, other strategies can also be used to execute the collection task, such as changing the transmission link, etc. The embodiment of the present application does not specifically limit the strategy adopted, which can be determined based on the specific application scenario.

[0112] Therefore, in the embodiment of the present application, when an acquisition task is executed abnormally, the reissue mechanism can ensure that the collected data will not be lost and the integrity of the data is guaranteed. In abnormal circumstances, the system can automatically reissue the data, reduce manual intervention, and improve the reliability of the system. The reissue mechanism also enhances the fault tolerance of the system, allowing the system to better cope with various abnormal situations. The automated reissue mechanism reduces manual maintenance costs and improves the operating efficiency of the system.

[0113] In one possible design, the method further includes:

[0114] The log information generated by the collection plug-in is managed to the client based on a predefined interface, and is visually displayed on the client so that the user can view the log information.

[0115] In the embodiment of the present application, since a predefined interface is provided, which is a globally unique log access portal, the log information generated by each collection plug-in can be uniformly managed to the client's log module to facilitate users to view logs and avoid the problem of deep log levels and scattered distribution.

[0116] The log information provides a historical record of system operations, which facilitates tracing back and analyzing the root cause of the problem. Through the predefined interface, the display mode of the log information can be flexibly configured to meet different business needs.

[0117] It is understandable that users can view the log information generated by the collection plug-in in real time based on the client UI, understand the system's operating status and the execution of collection tasks, and by viewing the log information, users can quickly locate and identify problems in the system, simplifying the troubleshooting process.

[0118] Therefore, the embodiment of the present application can perform unified log management based on a predefined interface. By monitoring the log information, the user can perform preventive maintenance of the collection plug-in, reduce the occurrence rate of system failures, and the visual display makes the log information more intuitive. The user can more easily understand and analyze the operation of the system, and can also conveniently access and view the log information on the client, thereby improving the user experience.

[0119] In the above embodiments, the acquisition plug-in management method provided by the embodiments of the present application is introduced. In order to realize the functions in the method provided by the embodiments of the present application, the electronic device as the execution subject may include a hardware structure and / or a software module, and the above functions are realized in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0120] For example, Figure 8 A schematic diagram of the structure of a collection plug-in management device provided in an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0121] The receiving module 801 is used to receive an upgrade request sent by the client and start the upgrade process based on the upgrade request;

[0122] The determination module 802 is used to obtain the version file of each acquisition plug-in after starting the upgrade process, and determine the acquisition plug-in to be upgraded from the acquisition plug-ins based on the version file; the acquisition plug-in is integrated in the full plug-in package;

[0123] The download module 803 is used to download the upgrade package corresponding to the acquisition plug-in to be upgraded based on a predefined interface, and back up the previous version of the acquisition plug-in to be upgraded;

[0124] The upgrade module 804 is used to upgrade the acquisition plug-in to be upgraded based on the upgrade package to obtain an upgrade result, and after determining that the upgrade of the acquisition plug-in to be upgraded fails based on the upgrade result, roll back the acquisition plug-in to be upgraded to the acquisition plug-in corresponding to the previous version.

[0125] In a possible design, the upgrade module 804 is specifically used to:

[0126] The preset files of the acquisition plug-in to be upgraded are upgraded based on the upgrade package, and the upgrade result of each preset file is obtained; the preset files include: executable files, dependent packages and configuration files.

[0127] In one possible design, the apparatus further includes an updating module, wherein the updating module is configured to:

[0128] After determining that the acquisition plug-in to be upgraded is upgraded successfully based on the upgrade result, closing the upgrade process, and updating the full plug-in package based on the acquisition plug-in to be upgraded that is upgraded successfully;

[0129] Back up the collection plug-in in the full plug-in package and restart the client.

[0130] In a possible design, each collection plug-in is independent of each other and is used to complete the collection task of the data source; the device also includes a monitoring module, and the monitoring module is used to:

[0131] Based on a predefined interface, monitoring whether the cache occupancy of the acquisition plug-in is greater than a preset threshold; the acquisition plug-in includes a streaming acquisition plug-in and a batch acquisition plug-in;

[0132] If so, control the acquisition plug-in to stop executing the acquisition task for the data source until the cache occupancy is greater than or equal to a preset threshold, and then execute the acquisition task for the data source;

[0133] If not, the collection transmission efficiency and / or the collection data sending result of the execution link of the collection task are monitored to determine whether the collection task is executed abnormally.

[0134] In a possible design, the apparatus further includes a reissue module, wherein the reissue module is configured to:

[0135] If it is determined based on the acquisition transmission efficiency and / or the acquisition data sending result that the acquisition task is executed abnormally, the data collected by the acquisition plug-in is resent.

[0136] In a possible design, the apparatus further includes a management module, and the management module is used to:

[0137] The log information generated by the collection plug-in is managed to the client based on a predefined interface, and is visually displayed on the client so that the user can view the log information.

[0138] The collection plug-in management device provided in the embodiment of the present application can be used to execute the technical solution of the collection plug-in management method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0139] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the download module 803 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the download module 803. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0140] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device may include: a transceiver 901 , a processor 902 , and a memory 903 .

[0141] The processor 902 executes the computer execution instructions stored in the memory, so that the processor 902 executes the scheme in the above embodiment. The processor 902 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0142] The memory 903 is connected to the processor 902 via a system bus and completes communication between them. The memory 903 is used to store computer program instructions.

[0143] The transceiver 901 may be configured to receive an upgrade request sent by a client.

[0144] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0145] The electronic device provided in the embodiment of the present application may be the client of the above embodiment.

[0146] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solution of the collection plug-in management method in the above embodiment.

[0147] An embodiment of the present application further provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed on a computer, the computer executes the technical solution of the acquisition plug-in management method in the above embodiment.

[0148] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the collection plug-in management method in the above embodiment can be implemented.

[0149] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0150] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0151] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0152] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0153] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0154] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0155] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0156] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0157] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0158] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A collection plug-in management method, characterized in that: The method comprises: Receiving an upgrade request sent by a client, and starting an upgrade process based on the upgrade request; After starting the upgrade process, obtaining the version file of each acquisition plug-in, and determining the acquisition plug-in to be upgraded from the acquisition plug-ins based on the version file; the acquisition plug-in is integrated into the full plug-in package; Downloading the upgrade package corresponding to the acquisition plug-in to be upgraded based on a predefined interface, and backing up the previous version of the acquisition plug-in to be upgraded; The acquisition plug-in to be upgraded is upgraded based on the upgrade package to obtain an upgrade result, and after determining that the acquisition plug-in to be upgraded fails to be upgraded based on the upgrade result, the acquisition plug-in to be upgraded is rolled back to the acquisition plug-in corresponding to the previous version.

2. The method according to claim 1, characterized in that The step of upgrading the acquisition plug-in to be upgraded based on the upgrade package to obtain an upgrade result includes: The preset files of the acquisition plug-in to be upgraded are upgraded based on the upgrade package, and the upgrade result of each preset file is obtained; the preset files include: executable files, dependent packages and configuration files.

3. The method according to claim 1, characterized in that The method further comprises: After determining that the acquisition plug-in to be upgraded is upgraded successfully based on the upgrade result, closing the upgrade process, and updating the full plug-in package based on the acquisition plug-in to be upgraded that is upgraded successfully; Back up the collection plug-in in the full plug-in package and restart the client.

4. The method according to claim 1, characterized in that: Each collection plug-in is independent of each other and is used to complete the collection task of the data source; the method also includes: Based on a predefined interface, monitoring whether the cache occupancy of the acquisition plug-in is greater than a preset threshold; the acquisition plug-in includes a streaming acquisition plug-in and a batch acquisition plug-in; If so, control the acquisition plug-in to stop executing the acquisition task for the data source until the cache occupancy is greater than or equal to a preset threshold, and then execute the acquisition task for the data source; If not, the collection transmission efficiency and / or the collection data sending result of the execution link of the collection task are monitored to determine whether the collection task is executed abnormally.

5. The method according to claim 4, characterized in that The method further comprises: If it is determined based on the acquisition transmission efficiency and / or the acquisition data sending result that the acquisition task is executed abnormally, the data collected by the acquisition plug-in is resent.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The log information generated by the collection plug-in is managed to the client based on a predefined interface, and is visually displayed on the client so that the user can view the log information.

7. A collection plug-in management device, characterized in that: The device comprises: A receiving module, used to receive an upgrade request sent by a client, and start an upgrade process based on the upgrade request; A determination module, used for obtaining a version file of each acquisition plug-in after starting the upgrade process, and determining the acquisition plug-in to be upgraded from the acquisition plug-ins based on the version file; the acquisition plug-in is integrated in the full plug-in package; A download module, used to download the upgrade package corresponding to the acquisition plug-in to be upgraded based on a predefined interface, and back up the previous version of the acquisition plug-in to be upgraded; The upgrade module is used to upgrade the acquisition plug-in to be upgraded based on the upgrade package to obtain an upgrade result, and after determining that the upgrade of the acquisition plug-in to be upgraded fails based on the upgrade result, roll back the acquisition plug-in to be upgraded to the acquisition plug-in corresponding to the previous version.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images