Integrated Data Acquisition Method for Multi-Source Data Fusion

Abstract

The present invention belongs to the technical field of multi-source data fusion, and discloses an integrated data acquisition method for multi-source data fusion. The 7xOPS Agent is automatically and manually registered as follows: S1: Implement global initialization of the configuration file, global initialization of logging, configure the proxy IP pool, and initialize the system; S2: Request the 7xOPS Server to allocate a Token, and automatically request the 7xOPS Server to obtain the Token every 5 minutes, starting in a coroutine manner. The present invention provides an integrated data acquisition method for multi-source data fusion. By managing the 7xOPS Agent through the 7xOPS Server, operations such as installation, upgrade, uninstallation, and editing can be realized. The task types include performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, UCPE monitoring, and other tasks. The 7xOPS Server realizes operations such as starting, stopping, editing, and deleting tasks. Data acquisition for different task types is realized through the 7xOPS Agent, and the 7xOPS Receiver realizes the storage of the collected data.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-source data fusion, and specifically relates to an integrated data acquisition method for multi-source data fusion. Background Art

[0002] With the popularization of broadband networks and the increasing demand of industry management departments for improving management levels, the demand of management departments such as urban public security, transportation, finance, environmental protection, electric power, medical care, and education for large-scale networked security monitoring platforms within the city has also increased significantly in the past two years. With the maturity of network transmission technologies and the Internet, significant breakthroughs have been made in realizing the needs of these emerging industry customers, and large-scale automated operation and maintenance monitoring systems are being rapidly built across the country. The currently common collectors mainly support operating systems such as Windows, various Linux, and Unix. R & D is carried out for specific operating systems, and each operating system has its own independent development environment, with disadvantages such as high R & D costs and difficult code maintenance. Summary of the Invention

[0003] To solve the problems raised in the above background art, the present invention provides an integrated data acquisition method for multi-source data fusion. By managing 7xOPS Agent through 7xOPS Server, operations such as installation, upgrade, and uninstallation can be realized. The task types include performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, UCPE monitoring, etc. 7xOPS Server realizes operations such as deletion, editing, start, and stop of tasks. 7xOPS Agent realizes unified entry management and data acquisition for different task types, and 7xOPS Receiver realizes storage of the collected data.

[0004] To achieve the above object, the present invention provides the following technical solution: an integrated data acquisition method for multi-source data fusion, where the 7xOPS Agent is registered automatically and manually;

[0005] S1: Implement global initialization of configuration files, global initialization of logging, configure the proxy IP pool, and system initialization;

[0006] S2: Request 7xOPS Server to obtain a Token, automatically request 7xOPS Server to obtain a Token every 5 minutes, and start in a coroutine manner;

[0007] The Token generation rule is to randomly generate based on the username and password. After obtaining the returned Token, an assignment operation is performed, and 7xOPS Agent is assigned to the object globally configured;

[0008] S3: Register and verify the obtained Token in 7xOPS Server to determine whether 7xOPS Agent is automatically or manually deployed;

[0009] S31: Automatically deploy and request 7xOPS Server, update 7xOPS Agent management status, and return the request result. If successful, update the global configuration. If failed, exit the current function and re-initiate the request.

[0010] S32: Manual deployment requests 7xOPS Server to register the newly deployed 7xOPS Agent. If successful, the global configuration is updated. If failed, the program is exited and the process is terminated.

[0011] S4: 7xOPS Agent registration is completed.

[0012] Preferably, the 7xOPS Agent automatically deploys:

[0013] S1: Create a new Agent in the 7xOPS Agent management list. You will need the installation address IP, root account and password, installation path, download link and other information.

[0014] S2: 7xOPS Server starts SSH to establish a link. After the link is successfully established, the shell script is uploaded, the shell script is executed, and the path is jumped to the specified path to download the installation package file. After the download is successful, it is started;

[0015] S3: 7xOPS Agent is started successfully and registered with 7xOPS Server. 7xOPS Agent will send Agent status updates regularly and send installation logs to MQ.

[0016] S4: 7xOPS Server receives the status update message from MQ, changes the status of the 7xOPS Agent created in the 7xOPS Agent management list to running, and saves the installation log.

[0017] Preferably, the 7xOPS Agent is automatically upgraded and uninstalled:

[0018] S1: Click Upgrade in the 7xOPS Agent management list. The 7xOPS Server sends tasks to the corresponding MQ queue, and the 7xOPS Agent listens to the MQ tasks sent by the 7xOPS Server.

[0019] S2: The 7xOPS Agent monitors the information to be upgraded, parses the received information, executes the shell script, downloads the installation package through the link, and determines whether the 7xOPS Agent is started by checking the process after installation;

[0020] S21: During startup: Back up the currently running installation package, kill the current program process, run the new installation package, and send a status message to the Mq task after successful startup;

[0021] S22: Not started: Back up the old installation package, run the new installation package, and send a status message to the Mq task after successful startup;

[0022] S3: The 7xOPS Server receives the status message from the Mq, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list to running, and saves the upgrade log.

[0023] Preferably, the monitoring tasks collect:

[0024] The 7xOPS Server realizes the management of the 7xOPS Agent monitoring tasks, starts and stops various tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring. After the task starts, according to the task type, the monitoring information template is sent to the corresponding Mq queue;

[0025] The 7xOPS Agent monitors the Mq tasks issued by the 7xOPS Server, performs different data monitoring according to the task type, and performs data cleaning, data filtering, data integration, data queuing, data structure optimization, data compression on the monitored data, and finally sends the data;

[0026] The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

[0027] Preferably, the performance collection and monitoring process:

[0028] S1: The 7xOPS Agent task list starts the performance monitoring task. The 7xOPS Agent receives the Mq task information issued by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required;

[0029] S2: Assign the Context context to the data after passing the verification so that it can be used throughout the next monitoring task, install the plugin shell script, and user-defined collection metrics;

[0030] S3: Run the performance monitoring task, and use the Linux (obtain built-in files) and Windows (use commands and dll files) methods to read data and obtain system device information such as cpu, memory, and wireless network cards;

[0031] S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending the data;

[0032] S5: Send the data monitored by the 7xOPS Agent to the 7xOPS Receiver in the way of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data;

[0033] The SNMP collection process:

[0034] S1: The 7xOPS Agent task list starts the SNMP monitoring task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required;

[0035] S2: Assign the Context context to the data after verification so that it can be used throughout the next monitoring task;

[0036] S3: Run the SNMP monitoring task. Use the SNMP walk / get request method to obtain basic information such as the cpu and memory of the device with the Oid and IP as input parameters, and data monitoring can also be performed through the Ping method;

[0037] S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending the data;

[0038] S5: Send the data monitored by the 7xOPS Agent to the 7xOPS Receiver in the way of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

[0039] Preferably, the asset scanning process:

[0040] S1: The 7xOPS Agent task list starts the asset scanning task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required;

[0041] S2: For the data that has passed the verification, the task ID needs to be added to the global variable Map array created for the asset scanning task, so that the received task can be started / stopped subsequently;

[0042] S3: Run the asset scanning task, and perform different data monitoring according to the data type (icmp / snmp) in the task information. Using the SNMP walk / get request method with the Oid and IP as input parameters, the basic information of the system environment can be obtained, and data monitoring can also be performed through the Ping method;

[0043] S4: Clean, filter, and integrate the monitored data before sending it;

[0044] S5: Send the data monitored by the 7xOPS Agent to the 7xOPS Server, and the 7xOPS Server stores the collected data.

[0045] Preferably, the status monitoring process is as follows:

[0046] S1: The 7xOPS Agent starts the status monitoring task in the task list. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required;

[0047] S2: The task ID needs to be added to the global variable Map array created for the status monitoring task, so that the received task can be started and stopped subsequently;

[0048] S3: Run the status monitoring task, and perform data monitoring based on the IP in the task information. Use the Ping method to obtain data such as whether normal communication is possible;

[0049] S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending it;

[0050] S5: Send the data monitored by the 7xOPS Agent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver stores the collected data;

[0051] The sensor monitoring process is as follows:

[0052] S1: The 7xOPS Agent starts the sensor monitoring task in the task list. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required;

[0053] S2: The task ID needs to be added to the global variable Map array created by the sensor monitoring task so that the received task can be started and stopped subsequently;

[0054] S3: Run the sensor monitoring task. The IOT device used by the user periodically sends device information to the specified queue, and the 7xOPS Agent starts listening to this queue after the task is started, and reads data through the method of listening to the exclusive queue;

[0055] S4: The monitored data is subjected to data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression before being sent;

[0056] S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the way of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

[0057] Preferably, the domain name detection process:

[0058] S1: The 7xOPS Agent task list starts the domain name monitoring task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required;

[0059] S2: The task ID needs to be added to the global variable Map array created by the domain name monitoring task so that the received task can be started and stopped subsequently;

[0060] S3: Run the domain name monitoring task, monitor data according to the IP in the task information, and use the Ping method to obtain data such as packet loss, latency, and status code;

[0061] S4: The monitored data is subjected to data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression before being sent;

[0062] S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the way of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

[0063] Preferably, the UCPE acquisition process:

[0064] S1: The 7xOPS Agent task list starts the UCPE monitoring task. The 7xOPS Agent receives the MQ task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before starting the monitoring task, data verification is required.

[0065] S2: The task ID needs to be added to the global variable Map array created by the UCPE monitoring task so that the received task can be started and stopped later;

[0066] S3: Run the UCPE monitoring task, execute the shell script to obtain the 7xNetWork device information, and read the shell script to monitor the data;

[0067] S4: The monitored data is cleaned, filtered, integrated, queued, and structured, and the data is compressed before being sent;

[0068] S5: The data monitored by 7xOPS Agent is sent to 7xOPS Receiver via Http / Rpc, and 7xOPS Receiver stores the collected data.

[0069] Preferably, the 7xOPS Agent task starts and stops: 7xOPS Server implements 7xOPS Agent monitoring task management, and can start and stop different tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring;

[0070] When 7xOPS Server sends tasks to 7xOPS Agent through MQ, 7xOPS Agent runs the start and stop programs according to the data parsed in the send.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] The present invention provides an integrated data collection method for multi-source data fusion. Through 7xOPS Server, 7xOPSAgent is managed to achieve installation, upgrade, uninstallation and other operations. Task types include performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, UCPE monitoring and other tasks. 7xOPS Server implements the installation, upgrade, uninstallation, start, stop and other operations of tasks. Through 7xOPS Agent, unified entry management and data collection of different task types are achieved. 7xOPS Receiver implements the storage of collected data. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 Schematic diagram of the architecture of the present invention;

[0074] Figure 2 Schematic diagram of the automatic and manual registration process of the 7xOPS Agent of the present invention;

[0075] Figure 3 Schematic diagram of the automatic deployment process of the 7xOPS Agent of the present invention;

[0076] Figure 4 Schematic diagram of the automatic upgrade and uninstallation process of the 7xOPS Agent of the present invention;

[0077] Figure 5 Schematic diagram of the task collection process of the present invention;

[0078] Figure 6 Schematic diagram of the performance collection process of the present invention;

[0079] Figure 7 Schematic diagram of the SNMP collection process of the present invention;

[0080] Figure 8 Schematic diagram of the asset scanning process of the present invention;

[0081] Figure 9 Schematic diagram of the status detection process of the present invention;

[0082] Figure 10 Schematic diagram of the sensor detection process of the present invention;

[0083] Figure 11 Schematic diagram of the domain name detection process of the present invention;

[0084] Figure 12 Schematic diagram of the UCPE collection process of the present invention;

[0085] Figure 13 Schematic diagram of the task start and stop process of the 7xOPS Agent of the present invention. Detailed implementation manners

[0086] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0087] Such as Figures 1 to 13As shown in the figure, the present invention provides an integrated data acquisition method for multi-source data fusion. For the integrated data acquisition method of multi-source data fusion, the 7xOPS Agent can be registered automatically or manually:

[0088] S1: Implement global initialization of the configuration file, global initialization of logging, configure the proxy IP pool, and system initialization;

[0089] S2: Request the 7xOPS Server to obtain a Token. Automatically request the 7xOPS Server to obtain a Token every 5 minutes and start in a coroutine manner;

[0090] The Token generation rule is to randomly generate based on the username and password. After obtaining the returned Token, perform an assignment operation and assign it to the object in the global configuration;

[0091] S3: Verify the registration of the obtained Token on the 7xOPS Server and determine whether the 7xOPS Agent is deployed automatically / manually;

[0092] S31: For automatic deployment, request the 7xOPS Server, update the management status of the 7xOPS Agent, and return the request result. If successful, update the global configuration; if failed, exit the current function and make a new request;

[0093] S32: For manual deployment, request the 7xOPS Server to register the newly deployed 7xOPS Agent. If successful, update the global configuration; if failed, exit the program and end the process;

[0094] S4: The registration of the 7xOPS Agent is completed;

[0095] As Figure 3 shown, the automatic deployment of the 7xOPS Agent:

[0096] S1: Create a new monitoring task in the 7xOPS Agent management list, which requires information such as the installation address IP, root account and password, installation path, download link, etc.;

[0097] S2: The 7xOPS Server starts SSH to establish a connection. After the connection is successfully established, upload the shell script, execute the shell script, jump to the specified path, download the installation package file, and start after successful download;

[0098] S3: The 7xOPS Agent starts successfully and is registered on the 7xOPS Server. The 7xOPS Agent will regularly send updates on the monitoring task status and attach the installation log;

[0099] S4: The 7xOPS Server receives the status update message, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list to running, and saves the installation log.

[0100] As Figure 4 shown, the 7xOPS Agent automatic upgrade and uninstallation are as follows:

[0101] S1: Click to start in the 7xOPS Agent management list, the 7xOPS Server sends tasks to the corresponding Mq queue, and the 7xOPS Agent listens for the Mq tasks sent by the 7xOPS Server;

[0102] S2: The 7xOPS Agent listens for the information to be upgraded, executes the shell script, downloads the installation package through the link, and determines whether the 7xOPS Agent is started;

[0103] S21: During startup: Back up the currently running installation package, kill the current program process, run the new installation package, and send a status message to the Mq task after successful startup;

[0104] S22: Not started: Back up the old installation package, run the new installation package, and send a status message to the Mq task after successful startup;

[0105] S3: The 7xOPS Server receives the status update message, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list to running, and saves the upgrade / uninstall log.

[0106] As Figure 5 shown, the monitoring task collection is as follows:

[0107] The 7xOPS Server realizes the management of the 7xOPS Agent monitoring tasks, starts and stops various tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring. After the task is started, according to the task type, the monitoring information template is sent to the corresponding Mq queue;

[0108] The 7xOPS Agent listens for the tasks sent by the 7xOPS Server, performs different data monitoring according to the task type, and performs data cleaning, data filtering, data integration, data queuing, data structure optimization, data compression on the monitored data, and finally sends the data;

[0109] The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data;

[0110] The 7xOPS Agent is unified and applicable to monitoring most business scenarios in the market. It can also create custom templates. For example, some systems only have SNMP monitoring / ping monitoring, and most only have SNMP monitoring;

[0111] The "7xOPS System Agent Unified Collector" integrates tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, UCPE monitoring, and custom templates in a project, with simple configuration, facilitating user operations such as use and unified management.

[0112] System compatibility, suitable for development in system environments such as Linux, Windows, and Mac. For example, due to the lack of uniformity in system commands in Linux, Windows, and Mac, project development needs to be independently developed in different system environments.

[0113] The "7xOPS System Agent Unified Collector" solves this problem. It executes the corresponding business line according to the currently obtained system environment (including the temporarily assigned system environment), thereby reducing the expenditure on R & D costs.

[0114] Such as Figure 6 、 Figure 7 As shown, the performance collection and monitoring process:

[0115] S1: The 7xOPS Agent task list starts the performance monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required;

[0116] S2: Assign the Context context to the data after verification so that it can be used at any time in the subsequent monitoring tasks. Install the plugin shell script and user-defined collection metrics;

[0117] S3: Run the performance monitoring task, and use methods such as Linux (obtaining built-in files) and Windows (using commands and dll files) to read data and obtain system device information such as cpu, memory, and wireless network cards;

[0118] S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending it;

[0119] S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data;

[0120] Intelligent deployment enables automated operations for system installation / upgrade / uninstallation and other services, reducing human resources and enhancing the customer experience. It addresses the differences between the internal and external networks in implementing these services.

[0121] The "7xOPS System Agent Unified Collector" solves this problem and supports operations such as system self-startup / installation / upgrade in different environments.

[0122] SNMP collection process:

[0123] S1: The 7xOPS Agent task list starts the SNMP monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required.

[0124] S2: Assign the Context context to the verified data so that it can be used at any time in the subsequent monitoring task.

[0125] S3: Run the SNMP monitoring task. Using the SNMP walk / get request method with the Oid and IP as input parameters, basic information such as the device's CPU and memory can be obtained, and data monitoring can also be performed through the Ping method.

[0126] S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending it.

[0127] S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver stores the collected data.

[0128] As Figure 8 shown, the asset scanning process:

[0129] S1: The 7xOPS Agent task list starts the asset scanning task. The 7xOPS Agent receives the task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required.

[0130] S2: For the verified data, the task Id needs to be added to the global variable Map array created by the asset scanning task so that the received task can be started / stopped subsequently.

[0131] S3: Run the asset scanning task, perform different data monitoring according to the data types (icmp / snmp) in the task information. Use the SNMP walk / get request method to obtain the basic system environment information with Oid and IP as input parameters, or perform data monitoring through the Ping method;

[0132] S4: Clean, filter, and integrate the monitored data before sending it;

[0133] S5: Send the data monitored by the 7xOPS Agent to the 7xOPS Server, and the 7xOPS Server stores the collected data;

[0134] Data interaction: The size of the data volume has little impact on the data transmission bandwidth in the intranet transmission, while data loss and connection timeouts may occur in the interaction with the external network due to excessive transmission memory;

[0135] The "7xOPS System Agent Unified Collector" provides solutions such as data structure optimization and transmission data compression to solve the data loss problem caused by excessive data and resulting bandwidth in the external network transmission;

[0136] Such as Figure 9 、 Figure 10 As shown, the status monitoring process:

[0137] S1: The 7xOPS Agent task list starts the status monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required;

[0138] S2: The task Id needs to be added to the global variable Map array created by the status monitoring task so that the received tasks can be started and stopped subsequently;

[0139] S3: Run the status monitoring task, perform data monitoring based on the IP in the task information, and use the Ping method to obtain data such as whether normal communication is possible;

[0140] S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending it;

[0141] S5: Send the data monitored by the 7xOPS Agent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver stores the collected data;

[0142] Sensor monitoring process:

[0143] S1: The 7xOPS Agent task list starts the sensor monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required.

[0144] S2: The task ID needs to be added to the global variable Map array created by the sensor monitoring task so that the received task can be started and stopped subsequently.

[0145] S3: Run the sensor monitoring task. The IOT device used by the user regularly sends device information to the specified queue. The 7xOPS Agent starts listening to this queue after the task is started and reads data through the method of listening to the exclusive queue.

[0146] S4: The monitored data is subjected to data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression before being sent.

[0147] S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

[0148] As Figure 11 shown, the domain name detection process:

[0149] S1: The 7xOPS Agent task list starts the domain name monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required.

[0150] S2: The task ID needs to be added to the global variable Map array created by the domain name monitoring task so that the received task can be started and stopped subsequently.

[0151] S3: Run the domain name monitoring task. Monitor data according to the IP in the task information and use the Ping method to obtain data such as packet loss, latency, and status code.

[0152] S4: The monitored data is subjected to data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression before being sent.

[0153] S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

[0154] like Figure 12 As shown, the UCPE collection process:

[0155] S1: The 7xOPS Agent task list starts the UCPE monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before starting the monitoring task, data verification is required.

[0156] S2: The task ID needs to be added to the global variable Map array created by the UCPE monitoring task so that the received task can be started and stopped later;

[0157] S3: Run the UCPE monitoring task, execute the shell script to obtain the 7xNetWork device information, and read the shell script to monitor the data;

[0158] S4: The monitored data is cleaned, filtered, integrated, queued, and structured, and the data is compressed before being sent;

[0159] S5: The data monitored by 7xOPS Agent is sent to 7xOPS Receiver via Http / Rpc, and 7xOPS Receiver stores the collected data.

[0160] Data collection, real-time data collection. Sometimes a data collection task needs to maintain data collection of more than 500 machines, which poses a problem of real-time data acquisition.

[0161] "7xOPS System Agent Unified Collector" uses go coroutine to obtain data. Because the number of coroutines opened is too large, the server CPU usage rate increases. The coroutine pool is used for optimization to solve the problem of data real-time performance not meeting standards due to excessive collected data.

[0162] like Figure 13 As shown, 7xOPS Agent task start and stop: 7xOPS Server implements 7xOPS Agent monitoring task management, and can start and stop different tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring;

[0163] When 7xOPS Server sends tasks to 7xOPS Agent through MQ, 7xOPS Agent runs the start and stop programs according to the data parsed in the send.

[0164] The 7xOPS Agent receives a task to start implementing monitoring services according to different task types; when the parsed data needs to be passed with context, the received message is assigned to the context, and when it is not needed, only the task ID is assigned to the Map array generated by the corresponding monitoring task.

[0165] When the 7xOPS Agent receives a stop task, it stops the monitoring services according to different task types; according to the assignment operation during startup, it operates on the context / Map to interrupt and stop the monitoring task.

[0166] When the monitoring task of the 7xOPS Agent is completed, the scheduled task status maintenance sends the executed task status to the corresponding Mq queue, and the 7xOPS Server consumes it regularly, reads this information, and updates the task status in the monitoring task list of the 7xOPS Agent in a timely manner.

[0167] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0168] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An integrated data acquisition method for multi-source data fusion, characterized in that: The specific steps for automatic and manual registration of 7xOPS Agent are as follows: S1: Implement global initialization of configuration files, global initialization of print logs, configuration of proxy IP pool, and system initialization; S2: Request 7xOPS Server to obtain Token. Automatically request 7xOPS Server to obtain Token every 5 minutes and start in coroutine mode. The token generation rule is randomly generated based on the user name and password. After obtaining the returned token, the token is assigned to the object of the global configuration by 7xOPS Agent. S3: Register and verify the obtained Token in 7xOPS Server to determine whether 7xOPS Agent is automatically or manually deployed; S31: Automatically deploy and request 7xOPS Server, update 7xOPS Agent management status, and return the request result. If successful, update the global configuration. If failed, exit the current function and re-initiate the request. S32: Manual deployment requests 7xOPS Server to register the newly deployed 7xOPS Agent. If successful, the global configuration is updated. If failed, the program is exited and the process is terminated. S4: 7xOPS Agent registration completed; The 7xOPS Agent also includes monitoring task collection: 7xOPS Server manages 7xOPS Agent monitoring tasks, including the following tasks: performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and start and stop of UCPE monitoring; After the task is started, the monitoring information template is sent to the corresponding MQ queue according to the task type; 7xOPS Agent listens to the MQ tasks sent by 7xOPS Server, monitors different data according to the type of task, cleans, filters, integrates, queues, optimizes the data structure, compresses the data, and finally sends the data; The data monitored by 7xOPS Agent is sent to 7xOPS Receiver via Http / Rpc, and 7xOPS Receiver stores the collected data. 7xOPS Agent receives a task to start monitoring services according to different task types. When the parsed data needs to be passed in context, the received message is assigned to the context. If not, the task ID is assigned to the Map array generated by the corresponding monitoring task. When 7xOPS Agent receives a task to stop, it stops the monitoring service according to the task type. According to the assignment operation at startup, it operates the Context / Map to interrupt the monitoring task and stop monitoring. When the 7xOPS Agent monitoring task is completed, the timing task status maintenance will send the status of the completed task to the corresponding Mq queue, and the 7xOPS Server will consume it regularly, read this information and update the task status in the 7xOPS Agent monitoring task list in a timely manner.

2. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: The automatic deployment of the 7xOPS Agent is as follows: S1: Create a new Agent in the 7xOPS Agent management list, which requires information such as the installation address IP, root account and password, installation path, download link, etc. S2: The 7xOPS Server starts an SSH connection. After the connection is successfully established, upload the shell script, execute the shell script, jump to the specified path, download the installation package file, and start it after the download is successful. S3: When the 7xOPS Agent starts successfully and is registered in the 7xOPS Server, the 7xOPS Agent will regularly send the Agent status for update and attach the installation log to send to Mq. S4: The 7xOPS Server receives the status update message from Mq, changes the status of the 7xOPS Agent created in the 7xOPS Agent management list to running, and saves the installation log.

3. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: The automatic upgrade and uninstallation of the 7xOPS Agent are as follows: S1: Click upgrade in the 7xOPS Agent management list, and the 7xOPS Server issues a task to the corresponding Mq queue. The 7xOPSAgent listens to the Mq task issued by the 7xOPS Server. S2: When the 7xOPS Agent monitors the information to be upgraded, parses the received information, executes the shell script, downloads the installation package through the link, and checks whether the 7xOPS Agent is started by viewing the process after the installation is completed. S21: During startup: Back up the currently running installation package, kill the current program process, run the new installation package, and send a status message to the Mq task after successful startup. S22: Not started: Back up the old installation package, run the new installation package, and send a status message to the Mq task after successful startup. S3: The 7xOPS Server receives the status message from Mq, changes the status of the newly created 7xOPSAgent in the 7xOPS Agent management list to running, and saves the upgrade log.

4. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: The performance collection and monitoring process is as follows: S1: The 7xOPS Agent task list starts the performance monitoring task. The 7xOPS Agent receives the Mq task information issued by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required. S2: Assign values to the Context context for the data after passing the verification, so that it can be used throughout the next monitoring task, install the plugin shell script, and user-defined collection metrics. S3: Run the performance monitoring task, and use the Linux (obtain built-in files) and Windows (use commands and dll files) methods to read data and obtain system device information such as cpu, memory, and wireless network card; S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending it; S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the way of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data; The SNMP collection process: S1: The 7xOPS Agent task list starts the SNMP monitoring task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required; S2: Assign the Context context to the data after passing the verification so that it can be used throughout the subsequent monitoring task; S3: Run the SNMP monitoring task. Use the SNMP walk / get request method to obtain basic information such as the cpu and memory of the device with the Oid and IP as input parameters, and data monitoring can also be performed through the Ping method; S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending it; S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the way of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

5. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: The asset scanning process: S1: The 7xOPS Agent task list starts the asset scanning task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required; S2: For the data after passing the verification, the task Id needs to be added to the global variable Map array created by the asset scanning task so that the received task can be started / stopped subsequently; S3: Run the asset scanning task, perform different data monitoring according to the data type (icmp / snmp) in the task information. Use the SNMP walk / get request method to obtain basic information about the system environment with the Oid and IP as input parameters, and data monitoring can also be performed through the Ping method; S4: Clean, filter, and integrate the monitored data before sending it; S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Server, and the 7xOPS Server realizes the storage of the collected data.

6. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: The status monitoring process: S1: The 7xOPS Agent task list starts the status monitoring task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required; S2: The task ID needs to be added to the global variable Map array created by the status monitoring task so that the received tasks can be started and stopped later; S3: Run the status monitoring task, monitor data according to the IP in the task information, and use the Ping method to obtain data such as whether normal communication can be achieved; S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending; S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data; The sensor monitoring process is as follows: S1: The 7xOPS Agent task list starts the sensor monitoring task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required; S2: The task ID needs to be added to the global variable Map array created by the sensor monitoring task so that the received tasks can be started and stopped later; S3: Run the sensor monitoring task. The IOT device used by the user periodically sends device information to the specified queue. The 7xOPS Agent starts listening to this queue after the task is started and reads data through the method of listening to the exclusive queue; S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending; S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

7. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: The domain name detection process is as follows: S1: The 7xOPS Agent task list starts the domain name monitoring task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required; S2: The task ID needs to be added to the global variable Map array created by the domain name monitoring task so that the received tasks can be started and stopped later; S3: Run the domain name monitoring task, monitor data according to the IP in the task information, and use the Ping method to obtain data such as packet loss, latency, and status code; S4: Clean, filter, integrate, queue, optimize the data structure, and compress the monitored data before sending; S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the way of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

8. The integrated data acquisition method for multi-source data fusion according to claim 1, wherein: The UCPE collection process: S1: The 7xOPS Agent task list starts the UCPE monitoring task. The 7xOPS Agent receives the Mq task information sent by the 7xOPS Server and executes the corresponding monitoring task. Before preparing to start the monitoring task, data verification is required. S2: The task Id needs to be added to the global variable Map array created by the UCPE monitoring task so that the received tasks can be started and stopped subsequently. S3: Run the UCPE monitoring task, execute the shell script to obtain the 7xNetWork device information, and read the data monitored by the shell script. S4: The monitored data is subjected to data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression before being sent. S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the way of Http / Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

9. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: The start and stop of the 7xOPS Agent task: The 7xOPS Server realizes the management of the 7xOPS Agent monitoring task, and can start and stop the performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring tasks. When the 7xOPS Server sends the task to the 7xOPS Agent through Mq, the 7xOPS Agent runs the start and stop programs according to the data parsed from the sent task.

Images