Data acquisition method and device based on adaptive algorithm, equipment and medium

Abstract

The invention discloses a data acquisition method and device based on an adaptive algorithm, equipment and a medium. The method comprises the following steps: acquiring system information corresponding to a target operation system, and determining a target probe script corresponding to the target operation system in a preset probe library based on the system information and a preset adaptive algorithm; performing data acquisition on the target operation system based on the target probe script to obtain a basic data acquisition result, and performing data configuration on the basic data acquisition result based on a preset data structure matching algorithm and a preset data template library to obtain a to-be-selected data acquisition result corresponding to the target operation system; and performing format conversion on the to-be-selected data acquisition result based on the docking data type of the target docking device to obtain a target data acquisition result corresponding to the target operation system. By means of the technical scheme, automatic data collection can be carried out on various heterogeneous data sources, redundant configuration of data collection in the early stage is reduced, and the data collection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a data acquisition method, device, equipment and medium based on an adaptive algorithm. Background Art

[0002] With the rapid development of the informatization process, cloud services based on computer systems have become an integral part of modern enterprises. However, to ensure the normal operation of cloud services and improve the reliability and security of each system environment, it is essential to collect the operation data of cloud services so as to perform repair and solution in a timely manner when emergencies occur.

[0003] In the prior art, monitoring software is usually used for data acquisition. However, the monitoring software in the prior art cannot achieve a full-coverage scenario for various types of data sources, that is, it cannot use a single acquisition probe to collect various types of operation data. This may lead to incompatibility, unusable data or incomplete data docking during the data acquisition process, reducing the data acquisition efficiency. Therefore, how to automatically collect data from multiple heterogeneous data sources, reduce the redundant configuration in the early stage of data acquisition, and improve the data acquisition efficiency is an urgent problem to be solved at present. Summary of the Invention

[0004] The present invention provides a data acquisition method, device, equipment and medium based on an adaptive algorithm, which can solve the problems of incompatibility, unusable data or incomplete data docking during the data acquisition process.

[0005] According to one aspect of the present invention, a data acquisition method based on an adaptive algorithm is provided, including:

[0006] Obtain the system information corresponding to the target operating system, and determine the target probe script corresponding to the target operating system in a preset probe library based on the system information and a preset adaptive algorithm;

[0007] Collect data from the target operating system based on the target probe script to obtain a basic data acquisition result, and perform data configuration on the basic data acquisition result based on a preset data structure matching algorithm and a preset data template library to obtain a candidate data acquisition result corresponding to the target operating system;

[0008] Perform format conversion on the candidate data acquisition result based on the docking data type of the target docking device to obtain a target data acquisition result corresponding to the target operating system.

[0009] According to another aspect of the present invention, a data acquisition device based on an adaptive algorithm is provided, including:

[0010] A script determination module, configured to obtain system information corresponding to a target operating system, and determine a target probe script corresponding to the target operating system in a preset probe library based on the system information and a preset adaptive algorithm;

[0011] A basic data collection module, configured to collect data from the target operating system based on the target probe script to obtain a basic data collection result, and perform data configuration on the basic data collection result based on a preset data structure matching algorithm and a preset data template library to obtain an alternative data collection result corresponding to the target operating system;

[0012] A format conversion module, configured to perform format conversion on the alternative data collection result based on the docking data type of a target docking device to obtain a target data collection result corresponding to the target operating system.

[0013] According to another aspect of the present invention, there is provided an electronic device, which includes:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data collection method based on an adaptive algorithm according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, there is provided a computer-readable storage medium, which stores computer instructions for causing a processor to implement the data collection method based on an adaptive algorithm according to any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, there is provided a computer program product, including a computer program, which implements the data collection method based on an adaptive algorithm according to any embodiment of the present invention when executed by a processor.

[0019] In the technical solution of the embodiment of the present invention, the target probe script corresponding to the target operating system is determined in the preset probe library through the system information corresponding to the target operating system and the preset adaptive algorithm. Furthermore, based on the target probe script, data is collected from the target operating system to obtain a basic data collection result, and the basic data collection result is configured based on the preset data structure matching algorithm and the preset data template library to obtain an alternative data collection result corresponding to the target operating system. Finally, based on the docking data type of the target docking device, the format of the alternative data collection result is converted to obtain the target data collection result corresponding to the target operating system. Since the adaptive algorithm is used to automatically match the corresponding probe script, and the algorithms and templates in the probe script are used for data collection and processing, the problems of incompatibility, unusable data, or incomplete data docking in the data collection process are solved, and data can be automatically collected from a variety of heterogeneous data sources, reducing the redundant configuration in the early stage of data collection and improving the efficiency of data collection.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 is a flowchart of a data collection method based on an adaptive algorithm provided in Embodiment 1 of the present invention;

[0023] Figure 2 is a flowchart of a data collection method based on an adaptive algorithm provided in Embodiment 2 of the present invention;

[0024] Figure 3 is a flowchart of an alternative data collection method based on an adaptive algorithm provided in Embodiment 2 of the present invention;

[0025] Figure 4 is a schematic structural diagram of a data collection device based on an adaptive algorithm provided in Embodiment 3 of the present invention;

[0026] Figure 5 is a schematic structural diagram of an electronic device for implementing the data collection method based on an adaptive algorithm of the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0028] It should be noted that the terms "original", "target", etc. in the specification and claims of the present invention and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] In the technical solution of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of national laws and regulations.

[0030] Embodiment 1

[0031] Figure 1 It is a flowchart of a data acquisition method based on an adaptive algorithm provided for Embodiment 1 of the present invention. This embodiment is applicable to the situation of automatically acquiring data from multiple heterogeneous data sources. This method can be executed by a data acquisition device based on an adaptive algorithm. The data acquisition device based on an adaptive algorithm can be implemented in the form of hardware and / or software, and the data acquisition device based on an adaptive algorithm can be configured in an electronic device. As Figure 1 shown, this method includes:

[0032] S110. Obtain the system information corresponding to the target operating system, and determine the target probe script corresponding to the target operating system in a preset probe library based on the system information and a preset adaptive algorithm.

[0033] Among them, cloud services can refer to a service model that provides users with on-demand computing, storage, network, software, and other resources via the Internet based on cloud computing technology. The operating system can refer to a computer system that includes cloud services. That is, a computer system that applies cloud services. The target operating system can refer to the operating system selected for data collection. Usually, users can determine the target operating system according to actual application requirements. System information can refer to the parameter information corresponding to the target operating system. Exemplarily, the system information can include basic hardware information, operating system information, software running environment information, data source information, and the like.

[0034] Among them, the preset adaptive algorithm can refer to an algorithm preset for adaptively matching the identifiers of system information. Usually, the preset adaptive algorithm can be constructed based on the correlation between system information and corresponding identifiers within a historical time period. The probe script can refer to an automated detection tool written in a programming language. Usually, different probe scripts are adapted to different system information, and the monitoring data of the operating system can be collected through the probe script. The preset probe library can refer to a database preset for storing each probe script. Usually, a preset probe library can contain multiple probe scripts. The target probe script can refer to the probe script in the preset probe library that matches the target operating system.

[0035] S120. Collect data from the target operating system based on the target probe script to obtain a basic data collection result, and perform data configuration on the basic data collection result based on a preset data structure matching algorithm and a preset data template library to obtain a candidate data collection result corresponding to the target operating system.

[0036] Among them, the data collection result can refer to the monitoring data obtained after collecting data from the operating system. Exemplarily, the data collection result can be various heterogeneous data such as database data, log data, message queue data, or Application Programming Interface (API) data streams. The basic data collection result can refer to the unprocessed data collection result obtained through preliminary collection.

[0037] Among them, the preset data structure matching algorithm can refer to an algorithm preset for judging the data structure type of the data collection result. Usually, the preset data structure matching algorithm can be constructed based on the correlation between the data collection result and the corresponding data structure type within a historical time period. The preset data template library can refer to a database preset for storing each data template. Data configuration can refer to an operation of adjusting the data collection result. The candidate data collection result can refer to the data collection result obtained after performing data configuration on the basic data collection result.

[0038] S130. Perform format conversion on the to-be-selected data acquisition result according to the docking data type of the target docking device to obtain the target data acquisition result corresponding to the target operating system.

[0039] Among them, the docking device may refer to a third-party application used for visual display of the data acquisition result. Exemplarily, the docking device may be a monitoring platform. The target docking device may refer to the docking device corresponding to the target operating system. Generally, it is necessary for the docking device to send a monitoring data display request to the target operating system. When the target operating system agrees to the monitoring data display request, this docking device can be used as the target docking device. The docking data type may refer to the data types compatible with the docking device. Exemplarily, the docking data type may be JavaScript Object Notation (JSON), Parquet, or Comma-Separated Values (CSV), etc.

[0040] Among them, format conversion may refer to the process of converting the to-be-selected data acquisition result from one data format to another. The target data acquisition result may refer to the data acquisition result obtained after performing format conversion on the to-be-selected data acquisition result.

[0041] In the technical solution of the embodiment of the present invention, the target probe script corresponding to the target operating system is determined in the preset probe library through the system information corresponding to the target operating system and the preset adaptive algorithm. Furthermore, data acquisition is performed on the target operating system based on the target probe script to obtain the basic data acquisition result, and data configuration is performed on the basic data acquisition result based on the preset data structure matching algorithm and the preset data template library to obtain the to-be-selected data acquisition result corresponding to the target operating system. Finally, format conversion is performed on the to-be-selected data acquisition result according to the docking data type of the target docking device to obtain the target data acquisition result corresponding to the target operating system. Since the adaptive algorithm is used to automatically match the corresponding probe script, and the algorithms and templates in the probe script are used for data acquisition and processing, the problems of incompatibility, unusable data, or incomplete data docking in the data acquisition process are solved. It can automatically perform data acquisition on multiple heterogeneous data sources, reduce the redundant configuration in the early stage of data acquisition, and improve the efficiency of data acquisition.

[0042] Embodiment 2

[0043] Figure 2The flowchart of a data acquisition method based on an adaptive algorithm provided in the second embodiment of the present invention. This embodiment is refined based on the above embodiment. Specifically, in this embodiment, the operation of performing data configuration on the basic data acquisition result based on a preset data structure matching algorithm and a preset data template library to obtain a candidate data acquisition result corresponding to the target operating system is refined. Specifically, it may include: judging the data structure type of the basic data acquisition result based on a preset data structure matching algorithm, and determining the basic data structure type corresponding to the basic data acquisition result; judging the consistency of the basic data structure type based on the historical data structure set, generating a consistency judgment result, and determining the target data structure type corresponding to the basic data acquisition result based on the consistency judgment result and the historical data structure set; performing template matching in the preset data template library based on the target data structure type, and determining the data template corresponding to the target data structure type as the target data template corresponding to the basic data acquisition result; performing data configuration on the basic data acquisition result based on the target data template to obtain a candidate data acquisition result corresponding to the target operating system. As Figure 2 shown, the method includes:

[0044] S210. Obtain the system information corresponding to the target operating system.

[0045] S220. Based on a preset adaptive algorithm, perform an identification judgment on the system information to determine the target identification information corresponding to the system information.

[0046] Among them, the identification information may refer to the information used to identify the attributes of the system information. Exemplarily, the identification information may be the operating system name, the running environment, the data source type, etc. The target identification information may refer to the identification information that matches the system information of the target operating system.

[0047] Generally, if the system information of the target operating system contains identification information, the preset adaptive algorithm can directly identify the fields of the identification information as the target identification information. If the system information of the target operating system does not contain identification information, the preset adaptive algorithm can automatically generate the target identification information using the existing system information.

[0048] S230. Based on the target identification information, perform script matching in the preset probe library to determine the probe script corresponding to the target identification information as the target probe script corresponding to the target operating system.

[0049] Specifically, after obtaining the system information corresponding to the target operating system, the preset adaptive algorithm can be used to perform identification and judgment on the system information to determine the target identification information corresponding to the target operating system. Furthermore, the probe script corresponding to the target identification information can be obtained by matching the target identification information in the preset probe library as the target probe script corresponding to the target operating system. Thus, it can be ensured that each operating system can use the matching probe script for data collection, improving the compatibility and execution efficiency of data collection.

[0050] S240. Perform data collection on the target operating system based on the target probe script to obtain a basic data collection result.

[0051] S250. Based on the preset data structure matching algorithm, perform data structure type judgment on the basic data collection result to determine the basic data structure type corresponding to the basic data collection result.

[0052] Among them, a data structure can refer to a rule for describing the logical relationship and physical storage method between data elements. Usually, the data items and logical order of data elements can be defined in a data structure. A data structure type can refer to the data type corresponding to a data structure. Exemplarily, the data structure type can be a superordinate structure type such as a network data type, a database data type, or a behavior data type, or a specific subordinate structure type such as a unicast data type, a broadcast data type, a numerical type, a string type, a binary type, a process transformation behavior type, or a time and frequency behavior type. Usually, the subordinate structure type is a further refinement of the superordinate structure type. A superordinate structure type can include multiple subordinate structure types. However, a data structure only corresponds to one subordinate structure type and the superordinate structure type to which it belongs. The basic data structure type can refer to the judgment result obtained after performing data structure type judgment on the basic data collection result. Usually, the basic data structure type can be a subordinate structure type.

[0053] S260. Based on the historical data structure set, perform consistency judgment on the basic data structure type to generate a consistency judgment result, and based on the consistency judgment result and the historical data structure set, determine the target data structure type corresponding to the basic data collection result.

[0054] Among them, historical data structure can refer to the standard data structure pre-stored in the historical time period. Exemplarily, the existing standard data structure rules can be used as the historical data structure. Historical data structure type can refer to the data structure type corresponding to the historical data structure. Historical data structure set can refer to the set composed of each historical data structure. Usually, each historical data structure in the historical data structure set exists in pairs with the corresponding historical data structure type.

[0055] Among them, the consistency judgment result can refer to the judgment result obtained by performing a matching search on the basic data structure type using the historical data structure set. Exemplarily, the consistency judgment result can be that the historical data structure set contains the basic data structure type, or that the historical data structure set does not contain the basic data structure type. The target data structure type can refer to the final data structure type corresponding to the basic data collection result.

[0056] In an alternative embodiment, determining the target data structure type corresponding to the basic data collection result based on the consistency judgment result and the historical data structure set includes: if the consistency judgment result is that the historical data structure set contains the basic data structure type, then using the basic data structure type as the target data structure type corresponding to the basic data collection result; if the consistency judgment result is that the historical data structure set does not contain the basic data structure type, then determining the original data structure type corresponding to the basic data structure type in the historical data structure set, and generating the target data structure type corresponding to the basic data collection result based on the original data structure type. Among them, the original data structure type can refer to the upper-level structure type corresponding to the basic data structure type. Specifically, after performing a consistency judgment on the basic data structure type using the historical data structure set and generating a consistency judgment result. If the consistency judgment result is that the historical data structure set contains the basic data structure type, then the basic data structure type can be used as the target data structure type for subsequent operations. Conversely, if the consistency judgment result is that the historical data structure set does not contain the basic data structure type, then the original data structure type corresponding to the basic data structure type can be determined in the historical data structure set, and the original data structure type can be used to process the basic data collection result to generate the corresponding target data structure type. Thereby, an effective basis can be provided for subsequent data processing and the efficiency of data collection can be improved.

[0057] In an optional embodiment, generating the target data structure type corresponding to the basic data collection result based on the original data structure type includes: performing field mapping on the basic data collection result based on the original data structure type to generate a mapped data collection result corresponding to the basic data collection result; determining the data structure type of the mapped data collection result based on a preset data structure matching algorithm to determine the mapped data structure type corresponding to the mapped data collection result; performing a consistency judgment on the mapped data structure type based on a historical data structure set. If the consistency judgment result is that the historical data structure set contains the mapped data structure type, then use the mapped data structure type as the target data structure type corresponding to the basic data collection result. Among them, field mapping can refer to an operation of adjusting the field content and field order in the data collection result. The mapped data collection result can refer to the data result obtained by performing field mapping on the basic data collection result using the original data structure type. Exemplarily, taking the original data structure corresponding to the original data structure type as (a1, a2, a3) and the basic data collection result as (a11, a31, a41) as an example, where a1, a2, and a3 can represent field names in the data structure, a11 can represent the specific field content under the a1 field name, a31 can represent the specific field content under the a3 field name, and a41 can represent the specific field content under the a4 field name. Then the mapped data collection result can be (a11,, a31). The mapped data structure type can refer to the data structure type corresponding to the mapped data collection result. Exemplarily, the mapped data structure type can be a subordinate structure type.

[0058] Specifically, after determining the original data structure type corresponding to the basic data structure type, the original data structure corresponding to the original data structure type can be used to perform field mapping on the basic data collection result to generate a mapped data collection result. Furthermore, use a preset data structure matching algorithm to determine the data structure type of the mapped data collection result and determine the mapped data structure type corresponding to the mapped data collection result. Finally, use the historical data structure set to perform a consistency judgment on the mapped data structure type. If the historical data structure set contains the mapped data structure type, then the mapped data structure type can be used as the target data structure type corresponding to the basic data collection result for subsequent operations. Exemplarily, taking the original data structure type as A, the subordinate structure types included as A1, A2, and A3, and the basic data structure type as A4 as an example. If the mapped data structure type after field mapping is A2, then A2 can be used as the target data structure type corresponding to the basic data collection result. Thus, a data structure type can be generated for the mismatched data collection result, providing an effective basis for subsequent data processing.

[0059] S270. Perform template matching in a preset data template library based on the target data structure type to determine the data template corresponding to the target data structure type, and use it as the target data template corresponding to the basic data collection result.

[0060] Among them, the target data template can refer to the data template in the preset data template library that matches the basic data collection result.

[0061] S280. Perform data configuration on the basic data collection result based on the target data template to obtain the candidate data collection result corresponding to the target operating system.

[0062] In an alternative embodiment, the performing data configuration on the basic data collection result based on the target data template to obtain the candidate data collection result corresponding to the target operating system includes: obtaining custom editing information; performing data filling on the basic data collection result based on the custom editing information and the target data template to obtain the candidate data collection result corresponding to the target operating system. Among them, the custom editing information can refer to the editing information established by the user according to actual operation requirements. Exemplarily, the custom editing information can be operation information for modifying the field content or moving the field position in the target data template. Specifically, after determining the target data template corresponding to the basic data collection result, custom editing information can be obtained, and the target data template can be adjusted according to the custom editing information. Furthermore, the basic data collection result is filled into the adjusted data template to obtain the candidate data collection result. Thus, the collection template can be dynamically optimized according to the feedback of user configuration, improving data compatibility and adaptability.

[0063] S290. Perform template matching in a preset format template library based on the docking data type of the target docking device to determine the format template corresponding to the docking data type.

[0064] Among them, the format template can refer to the conversion template that matches the docking data type. Usually, through the format template, data types in other formats can be converted into the docking data type. The preset format template library can refer to a database preset for storing format templates. Usually, one docking data type corresponds to one format template, and the preset format template library can contain format templates for multiple docking data types.

[0065] S2100. Perform format conversion on the candidate data collection result based on the format template to obtain the target data collection result corresponding to the target operating system.

[0066] Specifically, after obtaining the candidate data acquisition result corresponding to the target operating system, the docking data type of the target docking device can be used to perform template matching in the preset format template library to determine the format template corresponding to the docking data type. Furthermore, the candidate data acquisition result is subjected to format conversion according to the format template to obtain the target data acquisition result corresponding to the target operating system. Thus, an effective basis can be provided for the visual display of the data acquisition result.

[0067] In the technical solution of the embodiment of the present invention, the system information corresponding to the target operating system is subjected to identification judgment through a preset adaptive algorithm to determine the target identification information corresponding to the system information. Furthermore, script matching is performed in the preset probe library based on the target identification information to determine the probe script corresponding to the target identification information as the target probe script corresponding to the target operating system, and data acquisition is performed on the target operating system based on the target probe script to obtain the basic data acquisition result. Further, the data structure type of the basic data acquisition result is determined through a preset data structure matching algorithm, the consistency judgment of the basic data structure type is performed based on the historical data structure set to generate a consistency judgment result, and the target data structure type corresponding to the basic data acquisition result is determined based on the consistency judgment result and the historical data structure set. Further, template matching is performed in the preset data template library based on the target data structure type to determine the data template corresponding to the target data structure type as the target data template corresponding to the basic data acquisition result, and data configuration is performed on the basic data acquisition result based on the target data template to obtain the candidate data acquisition result corresponding to the target operating system. Finally, template matching is performed in the preset format template library based on the docking data type of the target docking device to determine the format template corresponding to the docking data type, and the candidate data acquisition result is subjected to format conversion based on the format template to obtain the target data acquisition result corresponding to the target operating system. Since the corresponding probe script is automatically matched by using the adaptive algorithm, and the algorithms and templates in the probe script are used for data acquisition and processing, the problems of incompatibility, unusable data, or incomplete data docking in the data acquisition process are solved, the data of multiple heterogeneous data sources can be automatically acquired, the redundant configuration in the early stage of data acquisition is reduced, and the efficiency of data acquisition is improved.

[0068] Figure 3The figure shows a flowchart of an optional data acquisition method based on an adaptive algorithm provided by an embodiment of the present invention. Specifically, after a user determines a target operating system, a script program containing a preset probe library can be downloaded in the target operating system. Then, the preset adaptive algorithm in the script program is used to perform an identification judgment on the system information corresponding to the target operating system to determine the target identification information corresponding to the system information. Furthermore, the target identification information is used to perform script matching in the preset probe library to determine the probe script corresponding to the target identification information as the target probe script corresponding to the target operating system, thereby realizing dynamic script matching. Further, the target probe script is used to perform data acquisition on the target operating system to obtain a basic data acquisition result. It should be noted that when the target probe script performs data acquisition, operations such as real-time filtering, preprocessing, and format conversion can also be performed to reduce data latency and improve processing efficiency. Further, based on a preset data structure matching algorithm, a data structure type judgment is performed on the basic data acquisition result to determine the basic data structure type corresponding to the basic data acquisition result, and a consistency judgment is performed on the basic data structure type based on a historical data structure set to generate a consistency judgment result. If the consistency judgment result is that the historical data structure set contains the basic data structure type, the basic data structure type is used as the target data structure type corresponding to the basic data acquisition result. If the consistency judgment result is that the historical data structure set does not contain the basic data structure type, the original data structure type corresponding to the basic data structure type is determined in the historical data structure set, and field mapping is performed on the basic data acquisition result based on the original data structure type to generate a mapped data acquisition result corresponding to the basic data acquisition result. A data structure type judgment is performed on the mapped data acquisition result based on the preset data structure matching algorithm to determine the mapped data structure type corresponding to the mapped data acquisition result, and a consistency judgment is performed on the mapped data structure type based on the historical data structure set. If the consistency judgment result is that the historical data structure set contains the mapped data structure type, the mapped data structure type is used as the target data structure type corresponding to the basic data acquisition result. Further, based on the target data structure type, template matching is performed in a preset data template library to determine the data template corresponding to the target data structure type as the target data template corresponding to the basic data acquisition result, and data configuration is performed on the basic data acquisition result based on custom editing information and the target data template to obtain a candidate data acquisition result corresponding to the target operating system. Finally, based on the docking data type of the target docking device, template matching is performed in a preset format template library to determine the format template corresponding to the docking data type, and format conversion is performed on the candidate data acquisition result based on the format template to obtain the target data acquisition result corresponding to the target operating system. Thus, through a dynamic probe management mechanism, the optimal probe script can be adaptively selected according to different target environments without manual adaptation by the user.Through the built-in data structure matching algorithm and data template library, the optimal data template can be automatically matched based on data types, structural features, and quality assessment. By dynamically optimizing the data template, data compatibility and adaptability can be improved. The general probe architecture (i.e., the preset probe library) can be compatible with a variety of heterogeneous data sources, realizing unified collection, conversion, and storage of data in different fields.

[0069] It should be noted that, in the embodiments of the present invention, each database and algorithm can be integrated into a software program. Thus, when the software program is included in the running system, automatic probe script adaptation can be achieved without manual adaptation by the user. In addition, the preset data template library described in the embodiments of the present invention has a variety of standardized data structure templates, covering common database structures, monitoring metrics, or log formats, etc. It supports users to customize templates, and new data types can be flexibly defined through a visual interface or configuration file, and automatic recommendation and optimization of templates are supported to improve adaptability.

[0070] Embodiment Three

[0071] Figure 4 It is a schematic structural diagram of a data acquisition device based on an adaptive algorithm provided in Embodiment Three of the present invention. As Figure 4 shown, the device includes: a script determination module 310, a basic acquisition module 320, and a format conversion module 330;

[0072] Among them, the script determination module 310 is used to obtain the system information corresponding to the target running system, and determine the target probe script corresponding to the target running system in the preset probe library based on the system information and the preset adaptive algorithm;

[0073] The basic acquisition module 320 is used to perform data acquisition on the target running system based on the target probe script to obtain a basic data acquisition result, and perform data configuration on the basic data acquisition result based on the preset data structure matching algorithm and the preset data template library to obtain a candidate data acquisition result corresponding to the target running system;

[0074] The format conversion module 330 is used to perform format conversion on the candidate data acquisition result based on the docking data type of the target docking device to obtain the target data acquisition result corresponding to the target running system.

[0075] In the technical solution of the embodiment of the present invention, the target probe script corresponding to the target operating system is determined in the preset probe library through the system information corresponding to the target operating system and the preset adaptive algorithm. Furthermore, based on the target probe script, data collection is performed on the target operating system to obtain a basic data collection result, and based on the preset data structure matching algorithm and the preset data template library, data configuration is performed on the basic data collection result to obtain a candidate data collection result corresponding to the target operating system. Finally, based on the docking data type of the target docking device, format conversion is performed on the candidate data collection result to obtain a target data collection result corresponding to the target operating system. Since the adaptive algorithm is used to automatically match the corresponding probe script, and the algorithms and templates in the probe script are used for data collection and processing, the problems of incompatibility, unusable data, or incomplete data docking in the data collection process are solved, and data collection can be automatically performed on a variety of heterogeneous data sources, reducing the redundant configuration in the early stage of data collection and improving the efficiency of data collection.

[0076] Optionally, the script determination module 310 may specifically be used for:

[0077] Based on the preset adaptive algorithm, perform identification judgment on the system information to determine the target identification information corresponding to the system information;

[0078] Based on the target identification information, perform script matching in the preset probe library to determine the probe script corresponding to the target identification information as the target probe script corresponding to the target operating system.

[0079] Optionally, the basic collection module 320 may specifically be used for:

[0080] Based on the preset data structure matching algorithm, perform data structure type judgment on the basic data collection result to determine the basic data structure type corresponding to the basic data collection result;

[0081] Based on the historical data structure set, perform consistency judgment on the basic data structure type to generate a consistency judgment result, and based on the consistency judgment result and the historical data structure set, determine the target data structure type corresponding to the basic data collection result;

[0082] Based on the target data structure type, perform template matching in the preset data template library to determine the data template corresponding to the target data structure type as the target data template corresponding to the basic data collection result;

[0083] Based on the target data template, perform data configuration on the basic data collection result to obtain a candidate data collection result corresponding to the target operating system.

[0084] Optionally, the basic collection module 320 may specifically be used for:

[0085] If the consistency judgment result is that the historical data structure set contains the basic data structure type, then use the basic data structure type as the target data structure type corresponding to the basic data collection result;

[0086] If the consistency judgment result is that the historical data structure set does not contain the basic data structure type, then determine the original data structure type corresponding to the basic data structure type in the historical data structure set, and generate the target data structure type corresponding to the basic data collection result based on the original data structure type.

[0087] Optionally, the basic collection module �20 can be specifically used for:

[0088] Perform field mapping on the basic data collection result based on the original data structure type to generate the mapped data collection result corresponding to the basic data collection result;

[0089] Perform data structure type judgment on the mapped data collection result based on a preset data structure matching algorithm to determine the mapped data structure type corresponding to the mapped data collection result;

[0090] Perform consistency judgment on the mapped data structure type based on the historical data structure set. If the consistency judgment result is that the historical data structure set contains the mapped data structure type, then use the mapped data structure type as the target data structure type corresponding to the basic data collection result.

[0091] Optionally, the basic collection module �20 can be specifically used for:

[0092] Obtain custom editing information;

[0093] Perform data filling on the basic data collection result based on the custom editing information and the target data template to obtain the candidate data collection result corresponding to the target operating system.

[0094] Optionally, the format conversion module 330 can be specifically used for:

[0095] Perform template matching in a preset format template library based on the docking data type of the target docking device to determine the format template corresponding to the docking data type;

[0096] Perform format conversion on the candidate data collection result based on the format template to obtain the target data collection result corresponding to the target operating system.

[0097] The data acquisition device based on the adaptive algorithm provided by the embodiments of the present invention can execute the data acquisition method based on the adaptive algorithm provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0098] Embodiment 4

[0099] Figure 5 FIG. shows a schematic structural diagram of an electronic device 410 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0100] As Figure 5 shown, the electronic device 410 includes at least one processor 420, and a memory communicatively connected to the at least one processor 420, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 into the random access memory (RAM) 440. In the RAM 440, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 420, the ROM 430, and the RAM 440 are connected to each other through a bus 450. The input / output (I / O) interface 460 is also connected to the bus 450.

[0101] Multiple components in the electronic device 410 are connected to the I / O interface 460, including: an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a magnetic disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0102] The processor 420 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 420 executes the various methods and processes described above, such as the data acquisition method based on the adaptive algorithm.

[0103] The method includes:

[0104] Obtaining the system information corresponding to the target operating system, and determining the target probe script corresponding to the target operating system in a preset probe library based on the system information and a preset adaptive algorithm;

[0105] Performing data acquisition on the target operating system based on the target probe script to obtain a basic data acquisition result, and performing data configuration on the basic data acquisition result based on a preset data structure matching algorithm and a preset data template library to obtain an alternative data acquisition result corresponding to the target operating system;

[0106] Performing format conversion on the alternative data acquisition result based on the docking data type of the target docking device to obtain a target data acquisition result corresponding to the target operating system.

[0107] In some embodiments, the data acquisition method based on the adaptive algorithm can be implemented as a computer program, which is tangibly included in a computer-readable storage medium, such as the storage unit 490. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the data acquisition method based on the adaptive algorithm described above can be executed. Alternatively, in other embodiments, the processor 420 can be configured to execute the data acquisition method based on the adaptive algorithm in any other suitable manner (e.g., by means of firmware).

[0108] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0109] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0110] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0112] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0113] A computing system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0114] An embodiment of the present application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the data acquisition method based on an adaptive algorithm provided in any embodiment of the present application. This program product and the data acquisition method based on an adaptive algorithm disclosed in each embodiment of the present application belong to the same inventive concept, and thus will not be elaborated herein.

[0115] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0116] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data acquisition method based on an adaptive algorithm, characterized in that, Including: Obtain the system information corresponding to the target operating system, and determine the target probe script corresponding to the target operating system in a preset probe library based on the system information and a preset adaptive algorithm; Collect data from the target operating system based on the target probe script to obtain a basic data collection result, and perform data configuration on the basic data collection result based on a preset data structure matching algorithm and a preset data template library to obtain a candidate data collection result corresponding to the target operating system; Perform format conversion on the candidate data collection result based on the docking data type of the target docking device to obtain a target data collection result corresponding to the target operating system.

2. The method according to claim 1, characterized in that, The step of determining the target probe script corresponding to the target operating system in the preset probe library based on the system information and the preset adaptive algorithm includes: Based on the preset adaptive algorithm, perform identification judgment on the system information to determine the target identification information corresponding to the system information; Perform script matching in the preset probe library based on the target identification information to determine the probe script corresponding to the target identification information as the target probe script corresponding to the target operating system.

3. The method according to claim 1, characterized in that, The step of performing data configuration on the basic data collection result based on the preset data structure matching algorithm and the preset data template library to obtain a candidate data collection result corresponding to the target operating system includes: Based on the preset data structure matching algorithm, perform data structure type judgment on the basic data collection result to determine the basic data structure type corresponding to the basic data collection result; Perform consistency judgment on the basic data structure type based on the historical data structure set to generate a consistency judgment result, and determine the target data structure type corresponding to the basic data collection result based on the consistency judgment result and the historical data structure set; Perform template matching in the preset data template library based on the target data structure type to determine the data template corresponding to the target data structure type as the target data template corresponding to the basic data collection result; Perform data configuration on the basic data collection result based on the target data template to obtain a candidate data collection result corresponding to the target operating system.

4. The method according to claim 3, wherein The step of determining the target data structure type corresponding to the basic data collection result based on the consistency judgment result and the historical data structure set includes: If the consistency judgment result is that the historical data structure set contains the basic data structure type, then use the basic data structure type as the target data structure type corresponding to the basic data collection result; If the consistency judgment result is that the historical data structure set does not contain the basic data structure type, then determine the original data structure type corresponding to the basic data structure type in the historical data structure set, and generate the target data structure type corresponding to the basic data collection result based on the original data structure type.

5. The method according to claim 4, wherein The step of generating the target data structure type corresponding to the basic data collection result based on the original data structure type includes: Perform field mapping on the basic data collection result based on the original data structure type to generate a mapped data collection result corresponding to the basic data collection result; Perform data structure type judgment on the mapped data collection result based on a preset data structure matching algorithm to determine the mapped data structure type corresponding to the mapped data collection result; Perform consistency judgment on the mapped data structure type based on the historical data structure set. If the consistency judgment result is that the mapped data structure type is included in the historical data structure set, then use the mapped data structure type as the target data structure type corresponding to the basic data collection result.

6. The method according to claim 3, characterized in that The data configuration of the basic data collection result based on the target data template to obtain a candidate data collection result corresponding to the target operating system includes: Obtain custom editing information; Perform data filling on the basic data collection result based on the custom editing information and the target data template to obtain a candidate data collection result corresponding to the target operating system.

7. The method according to claim 1, wherein The format conversion of the candidate data collection result based on the docking data type of the target docking device to obtain a target data collection result corresponding to the target operating system includes: Perform template matching in a preset format template library based on the docking data type of the target docking device to determine the format template corresponding to the docking data type; Perform format conversion on the candidate data collection result based on the format template to obtain a target data collection result corresponding to the target operating system.

8. A data acquisition device based on an adaptive algorithm, characterized in that, Includes: A script determination module, configured to obtain system information corresponding to the target operating system, and determine a target probe script corresponding to the target operating system in a preset probe library based on the system information and a preset adaptive algorithm; A basic collection module, configured to perform data collection on the target operating system based on the target probe script to obtain a basic data collection result, and perform data configuration on the basic data collection result based on a preset data structure matching algorithm and a preset data template library to obtain a candidate data collection result corresponding to the target operating system; A format conversion module, configured to perform format conversion on the candidate data collection result based on the docking data type of the target docking device to obtain a target data collection result corresponding to the target operating system.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the adaptive algorithm-based data collection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to implement the adaptive algorithm-based data collection method according to any one of claims 1-7 when executed.

Images