An ETL system and method for operation and maintenance data of multiple network management platforms

Abstract

The invention provides an ETL system and method for operation and maintenance data of multiple network management platforms. The system comprises: a security authentication component for obtaining an authentication token sent by a target network management platform; an interface calling component for configuring interface parameters of the target network management platform; a data conversion component for obtaining a core phrase of data transmitted by the target network management platform, determining whether the current core phrase has historical data, and if the current core phrase is not historical data, calculating a matching value between the core phrase and each candidate phrase in a preset recommendation mapping table, and taking the candidate phrase with the highest matching value as the mapping phrase of the core phrase; and a data persistence component for storing the transmitted data in a corresponding position according to the storage position of the current core phrase by the historical data if the data conversion component determines that the current core phrase is historical data, and storing the transmitted data in a storage position of the mapping phrase corresponding to the core phrase if the data conversion component determines that the current core phrase is not historical data.

Description

Technical Field

[0001] The present invention relates to the field of data operation and maintenance technology, and in particular to an ETL system and method applied to operation and maintenance data of multiple network management platforms. Background Art

[0002] At present, technologies such as cloud computing, big data, and artificial intelligence are developing rapidly, and information systems have become an important part of the core competitiveness of enterprises. Therefore, IT operation and maintenance, as a guarantee for the stable, secure, and efficient operation of information systems, has received more and more attention. With the gradual maturity of new IT technology practices such as microservices and containers, and the rapid changes in business needs, the operation and maintenance technology system is also constantly exploring new models and new practices. From the initial pure manual operation and maintenance to the use of scripts and open source tools, and then to platform construction, operation and maintenance has also kept pace with the times and gradually developed in the direction of intelligence. By applying some machine learning algorithms and models, the understanding of data by operation and maintenance will be greatly simplified, and efficiency will be greatly improved.

[0003] At the same time, with the increasing maturity of technologies such as cloud computing and big data, all industries are facing tremendous pressure for digital transformation. IT has moved from the backend to the foreground, and application scenarios are becoming more diverse and complex, requiring IT operations to be more agile and efficient. In the context of big data, industry applications usually need to use multi-cloud, multi-service provider equipment, and multi-platform collaboration. However, the monitoring and operation and maintenance platform provided by the equipment provider can only manage its own equipment. In such a scenario, the unified access of operation and maintenance data of multiple network management platforms, data extraction, transformation, and loading (Extract-Transform-Load, ETL) have become urgent problems to be solved. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an ETL system applied to operation and maintenance data of multiple network management platforms to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides an ETL system for operation and maintenance data of multiple network management platforms, the system comprising:

[0006] A security authentication component is used to send an authentication token request to a target network management platform and obtain an authentication token sent by the target network management platform;

[0007] The interface calling component is used to determine whether the current authentication token is available. If it is available, the interface parameters of the target network management platform are configured, and a data connection with the target network management platform is established based on the interface parameters to obtain data;

[0008] The data conversion component obtains the core phrase of the data transmitted by the target network management platform, determines whether there is historical data for the current core phrase, and if the current core phrase is not historical data, calculates the matching value between the core phrase and each candidate phrase in the preset recommendation mapping table, and uses the candidate phrase with the highest matching value as the mapping phrase of the core phrase;

[0009] Data persistence component, if the data conversion component determines that the current core phrase is historical data, then according to the storage position of the current core phrase in the historical data, the incoming data is stored in the corresponding position; if the data conversion component determines that the current core phrase is not historical data, then the incoming data is stored in the storage position of the mapping phrase corresponding to the core phrase.

[0010] By adopting the above scheme, there can be multiple target network management platforms in this scheme. Since the interface calling methods, security authentication processes and interface data return forms of each network management platform are different, in this scheme, users can build an ETL process through component configuration on the interface. This scheme can simultaneously meet the ETL processing of operation and maintenance data of multiple network management platforms. The ETL process is constructed in a componentized form. Compared with traditional customized program design, the flexibility and reusability are greatly improved, and it is easy to find and locate problems, and the maintainability is stronger.

[0011] In some embodiments of the present invention, the core phrase includes at least one core word, and the candidate phrase includes at least one candidate word. In the step of calculating the matching value between the core phrase and each candidate phrase in a preset recommendation mapping table in the data conversion component, the matching value between each core word in each core phrase and each candidate word in the candidate phrase is calculated respectively to obtain the matching value between the core phrase and each candidate phrase.

[0012] In some embodiments of the present invention, each candidate word in the candidate word group is preset with a score parameter. If the core word is the same as the candidate word, the score parameter of the candidate word is taken into account, and the matching score of each core word in the core word group and each candidate word in the same candidate word group is added to obtain the matching value.

[0013] In some embodiments of the present invention, the candidate words in the recommendation mapping table preset in the data conversion component are divided into first-level words, second-level words and third-level words, and the first-level words, second-level words and third-level words are respectively set with different score parameters, and the score parameter of the first-level word is greater than the score parameter of the second-level word> the score parameter of the third-level word.

[0014] In some embodiments of the present invention, the step of sending an authentication token request to the target network management platform by the security authentication component and obtaining the authentication token sent by the target network management platform further includes:

[0015] The duration of the authentication token is obtained, and a request interval is set according to the duration of the authentication token. The security authentication component sends an authentication token request to the target network management platform every request interval.

[0016] In some embodiments of the present invention, in the step of using the candidate phrase with the highest matching value as the mapping phrase of the core phrase, the mapping phrase and the core word corresponding to the core phrase are sent to the operation and maintenance end to determine whether the operation and maintenance end issues a change instruction for the mapping phrase. If the operation and maintenance end issues a change instruction, the mapping phrase corresponding to the core phrase is changed according to the change instruction.

[0017] In some embodiments of the present invention, the processing step of the data conversion component further includes:

[0018] Obtain the source resource of the incoming data of the target network management platform, match the source resource with the resource to be matched in the preset resource id mapping table, if the match is successful, assign the resource id of the matched resource to be matched to the source resource, if the match fails, generate a unique resource id for the source resource and record it in the resource id mapping table.

[0019] In some embodiments of the present invention, the step of storing the incoming data in the storage location of the mapping phrase corresponding to the core phrase by the data persistence component further includes adding the processed core phrase to the historical data and synchronously updating the historical data of the data conversion component.

[0020] In some embodiments of the present invention, the step of the data persistence component further includes:

[0021] Convert the data units of the incoming data to a uniform data unit.

[0022] Another aspect of the present invention provides an ETL method for multi-network management platform operation and maintenance data using the above system, the method comprising the following steps:

[0023] Security authentication: send an authentication token request to the target network management platform and obtain the authentication token sent by the target network management platform;

[0024] Call the interface to determine whether the current authentication token is available. If available, configure the interface parameters of the target network management platform, establish a data connection with the target network management platform based on the interface parameters, and obtain data;

[0025] Data conversion, obtaining the core phrase of the data transmitted by the target network management platform, determining whether there is historical data for the current core phrase, and if the current core phrase is not historical data, calculating the matching value between the core phrase and each candidate phrase in the preset recommendation mapping table, and taking the candidate phrase with the highest matching value as the mapping phrase of the core phrase;

[0026] Data persistence: if the data conversion component determines that the current core phrase is historical data, the incoming data will be stored in the corresponding position according to the storage position of the current core phrase in the historical data; if the data conversion component determines that the current core phrase is not historical data, the incoming data will be stored in the storage position of the mapping phrase corresponding to the core phrase.

[0027] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention can be specifically pointed out and obtained in the specification and the accompanying drawings.

[0028] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0030] Figure 1 It is a schematic diagram of a first implementation mode of the ETL system applied to operation and maintenance data of multiple network management platforms of the present invention;

[0031] Figure 2 It is a schematic diagram of a second implementation mode of the ETL system applied to operation and maintenance data of multiple network management platforms of the present invention;

[0032] Figure 3 It is a schematic diagram of a third implementation mode of the ETL system applied to operation and maintenance data of multiple network management platforms of the present invention;

[0033] Figure 4 A schematic diagram of an implementation of a data conversion component;

[0034] Figure 5 A schematic diagram of another implementation of a data conversion component. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0036] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0037] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0038] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0039] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0040] To solve the above problems, Figure 1 , 2 As shown, the present invention proposes an ETL system applied to operation and maintenance data of multiple network management platforms, the system comprising:

[0041] S100, a security authentication component, used to send an authentication token request to a target network management platform and obtain an authentication token sent by the target network management platform;

[0042] In some embodiments of the present invention, the security authentication component is mainly responsible for obtaining the authentication token of the third-party network management platform and keeping the token alive, that is, requesting the authentication token and keeping the authentication token alive. The authentication token is kept alive by setting a request interval duration according to the duration of the authentication token. The security authentication component sends an authentication token request to the target network management platform every request interval duration, and the interval duration is less than the duration of the authentication token.

[0043] The input content of the authentication token request includes the URL address, parameter list and keep-alive period (request interval duration);

[0044] By adopting the above solution, since the parameters and keep-alive periods required by each network management platform during the security authentication process are different, a separate security authentication component can be configured for each network management platform. This solution provides an interface-based configuration method, and users can customize the authentication parameter list, URL address, keep-alive period and other information.

[0045] The parameters in the parameter list may be passwords, etc.

[0046] S200, an interface calling component is used to determine whether the current authentication token is available. If it is available, the interface parameters of the target network management platform are configured, and a data connection with the target network management platform is established based on the interface parameters to obtain data;

[0047] In some embodiments of the present invention, if the current authentication token is in a keep-alive period, it is available; if it is not available, the security authentication component sends an authentication token request to the target network management platform to obtain the authentication token sent by the target network management platform.

[0048] In some embodiments of the present invention, the interface calling component is mainly responsible for calling the third-party platform interface through http requests, and passing the acquired operation and maintenance data to downstream components in the form of a message queue. Users can configure the interface call cycle, URL address, interface required parameters and other information through an interface.

[0049] like Figure 4 , 5 As shown, S300, a data conversion component obtains a core phrase of data transmitted from a target network management platform, determines whether there is historical data for the current core phrase, and if the current core phrase is not historical data, calculates a matching value between the core phrase and each candidate phrase in a preset recommendation mapping table, and uses the candidate phrase with the highest matching value as a mapping phrase for the core phrase;

[0050] In some embodiments of the present invention, the core phrase of the transmitted data can be a column name of column data or a table name of table data, etc. If there is historical data for the current core phrase, it means that the current core phrase has been processed by the system. If it is not historical data, the core phrase mapping method is used to calculate the matching value of the core phrase and each candidate phrase in the preset recommendation mapping table, and the candidate phrase with the highest matching value is used as the mapping phrase of the core phrase.

[0051] In some embodiments of the present invention, since the naming formats of operation and maintenance indicators of multiple network management platforms are not exactly the same, the system of the present invention maintains two field mapping tables, namely, a matched field mapping table (historical data) and a recommended matching field mapping table (preset recommended mapping table), which are used for mapping source fields to destination fields. The destination field refers to a unique indicator field maintained by the system of the present invention. The matched field mapping table stores the mapping relationship between the determined source field and the destination field. The recommended matching field mapping table stores the destination field and its core word sequence, which is mainly used for mapping relationship recommendation.

[0052] In some embodiments of the present invention, the data conversion template component of the present invention first completes the parsing of the interface response body. The component will parse and extract data from the response body according to the structure style configured by the user. The configurable parsing methods include but are not limited to: 1. List <map>; 2. Map; 3. List, the parsed data format is uniformly processed into List <map>Format.

[0053] Using the above solution, since the interface styles of multiple network management platforms are different and the interface response body formats are diverse, the data conversion component of this solution will unify the data format into List <map>Format.

[0054] S400, data persistence component, if the data conversion component determines that the current core phrase is historical data, then according to the storage position of the current core phrase of the historical data, the incoming data is stored in the corresponding position; if the data conversion component determines that the current core phrase is not historical data, then the incoming data is stored in the storage position of the mapping phrase corresponding to the core phrase.

[0055] In some implementations of the present invention, the data persistence component is mainly responsible for receiving the operation and maintenance indicator data output by the upstream data conversion component and persisting it in the database. The data persistence component needs to be configured with the database address and database table.

[0056] By adopting the above scheme, there can be multiple target network management platforms in this scheme. Since the interface calling methods, security authentication processes and interface data return forms of each network management platform are different, in this scheme, users can build an ETL process through component configuration on the interface. This scheme can simultaneously meet the ETL processing of operation and maintenance data of multiple network management platforms. The ETL process is constructed in a componentized form. Compared with traditional customized program design, the flexibility and reusability are greatly improved, and it is easy to find and locate problems, and the maintainability is stronger.

[0057] In some embodiments of the present invention, the core phrase includes at least one core word, and the candidate phrase includes at least one candidate word. In the step of calculating the matching value between the core phrase and each candidate phrase in a preset recommendation mapping table in the data conversion component, the matching value between each core word in each core phrase and each candidate word in the candidate phrase is calculated respectively to obtain the matching value between the core phrase and each candidate phrase.

[0058] In some embodiments of the present invention, each candidate word in the candidate word group is preset with a score parameter. If the core word is the same as the candidate word, the score parameter of the candidate word is taken into account, and the matching score of each core word in the core word group and each candidate word in the same candidate word group is added to obtain the matching value.

[0059] In some embodiments of the present invention, the candidate words in the recommendation mapping table preset in the data conversion component are divided into first-level words, second-level words and third-level words, and the first-level words, second-level words and third-level words are respectively set with different score parameters, and the score parameter of the first-level word is greater than the score parameter of the second-level word> the score parameter of the third-level word.

[0060] In some embodiments of the present invention, the step of mapping the core phrase to obtain the mapped phrase by the data conversion component of the present solution includes a primary mapping and a secondary mapping.

[0061] In the first mapping step, the matched field mapping table (historical data) is queried. If there is a mapping from the source field to the target field, it is returned. Otherwise, a matching recommendation process is performed, that is, the matching value of the core phrase and each candidate phrase in the preset recommendation mapping table is calculated, and the candidate phrase with the highest matching value is used as the mapping phrase of the core phrase;

[0062] In the recommended matching field mapping table, for each target field, set a core word and configure its importance level; specifically, the importance of the core word can be divided into three levels: 5 points for level one, 3 points for level two, and 2 points for level three. The matching value is obtained by matching the core word with the candidate word in the recommended matching field mapping table and scoring, and the target field corresponding to the highest score is returned as the recommended field.

[0063] The data of the matching field mapping table and the recommendation mapping table of the above two database forms are cached in redis, and the data will be queried in sequence according to the core word level during the matching process. When building the recommendation matching field mapping table, the words with the same first-level core word will be put together.

[0064] Through investigation, it is found that the indicator fields in the interface response bodies of various network management platforms are mostly composed of resource categories + indicator names. For example, CPU utilization can be described as: cpu_usage, cpuUsage; interface packet loss rate can be described as interface_discard_percentage, interface_packetloss_percentage. If the indicator fields are cpu_usage and cpuUsage, the core phrase is a core phrase consisting of cpu and usage. cpu can be the first level and usage can be the second level. If there is a candidate phrase including cpu and usage, the matching value is 8 points. If there is a candidate phrase including gpu and usage, gpu fails to match and is not included in the score. For the candidate phrase including gpu and usage, the matching value is 3 points. The source field interface_discard_percentage can set the core phrase to ["interface": first level,"discard": second level,"packetloss": second level,"percentage": third level].

[0065] Figure 4 The matched field mapping table in the table is the historical data; Figure 4 The matched destination field is returned in the above method, that is, the matching result of the core phrase in the historical data is used as the matching result, and the input data is stored in the corresponding position of the matching result in the historical data; Figure 4 The keyword splitting of source fields is to split the source fields into multiple core words; Figure 4 The keywords are used to match and score the core words of each target field, that is, the core words are matched and scored with the words to be selected; Figure 4 The destination field corresponding to the highest score is the mapping phrase.

[0066] Figure 5 The recommended purpose field is the mapping phrase obtained through the recommended mapping table; the updated core word sequence is whether the operation and maintenance personnel have changed the mapping phrase.

[0067] like Figure 5 As shown, in the step of secondary mapping, in the step of using the candidate phrase with the highest matching value as the mapping phrase of the core phrase, the mapping phrase and the core word corresponding to the core phrase are sent to the operation and maintenance end to determine whether the operation and maintenance end issues a change instruction for the mapping phrase. If the operation and maintenance end issues a change instruction, the mapping phrase corresponding to the core phrase is changed according to the change instruction.

[0068] The operation and maintenance personnel may also change the core phrase in the change instruction. If the core phrase is changed, the mapping phrase may be remapped from the above mapping table.

[0069] The operation and maintenance personnel can perform secondary mapping processing on the remaining fields after the core phrase mapping on the operation and maintenance side. The operation and maintenance personnel can view the mapping phrases corresponding to the core phrases mapped according to the preset recommended mapping table for reference and quick selection by the operation and maintenance personnel. The matching relationship that has been processed by the operation and maintenance personnel for secondary mapping will be stored in the mapping table of historical data. When the source field is encountered again in the future, the system will automatically process the matching process; at the same time, the operation and maintenance personnel can also add core words to the corresponding destination field, that is, the core phrase, according to the characteristics of the source field, to improve the matching recommendation function. If some core words are still not mapped after this step, the system will abandon these core words and will not perform mapping processing. Based on the above process, the content of the matched field mapping table and the recommended matching field mapping table will gradually become richer, and the applicability will gradually increase. Compared with the traditional field mapping method of summarizing rules, this gradually adaptive field mapping method has less workload and higher work efficiency.

[0070] By adopting the above scheme, in the early stage of processing a platform, the number of times the mapping phrases corresponding to the core phrases mapped according to the preset recommendation mapping table are sent to the operation and maintenance end is relatively large. As the number of times increases, the historical data gradually increases, and in the later stage, there is gradually no need to send the mapping phrases corresponding to the core phrases mapped according to the preset recommendation mapping table to the operation and maintenance end. This scheme can ensure the accuracy of the early stage matching, does not require manual processing in the later stage, has a smaller workload and higher work efficiency.

[0071] In some embodiments of the present invention, the step of sending an authentication token request to the target network management platform by the security authentication component and obtaining the authentication token sent by the target network management platform further includes:

[0072] The duration of the authentication token is obtained, and a request interval is set according to the duration of the authentication token. The security authentication component sends an authentication token request to the target network management platform every request interval.

[0073] First, configure the URL address of the authentication token request. After the configuration is completed, you can perform a call test. After the test passes, you need to configure the token's keep-alive validity.

[0074] Since the validity period of tokens on different network management platforms is different, the token keep-alive time interval needs to be configured according to the platform requirements in this step to ensure that valid tokens can always be obtained; for example, if the token validity period of network management platform A is 5 minutes, then the security authentication component needs to send token keep-alive requests at intervals of less than 5 minutes to complete the token keep-alive.

[0075] A specific feasible implementation plan for building a security authentication component is to store the component configuration information in the database; start a scheduled task and call http / https requests regularly to complete the token preservation; and pass the valid token to the downstream component in the form of a message queue.

[0076] In some embodiments of the present invention, in the step of using the candidate phrase with the highest matching value as the mapping phrase of the core phrase, the mapping phrase and the core word corresponding to the core phrase are sent to the operation and maintenance end to determine whether the operation and maintenance end issues a change instruction for the mapping phrase. If the operation and maintenance end issues a change instruction, the mapping phrase corresponding to the core phrase is changed according to the change instruction.

[0077] like Figure 3 As shown, in some embodiments of the present invention, the processing step of the data conversion component further includes:

[0078] Obtain the source resource of the incoming data of the target network management platform, match the source resource with the resource to be matched in the preset resource id mapping table, if the match is successful, assign the resource id of the matched resource to be matched to the source resource, if the match fails, generate a unique resource id for the source resource and record it in the resource id mapping table.

[0079] In some embodiments of the present invention, since resource IDs of multiple network management platforms may be repeated, the system of the present invention will maintain a resource ID mapping table for mapping source ID to destination ID (destination ID refers to a unique resource ID maintained by the system of the present invention). This step will add a destination ID field to the data record while retaining the source ID. If the source ID cannot be found in the form, a mapping record will be inserted and a unique destination ID will be generated for matching.

[0080] In some embodiments of the present invention, the step of storing the incoming data in the storage location of the mapping phrase corresponding to the core phrase by the data persistence component further includes adding the processed core phrase to the historical data and synchronously updating the historical data of the data conversion component.

[0081] In some embodiments of the present invention, the step of the data persistence component further includes:

[0082] Convert the data units of the incoming data to a uniform data unit.

[0083] In some embodiments of the present invention, for example, the basic unit corresponding to cpu_usage is % (percentage), and the standard unit corresponding to interface_receive_rate (interface receiving rate) is Mb / s. The data conversion template component in the data persistence component needs to unify the units of the network management platform indicator data. The system of the present invention maintains a unit mapping table, which contains the proportional factors for mapping units of various magnitudes to standard units, such as the proportional factor of kb / s corresponding to the standard unit Mb / s is 0.001. The calculation from non-standard units to standard units can be completed through the proportional factors. The present invention provides an interface-based unit mapping configuration method, and operation and maintenance personnel can define the mapping relationship between units of various magnitudes and standard units through interface configuration or csv file import to complete unit mapping.

[0084] The data persistence component receives the converted data output by the upstream data conversion template component through the message queue. The data persistence component needs to configure the database link information, database table and other information. The backend will insert the data into the corresponding database table according to the configuration information to complete the data persistence.

[0085] Another aspect of the present invention provides an ETL method for multi-network management platform operation and maintenance data using the above system, the method comprising the following steps:

[0086] Security authentication: send an authentication token request to the target network management platform and obtain the authentication token sent by the target network management platform;

[0087] Call the interface to determine whether the current authentication token is available. If available, configure the interface parameters of the target network management platform, establish a data connection with the target network management platform based on the interface parameters, and obtain data;

[0088] Data conversion, obtaining the core phrase of the data transmitted by the target network management platform, determining whether there is historical data for the current core phrase, and if the current core phrase is not historical data, calculating the matching value between the core phrase and each candidate phrase in the preset recommendation mapping table, and taking the candidate phrase with the highest matching value as the mapping phrase of the core phrase;

[0089] Data persistence: if the data conversion component determines that the current core phrase is historical data, the incoming data will be stored in the corresponding position according to the storage position of the current core phrase in the historical data; if the data conversion component determines that the current core phrase is not historical data, the incoming data will be stored in the storage position of the mapping phrase corresponding to the core phrase.

[0090] An embodiment of the present invention also provides an ETL device applied to operation and maintenance data of multiple network management platforms, the device comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps implemented by the components of the system described above.

[0091] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps implemented by the components of the ETL system for multi-network management platform operation and maintenance data are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0092] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0093] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0094] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / map> < / map> < / map>

Claims

1. An ETL system for operation and maintenance data of multiple network management platforms, characterized in that: The system comprises: A security authentication component is used to send an authentication token request to a target network management platform and obtain an authentication token sent by the target network management platform; The interface calling component is used to determine whether the current authentication token is available. If it is available, the interface parameters of the target network management platform are configured, and a data connection with the target network management platform is established based on the interface parameters to obtain data; A data conversion component obtains a core phrase of data transmitted by a target network management platform, determines whether there is historical data for the current core phrase, and if the current core phrase is not historical data, calculates a matching value between the core phrase and each candidate phrase in a preset recommendation mapping table, wherein each candidate phrase in the preset recommendation mapping table in the data conversion component is divided into a first-level phrase, a second-level phrase, and a third-level phrase, wherein different score parameters are respectively set for the first-level phrase, the score parameter of the first-level phrase> the score parameter of the second-level phrase> the score parameter of the third-level phrase, the core phrase includes at least one core phrase, and the candidate phrase includes at least one candidate phrase, and respectively calculates a matching value between each core phrase in each core phrase and each candidate phrase in the candidate phrase, and obtains a matching value between the core phrase and each candidate phrase; and the candidate phrase with the highest matching value is used as a mapping phrase for the core phrase; Data persistence component, if the data conversion component determines that the current core phrase is historical data, then according to the storage position of the current core phrase in the historical data, the incoming data is stored in the corresponding position; if the data conversion component determines that the current core phrase is not historical data, then the incoming data is stored in the storage position of the mapping phrase corresponding to the core phrase.

2. The ETL system for operation and maintenance data of multiple network management platforms according to claim 1 is characterized in that: Each candidate word in the candidate word group is preset with a score parameter. If the core word is the same as the candidate word, the score parameter of the candidate word is taken into account, and the matching score of each core word in the core word group and each candidate word in the same candidate word group is added to obtain a matching value.

3. The ETL system for operation and maintenance data of multiple network management platforms according to claim 1 or 2, characterized in that: The step of sending an authentication token request to the target network management platform by the security authentication component and obtaining the authentication token sent by the target network management platform also includes: The duration of the authentication token is obtained, and a request interval is set according to the duration of the authentication token. The security authentication component sends an authentication token request to the target network management platform every request interval.

4. The ETL system for operation and maintenance data of multiple network management platforms according to claim 1 is characterized in that: In the step of using the candidate phrase with the highest matching value as the mapping phrase of the core phrase, the mapping phrase and the core word corresponding to the core phrase are sent to the operation and maintenance end to determine whether the operation and maintenance end issues a change instruction for the mapping phrase. If the operation and maintenance end issues a change instruction, the mapping phrase corresponding to the core phrase is changed according to the change instruction.

5. The ETL system for operation and maintenance data of multiple network management platforms according to claim 1, characterized in that: The processing steps of the data conversion component also include: Obtain the source resource of the incoming data of the target network management platform, match the source resource with the resource to be matched in the preset resource id mapping table, if the match is successful, assign the resource id of the matched resource to be matched to the source resource, if the match fails, generate a unique resource id for the source resource and record it in the resource id mapping table.

6. The ETL system for operation and maintenance data of multiple network management platforms according to claim 1, characterized in that: The step of storing the incoming data in the storage location of the mapping phrase corresponding to the core phrase by the data persistence component also includes adding the processed core phrase to the historical data and synchronously updating the historical data of the data conversion component.

7. The ETL system for operation and maintenance data of multiple network management platforms according to claim 1, characterized in that: The steps of the data persistence component also include: Convert the data units of the incoming data to a uniform data unit.

8. An ETL method for multi-network management platform operation and maintenance data using the system according to any one of claims 1 to 7, the method comprising the steps of: Security authentication: send an authentication token request to the target network management platform and obtain the authentication token sent by the target network management platform; Call the interface to determine whether the current authentication token is available. If available, configure the interface parameters of the target network management platform, establish a data connection with the target network management platform based on the interface parameters, and obtain data; Data conversion, obtaining the core phrase of the data transmitted by the target network management platform, determining whether the current core phrase has historical data, and if the current core phrase is not historical data, calculating the matching value between the core phrase and each candidate phrase in the preset recommendation mapping table, wherein each candidate phrase in the preset recommendation mapping table in the data conversion component is divided into first-level words, second-level words and third-level words, and different score parameters are respectively set for the first-level words, the score parameter of the first-level words> the score parameter of the second-level words> the score parameter of the third-level words, the core phrase includes at least one core word, and the candidate phrase includes at least one candidate word, respectively calculating the matching value of each core word in each core phrase and each candidate word in the candidate phrase, and obtaining the matching value of the core phrase and each candidate phrase; taking the candidate phrase with the highest matching value as the mapping phrase of the core phrase; Data persistence: if the data conversion component determines that the current core phrase is historical data, the incoming data will be stored in the corresponding position according to the storage position of the current core phrase in the historical data; if the data conversion component determines that the current core phrase is not historical data, the incoming data will be stored in the storage position of the mapping phrase corresponding to the core phrase.

Images