A data conversion method, device, equipment and storage medium

Abstract

The present invention discloses a data conversion method, device, equipment and storage medium. The method includes: reading the process identifier of the data processor through a rule loader, and obtaining the data conversion rules from the rule generator; writing the process identifier and the data conversion rules into a preset eBPF storage space through the rule loader; intercepting the target network data whose data transmission end point points to the process identifier based on the eBPF storage space through the rule loader, and using the data conversion rules to perform data conversion on the target network data to obtain target conversion data; resending the target conversion data to the data processor for the data processor to perform matching data processing operations on the target conversion data. The technical solution of the embodiment of the present invention avoids the defects of restarting the data processor when loading rules and excessive manual participation, establishes the correlation between data, improves the efficiency and accuracy of data conversion, and improves the quality of the data conversion link.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, specifically to the field of data governance, and in particular to a data conversion method, device, equipment and storage medium. Background Art

[0002] With the introduction of the concept of observability into the IT field, it is mainly used to help operation and maintenance teams detect problems as early as possible, control them and issue warnings in critical events. However, the construction of observability is not achieved overnight. There are great differences in data formats and specifications between different periods or different suppliers. Even the definition of metadata is inconsistent in the same type of data. This inconsistency will greatly reduce the quality of the data and cause data isolation.

[0003] If there is a unified data platform and all data are placed on one platform, it seems that the problem of correlation can be solved. However, the actual situation is often that a unified platform for observing observable data is established, and the data is piled in one place. When using it, it is still viewed in the traditional way, and the correlation still depends on human knowledge and experience.

[0004] Since the data comes from various observation tools, although the data format is unified, the metadata of different tools are completely different. Therefore, how to adopt effective data governance measures to achieve data format standardization and associate data is an important issue that needs to be resolved. Summary of the invention

[0005] Based on this, the present invention provides a data conversion method, device, equipment and storage medium to solve the problem that observable data cannot be unified and associated.

[0006] In a first aspect, an embodiment of the present invention provides a data conversion method, the method comprising:

[0007] The process identifier of the data processor is read through a rule loader, and the data conversion rule is obtained from a rule generator; wherein the rule loader is constructed based on eBPF (Extended Berkeley Packet Filter);

[0008] Write the process ID and data conversion rules into the preset eBPF storage space through the rule loader;

[0009] Through the rule loader based on the eBPF storage space, the target network data whose data transmission endpoint points to the process identifier is intercepted, and the target network data is converted using the data conversion rule to obtain the target conversion data;

[0010] The target conversion data is resent to the data processor so that the data processor performs a matching data processing operation on the target conversion data.

[0011] In a second aspect, an embodiment of the present invention provides a data conversion device, the device comprising:

[0012] A data reading module is used to read the process identifier of the data processor through a rule loader and obtain data conversion rules from a rule generator; wherein the rule loader is built based on eBPF;

[0013] The data writing module is used to write the process identifier and data conversion rules into the preset eBPF storage space through the rule loader;

[0014] A target conversion data generation module is used to intercept the target network data whose data transmission endpoint points to the process identifier through the rule loader based on the eBPF storage space, and perform data conversion on the target network data using the data conversion rule to obtain the target conversion data;

[0015] The data forwarding module is used to resend the target conversion data to the data processor so that the data processor performs a matching data processing operation on the target conversion data.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a data conversion method described in any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a data conversion method described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention introduces eBPF technology to intercept the data sent to the data processor in advance, and replaces the data processor with a rule loader to perform data conversion, thereby avoiding the defects of the data processor restarting when loading rules and excessive manual participation. It not only establishes the correlation between data, but also improves the efficiency and accuracy of data conversion, and improves the quality of the data conversion link.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a data conversion method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a schematic diagram of an original data conversion process applicable to Embodiment 1 of the present invention;

[0026] Figure 3 is a flow chart of another data conversion method provided according to the second embodiment of the present invention;

[0027] Figure 4 is a structural schematic diagram of a data conversion device provided according to Embodiment 3 of the present invention;

[0028] Figure 5 The present invention is a schematic diagram of the structure of an electronic device for implementing a data conversion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a data conversion method provided in the first embodiment of the present invention. This embodiment can be applied to the case where data conversion is performed on target data using data conversion rules. The method can be executed by a data conversion device. The data conversion device can be implemented in the form of hardware and / or software. The data conversion device can be configured in a rule loader. Figure 1 As shown, the method includes:

[0033] S110, reading the process identifier of the data processor through the rule loader, and obtaining the data conversion rule from the rule generator; wherein the rule loader is constructed based on eBPF.

[0034] Figure 2 This is a complete implementation flow of an original data conversion process applicable to the technical solution of the embodiment of the present invention. Specifically, Figure 2 As shown, the related technology usually uses a data processor to convert the data packets received from the data receiver according to the preloaded conversion rules, and sends the converted data to the data collector to perform the next step of operation. However, the data processor needs to be restarted each time before performing data conversion in order to reload the conversion rules, which causes the processing process of the current data stream to be interrupted.

[0035] Therefore, the embodiment of the present invention creatively proposes to use a rule loader to Figure 2 In process ③, the data that is about to flow to the data processor is identified and intercepted, and the data conversion rules generated in advance are extracted in the rule generator, and the intercepted data is converted before being sent to the data processor. Among them, the data processor is used to implement other data processing operations besides data conversion in the embodiment of the present invention; and the data conversion rule refers to the general conversion specification established for data under different specifications.

[0036] Since eBPF can support developers to dynamically load, update and run user-defined code without restarting the kernel or changing the kernel source code, the embodiment of the present invention creatively proposes to replace the existing means to achieve data conversion based on eBPF. Specifically, the rule loader loads the eBPF program to identify and intercept the network data that is about to be passed into the rule generator, and the process identifier is used as the identification basis.

[0037] S120. Write the process identifier and the data conversion rule into a preset eBPF storage space through a rule loader.

[0038] In the embodiment of the present invention, the concept of eBPF storage space is introduced to store process identifiers and data conversion rules for subsequent eBPF extraction. In the embodiment of the present invention, the eBPF storage space can be specifically expressed as a Map space. Since the Map space is shared by the user and the kernel, the data processed in the kernel can be directly stored in the Map space under the eBPF technology, and then the data can be retrieved from the user space.

[0039] S130. Intercept the target network data whose data transmission endpoint points to the process identifier through the rule loader based on the eBPF storage space, and perform data conversion on the target network data using the data conversion rule to obtain target converted data.

[0040] Among them, the process identifier of the data processor mainly refers to pid and network socket. When the program is running, the system will automatically assign a unique pid to the process, which is responsible for identifying each process; and the network socket is used to realize the mutual communication between processes in the network. The rule loader reads the process identifier of the network data and matches it with the process identifier in the eBPF storage space to determine whether the end point of the network data flow is the rule generator. The network data whose data flow end point is the rule generator is called the target network data, wherein the target network data may include a data identifier name, an attribute name, and an attribute value; after the target network data is converted, the target conversion data can be obtained, and multiple target network data may be output as the same target conversion data after being processed by the data conversion rule; at the same time, the data conversion rule can be one or more of the first type of data conversion rule, the second type of data conversion rule, or the third type of data conversion rule.

[0041] Optionally, the data conversion rule may include: at least one first-category data conversion rule, the first-category data conversion rule including a data indicator name set; pre-specifying a unique conversion data indicator name in each data indicator name set;

[0042] Specifically, the first type of data conversion rules refers to a conversion rule constructed for data indicator names, which are also data asset or entity names. Due to the lack of a unified standardization system, even if they refer to the same asset or entity, the data indicator names are not exactly the same. Therefore, each data indicator name set can be understood as a collection of indicator names referring to the same entity, and the pre-specified unique conversion data indicator name can be the data indicator name of a specific position in the set.

[0043] Further, the data conversion rule may include: at least one second type of data conversion rule, the second type of data conversion rule including a set of attribute names under a set of data indicator names; pre-specifying a unique conversion attribute name in each attribute name set;

[0044] Specifically, the second type of data conversion rules refers to a conversion rule constructed for all attributes contained in a certain data indicator. Due to the lack of a unified standardization system, even if the same attribute is referred to under the same data indicator name, different suppliers may name the attribute differently. Therefore, each attribute name set can be understood as a collection of attribute names referring to the same attribute, and the data in the set can also be unified into the final output attribute name according to the attribute name of a specific position in the set.

[0045] Optionally, the data conversion rules may include: at least one third-category data conversion rule, the third-category data conversion rule including a set of attribute values ​​under a set attribute name of a set data indicator name; and pre-specifying a unique conversion attribute value in each attribute value set.

[0046] Specifically, the third type of data conversion rules refers to a conversion rule constructed for the attribute value of a specific attribute contained in a specific data indicator. Since the same attribute may contain multiple attribute values, each attribute value set can be understood as the collection of all attribute values ​​contained in the same attribute, and the data in the set can also be unified into the final output attribute value according to the attribute value of a specific position in the set.

[0047] S140: resend the target conversion data to the data processor, so that the data processor performs a matching data processing operation on the target conversion data.

[0048] After the target network data is processed using the conversion rules, the target conversion data is obtained. After the target conversion data is sent to the data processor, the data processor no longer needs to perform repeated data conversion operations on the target conversion data, but can continue to perform some other preset processing except data conversion. This avoids the defect that the rule processor mentioned above needs to be restarted every time the conversion rules are loaded, thereby interrupting the data flow. At the same time, automatic data conversion based on conversion rules is also realized without human intervention.

[0049] In an embodiment of the present invention, the process identifier of the data processor is read by the rule loader, and the data conversion rules are obtained from the rule generator; the process identifier and the data conversion rules are written into the preset eBPF storage space by the rule loader; the target network data whose data transmission end point points to the process identifier is intercepted by the rule loader based on the eBPF storage space, and the target network data is converted using the data conversion rules to obtain the target converted data; the target converted data is resent to the data processor so that the data processor can perform matching data processing operations on the target converted data. A data conversion method is provided, which introduces the eBPF technology to intercept the data sent to the data processor in advance, and replaces the data processor with the rule loader to perform data conversion, thereby avoiding the defects of the data processor restarting when loading the rules and too much manual participation, not only establishing the correlation between the data, but also improving the efficiency and accuracy of the data conversion, and improving the quality of the data conversion link.

[0050] Based on the above embodiments, the rule loader uses the matching data conversion rules to convert the target network data to obtain the target converted data, which may include:

[0051] Obtain the target data indicator name corresponding to the target network data through the rule loader;

[0052] Matching the target data index name with each first-category data conversion rule through a rule loader to obtain a target first-category data conversion rule containing the target data index name;

[0053] The target conversion data indicator name matching the target first-category data conversion rule is obtained through the rule loader, and the target data indicator name of the target network data is converted into the target conversion data indicator name.

[0054] For ease of understanding, the target network data may be data 1, data 2, data 3, data 4, data 5, and data 6. The target data index names corresponding to each target network data are index 1, index 2, index 3, index 4, index 5, and index 6, respectively. Assume that the first type of conversion rule is: data index name set A: <index 1, index 2, index 5>; data index name set B: <index 3, index 4, index 6>. If the first name in the pre-specified set is the only conversion data index name, the first type of data conversion rule can be used to convert the target data index name of data 2 to index 1, and the target data index name of data 5 to index 1. Similarly, the target data index name of data 4 can be converted to index 3, and the target data index name of data 6 can be converted to index 3.

[0055] Based on the above embodiments, the rule loader uses the matching data conversion rules to convert the target network data to obtain the target converted data, which may include:

[0056] Obtain the target data indicator name and target attribute name corresponding to the target network data through the rule loader;

[0057] Match the target data index name and the target attribute name with each second-category data conversion rule through the rule loader to obtain the target second-category data conversion rule containing both the target data index name and the target attribute name;

[0058] The target conversion attribute name matching the target second-category data conversion rule is obtained through the rule loader, and the target attribute name of the target network data is converted into the target conversion attribute name.

[0059] In a specific example, if the target data indicator name of the target network data 1 is indicator 1, the target attribute name is attribute a, the target data indicator name of the target network data 2 is indicator 2, and the target attribute name is attribute c, and the second type of data conversion rules include: attribute name set C: <indicator 1, attribute a, attribute b>; attribute name set D: <indicator 2, attribute c, attribute d, attribute e>, and at the same time, it is assumed that the last digit in the pre-specified set is a unique conversion attribute name.

[0060] By matching the rules, it is determined that the target network data 1 matches the attribute name set C, and the attribute a of the target network data 1 is converted to the attribute b; at the same time, it is determined that the target network data 2 matches the attribute name set D, and the attribute c of the target network data 2 is converted to the attribute e.

[0061] The rule loader uses the matching data conversion rules to convert the target network data to obtain the target conversion data, which may include:

[0062] Obtain the target data indicator name, target attribute name and target attribute value corresponding to the target network data through the rule loader;

[0063] Match the target data indicator name, target attribute name and target attribute value with each of the three types of data conversion rules through the rule loader to obtain the target third type of data conversion rule that contains the target data indicator name, target attribute name and target attribute value;

[0064] The target conversion attribute value matching the target third-category data conversion rule is obtained through the rule loader, and the target attribute value of the target network data is converted into the target conversion attribute value.

[0065] In a specific example, if the target data index name of target network data 1 is index 1, the target attribute name is attribute a, and the target attribute value is I, the target data index name of target network data 2 is index 2, the target attribute name is attribute c, and the target attribute value is IV. The three types of data conversion rules include: attribute value set E: <index 1, attribute a, I, II>; attribute value set F: <index 2, attribute b, III, IV>, and at the same time, it is assumed that the last digit in the pre-specified set is the only conversion attribute value.

[0066] By performing rule matching, it is determined that the target network data 1 matches the attribute value set E, and the attribute value I of the attribute a of the target network data 1 is converted to II; at the same time, it is determined that the target network data 2 matches the attribute value set F, and the attribute value IV of the attribute c of the target network data 2 is output as is without conversion.

[0067] Embodiment 2

[0068] Figure 3 This is a flowchart of another data conversion method provided in the second embodiment of the present invention. This embodiment can be applied to the case where corresponding data conversion rules are generated according to the tree structure of historical data. The method can be executed by a data conversion device. The data conversion device can be implemented in the form of hardware and / or software. The data conversion device can be executed by a rule generator and a rule loader. Figure 3 As shown, the method includes:

[0069] S310. Collect observable historical data in the database through a rule generator, and generate at least one tree structure with the indicator data name as the root node based on the observable historical data; wherein, in each tree structure, the attribute name is the first-level leaf node, and the attribute value is the second-level leaf node.

[0070] First of all, observability is a property. It means starting from the inside of the system, monitoring the internal operation of the system based on a white-box approach, improving the observable nature of the system, and helping developers understand "what is happening" and "why it is happening". The observability type relies on three types of telemetry data: logs, metrics, and traces to provide in-depth visibility into distributed systems. Among them, metrics are usually measurable monitoring indicator information over a period of time, including specific attributes such as timestamps, names, KPIs, and values; logs are text records of events that occur at a specific time, including plain text, structured, and binary formats; traces are used to record processing information within a single request, including data such as service calls and processing time.

[0071] Furthermore, according to the type and subordinate structure of the data, the historical data is parsed into a multi-level tree structure of indicator data name-attribute name-attribute value, that is, the root node indicator data name contains multiple first-level leaf node attribute names, and each first-level leaf node attribute name may contain multiple second-level leaf node attribute values.

[0072] It can be understood that the number of tree structures matches the number of indicator data names.

[0073] S320: Generate at least one first-category data conversion rule according to the root node of each tree structure through a rule generator.

[0074] Optionally, generating at least one first-category data conversion rule according to the root node of each tree structure by a rule generator may include:

[0075] Input each root node into a pre-trained word vector model for vectorization processing to obtain a root node vector corresponding to each root node;

[0076] Calculate the Euclidean distance between two root node vectors, and determine the similarity between two root node vectors based on the Euclidean distance calculation result;

[0077] Compare each similarity with a preset similarity threshold, and divide the root nodes corresponding to the root node vectors whose similarity is greater than the similarity threshold into at least one data index name set;

[0078] A data indicator name is selected from each data indicator name set as a conversion data indicator name corresponding to the data indicator name set, so as to obtain at least one first-category data conversion rule.

[0079] As can be seen from Example 1, the first type of data conversion rule is the conversion rule corresponding to the data index name, and the word conversion model, as a model tool for generating word vectors, can output the data index name as a vectorized representation, that is, the root node vector. As a common method for solving the natural length between vectors, the Euclidean distance is used to calculate the distance between each root node vector in the present invention. The similarity threshold refers to the minimum threshold set when performing a similarity comparison. The similarity between the two root nodes is obtained by calculating the Euclidean distance of the two root node vectors. If the similarity between the two root nodes exceeds the set similarity threshold, they can be considered similar and divided into the same set, that is, the data index name set A in Example 1: <index 1, index 2, index 5> or the data index name set B: <index 3, index 4, index 6>.

[0080] Furthermore, indicator 1 may be selected as the conversion data indicator name of data indicator name set A, and indicator 3 may be selected as the conversion data indicator name of data indicator name set B.

[0081] S330. Generate at least one second-category data conversion rule according to each first-level leaf node under the same root node through a rule generator.

[0082] Furthermore, at least one second-type data conversion rule is generated by the rule generator according to each first-level leaf node under the same root node, which may include:

[0083] Input each first-level leaf node under the same root node into a pre-trained word vector model for vectorization processing to obtain a first-level leaf node vector;

[0084] Calculate the Euclidean distance between the first-level leaf node vectors, and determine the similarity between the first-level leaf node vectors according to the Euclidean distance calculation result;

[0085] Compare each similarity with a preset similarity threshold, and divide the first-level leaf nodes corresponding to the first-level leaf node vectors whose similarity is greater than the similarity threshold into the attribute name set under the root node to which each first-level leaf node belongs;

[0086] An attribute name is selected from each attribute name set as a conversion attribute name corresponding to the attribute name set, so as to obtain at least one second-category data conversion rule.

[0087] Specifically, the second type of data conversion rules are conversion rules corresponding to data attribute names. Similarly, the similarity between two first-level leaf nodes is calculated based on the Euclidean distance between the two first-level leaf node vectors. If the similarity between the two first-level leaf nodes exceeds the set similarity threshold, they can be considered similar and divided into the same set, that is, the attribute name set C mentioned in Example 1: <Indicator 1, attribute a, attribute b>; attribute name set D: <Indicator 2, attribute c, attribute d, attribute e>.

[0088] Similarly, attribute b can be used as the conversion attribute name of attribute name set C; and attribute e can be used as the conversion attribute name of attribute name set D.

[0089] S340. Generate at least one third-category data conversion rule according to each second-level leaf node under the same first-level leaf node of the same root node through a rule generator.

[0090] Optionally, generating at least one third-category data conversion rule according to each second-level leaf node under the same first-level leaf node of the same root node by a rule generator may include:

[0091] Get all the second-level leaf nodes under the same first-level leaf node of the same root node;

[0092] Identify in the configuration management database the secondary leaf node information that matches all the secondary leaf nodes under the same primary leaf node of the same root node;

[0093] In combination with the dependency relationship between the second-level leaf node information in the configuration management database, each second-level leaf node is divided into at least one attribute value set of each second-level leaf node under the same first-level leaf node of the same root node;

[0094] An attribute value is selected from each attribute value set as a conversion attribute value corresponding to the attribute value set, so as to obtain at least one third-category data conversion rule.

[0095] Specifically, the third type of data conversion rules are conversion rules corresponding to attribute values. The configuration management database, as a logical database composed of multiple physical databases, stores all configuration information of equipment in the IT architecture of the management enterprise, and can track all IT components and the relationships between components. Therefore, the data stored in the configuration management database is ordered information integrated by a certain logical model. By obtaining the secondary leaf node (attribute value), all attribute values ​​under a certain level of leaf nodes under the same root node corresponding to the secondary leaf node can be found in the configuration management database. If the secondary leaf node matches the attribute value, it is added to the attribute value set under the current first-level leaf node, that is, the attribute value set E mentioned in Example 1: <Indicator 1, attribute a, I, II>; attribute value set F: <Indicator 2, attribute b, III, IV>.

[0096] Similarly, II can be used as the conversion attribute value of the attribute value set E; and IV can be used as the conversion attribute value of the attribute value set F.

[0097] Furthermore, the method may also include:

[0098] The rule generator sends the first type of data conversion rules, the second type of data conversion rules, and the third type of data conversion rules to the rule modifier for rule modification;

[0099] The rule generator receives the first type of data conversion rules, the second type of data conversion rules and the third type of data conversion rules corrected by the rule corrector.

[0100] Specifically, since the rule generator automatically generates various data conversion rules, in order to further ensure the accuracy of these data conversion rules, the generated data conversion rules can be first sent to the rule corrector, in which various regular expressions can be pre-customized for rule correction. Alternatively, the rule corrector can be connected to a manual correction platform, which can implement further rule correction.

[0101] S350, reading the process identifier of the data processor through the rule loader, and obtaining the data conversion rule from the rule generator; wherein the rule loader is constructed based on eBPF.

[0102] S360. Write the process identifier and data conversion rules into a preset eBPF storage space through a rule loader.

[0103] S370. Intercept the target network data whose data transmission endpoint points to the process identifier through the rule loader based on the eBPF storage space, and perform data conversion on the target network data using the data conversion rule to obtain target converted data.

[0104] S380, resending the target conversion data to the data processor, so that the data processor performs a matching data processing operation on the target conversion data.

[0105] The technical solution of the embodiment of the present invention further refines the overall solution, provides a vector construction method based on a multi-level tree structure, and determines the similarity between different node vectors by calculating the Euclidean distance, divides multiple sets based on the similarity, and uniformly outputs the data in the sets according to pre-set conversion rules. The technical solution of the embodiment of the present invention realizes rule-based automatic data conversion, greatly improves the governance efficiency of observable data, saves time and labor costs, and contributes to the establishment of an observability platform.

[0106] Embodiment 3

[0107] Figure 4 This is a schematic diagram of the structure of a data conversion device provided by Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0108] The data reading module 410 is used to read the process identifier of the data processor through the rule loader and obtain the data conversion rule from the rule generator; wherein the rule loader is constructed based on eBPF;

[0109] A data writing module 420 is used to write the process identifier and the data conversion rule into a preset eBPF storage space through a rule loader;

[0110] The target conversion data generation module 430 is used to intercept the target network data whose data transmission endpoint points to the process identifier through the rule loader based on the eBPF storage space, and perform data conversion on the target network data using the data conversion rule to obtain the target conversion data;

[0111] The data forwarding module 440 is used to resend the target conversion data to the data processor so that the data processor performs a matching data processing operation on the target conversion data.

[0112] In an embodiment of the present invention, the data reading module 410 reads the process identifier of the data processor through the rule loader and obtains the data conversion rule from the rule generator; the data writing module 420 writes the process identifier and the data conversion rule into the preset eBPF storage space through the rule loader; the target conversion data generation module 430 intercepts the target network data whose data transmission end point points to the process identifier based on the eBPF storage space through the rule loader, and uses the data conversion rule to perform data conversion on the target network data to obtain the target conversion data; the data forwarding module 440 resends the target conversion data to the data processor so that the data processor performs matching data processing operations on the target conversion data. A new data conversion method is provided, which intercepts the data sent to the data processor in advance by introducing the eBPF technology, and replaces the data processor by the rule loader to perform data conversion, avoiding the defects of restarting the data processor when loading the rules and too much manual participation, not only establishing the correlation between the data, but also improving the efficiency and accuracy of the data conversion, and improving the quality of the data conversion link.

[0113] Based on the above embodiments, the data conversion rule includes: at least one first-class data conversion rule, the first-class data conversion rule includes a data indicator name set; a unique conversion data indicator name is pre-specified in each data indicator name set;

[0114] The rule loader uses the matching data conversion rules to convert the target network data to obtain target conversion data, including:

[0115] Obtain the target data indicator name corresponding to the target network data through the rule loader;

[0116] Matching the target data index name with each first-category data conversion rule through a rule loader to obtain a target first-category data conversion rule containing the target data index name;

[0117] The target conversion data indicator name matching the target first-category data conversion rule is obtained through the rule loader, and the target data indicator name of the target network data is converted into the target conversion data indicator name.

[0118] Based on the above embodiments, the data conversion rule further includes: at least one second type of data conversion rule, the second type of data conversion rule includes setting a set of attribute names under a data indicator name; pre-specifying a unique conversion attribute name in each attribute name set;

[0119] The rule loader uses the matching data conversion rules to convert the target network data to obtain target conversion data, including:

[0120] Obtain the target data indicator name and target attribute name corresponding to the target network data through the rule loader;

[0121] Match the target data index name and the target attribute name with each second-category data conversion rule through the rule loader to obtain the target second-category data conversion rule containing both the target data index name and the target attribute name;

[0122] The target conversion attribute name matching the target second-category data conversion rule is obtained through the rule loader, and the target attribute name of the target network data is converted into the target conversion attribute name.

[0123] Based on the above embodiments, the data conversion rule may further include: at least one third type of data conversion rule, the third type of data conversion rule including a set of attribute values ​​under a set attribute name of a set data indicator name; pre-specifying a unique conversion attribute value in each attribute value set;

[0124] The rule loader uses the matching data conversion rules to convert the target network data to obtain target conversion data, including:

[0125] Obtain the target data indicator name, target attribute name and target attribute value corresponding to the target network data through the rule loader;

[0126] Match the target data indicator name, target attribute name and target attribute value with each of the three types of data conversion rules through the rule loader to obtain the target third type of data conversion rule that contains the target data indicator name, target attribute name and target attribute value;

[0127] The target conversion attribute value matching the target third-category data conversion rule is obtained through the rule loader, and the target attribute value of the target network data is converted into the target conversion attribute value.

[0128] Based on the above embodiments, the module further includes:

[0129] A tree structure generating unit of a root node is used to collect observable historical data in a database through a rule generator, and generate at least one tree structure with an indicator data name as a root node according to the observable historical data, wherein in each tree structure, an attribute name is a first-level leaf node, and an attribute value is a second-level leaf node;

[0130] A first-category data conversion rule generating unit, configured to generate at least one first-category data conversion rule according to a root node of each tree structure through a rule generator;

[0131] A second-type data conversion rule generating unit, configured to generate at least one second-type data conversion rule according to each first-level leaf node under the same root node through a rule generator;

[0132] A third-category data conversion rule generating unit, configured to generate at least one third-category data conversion rule according to each second-level leaf node under the same first-level leaf node of the same root node through a rule generator;

[0133] Based on the above embodiments, the first type of data conversion rule generating unit is specifically used for:

[0134] Input each root node into a pre-trained word vector model for vectorization processing to obtain a root node vector corresponding to each root node;

[0135] Calculate the Euclidean distance between two root node vectors, and determine the similarity between two root node vectors based on the Euclidean distance calculation result;

[0136] Compare each similarity with a preset similarity threshold, and divide the root nodes corresponding to the root node vectors whose similarity is greater than the similarity threshold into at least one data index name set;

[0137] A data indicator name is selected from each data indicator name set as a conversion data indicator name corresponding to the data indicator name set, so as to obtain at least one first-category data conversion rule.

[0138] Based on the above embodiments, the second type of data conversion rule generating unit is specifically used for:

[0139] Input each first-level leaf node under the same root node into a pre-trained word vector model for vectorization processing to obtain a first-level leaf node vector;

[0140] Calculate the Euclidean distance between the first-level leaf node vectors, and determine the similarity between the first-level leaf node vectors according to the Euclidean distance calculation result;

[0141] Compare each similarity with a preset similarity threshold, and divide the first-level leaf nodes corresponding to the first-level leaf node vectors whose similarity is greater than the similarity threshold into the attribute name set under the root node to which each first-level leaf node belongs;

[0142] An attribute name is selected from each attribute name set as a conversion attribute name corresponding to the attribute name set, so as to obtain at least one second-category data conversion rule.

[0143] Based on the above embodiments, the third type of data conversion rule generation unit is specifically used for:

[0144] Get all the second-level leaf nodes under the same first-level leaf node of the same root node;

[0145] Identify in the configuration management database the secondary leaf node information that matches all the secondary leaf nodes under the same primary leaf node of the same root node;

[0146] In combination with the dependency relationship between the second-level leaf node information in the configuration management database, each second-level leaf node is divided into at least one attribute value set of each second-level leaf node under the same first-level leaf node of the same root node;

[0147] An attribute value is selected from each attribute value set as a conversion attribute value corresponding to the attribute value set, so as to obtain at least one third-category data conversion rule.

[0148] On the basis of the above embodiments, the following may also be included:

[0149] The rule generator sends the first type of data conversion rules, the second type of data conversion rules, and the third type of data conversion rules to the rule modifier for rule modification;

[0150] The rule generator receives the first type of data conversion rules, the second type of data conversion rules and the third type of data conversion rules corrected by the rule corrector.

[0151] A data conversion device provided in an embodiment of the present invention can execute a data conversion method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0152] Embodiment 4

[0153] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0154] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0155] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data conversion method.

[0157] That is: the process identifier of the data processor is read through the rule loader, and the data conversion rule is obtained from the rule generator; wherein the rule loader is constructed based on the extended Berkeley packet filter eBPF;

[0158] Write the process ID and data conversion rules into the preset eBPF storage space through the rule loader;

[0159] Through the rule loader based on the eBPF storage space, the target network data whose data transmission endpoint points to the process identifier is intercepted, and the target network data is converted using the data conversion rule to obtain the target conversion data;

[0160] The target conversion data is resent to the data processor so that the data processor performs a matching data processing operation on the target conversion data.

[0161] In some embodiments, a data conversion method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of a data conversion method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform a data conversion method in any other appropriate manner (e.g., by means of firmware).

[0162] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0164] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0166] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0167] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0168] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0169] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data conversion method, characterized in that: include: The process identifier of the data processor is read through the rule loader, and the data conversion rule is obtained from the rule generator; wherein the rule loader is constructed based on the extended Berkeley packet filter eBPF; the observable historical data is collected in the database through the rule generator, and at least one tree structure with the indicator data name as the root node is generated according to the observable historical data, wherein in each tree structure, the attribute name is the first-level leaf node, and the attribute value is the second-level leaf node; Generate at least one first-category data conversion rule according to the root node of each tree structure by a rule generator; Generate at least one second-category data conversion rule according to each first-level leaf node under the same root node through a rule generator; Generate at least one third-category data conversion rule according to each second-level leaf node under the same first-level leaf node of the same root node by a rule generator; Write the process ID and data conversion rules into the preset eBPF storage space through the rule loader; Through the rule loader based on the eBPF storage space, the target network data whose data transmission endpoint points to the process identifier is intercepted, and the target network data is converted using the data conversion rule to obtain the target conversion data; The target conversion data is resent to the data processor so that the data processor performs a matching data processing operation on the target conversion data.

2. The method according to claim 1, characterized in that The data conversion rules include: at least one first-class data conversion rule, the first-class data conversion rule includes a data indicator name set; a unique conversion data indicator name is pre-specified in each data indicator name set; The rule loader uses the matching data conversion rules to convert the target network data to obtain target conversion data, including: Obtain the target data indicator name corresponding to the target network data through the rule loader; Matching the target data index name with each first-category data conversion rule through a rule loader to obtain a target first-category data conversion rule containing the target data index name; The target conversion data indicator name matching the target first-category data conversion rule is obtained through the rule loader, and the target data indicator name of the target network data is converted into the target conversion data indicator name.

3. The method according to claim 1, characterized in that: The data conversion rules include: at least one second-category data conversion rule, the second-category data conversion rule includes a set of attribute names under a set of data indicators; a unique conversion attribute name is pre-specified in each attribute name set; The rule loader uses the matching data conversion rules to convert the target network data to obtain target conversion data, including: Obtain the target data indicator name and target attribute name corresponding to the target network data through the rule loader; Match the target data index name and the target attribute name with each second-category data conversion rule through the rule loader to obtain the target second-category data conversion rule containing both the target data index name and the target attribute name; The target conversion attribute name matching the target second-category data conversion rule is obtained through the rule loader, and the target attribute name of the target network data is converted into the target conversion attribute name.

4. The method according to claim 1, characterized in that: The data conversion rules include: at least one third-category data conversion rule, the third-category data conversion rule includes a set of attribute values ​​under a set attribute name of a set data indicator name; a unique conversion attribute value is pre-specified in each attribute value set; The rule loader uses the matching data conversion rules to convert the target network data to obtain target conversion data, including: Obtain the target data indicator name, target attribute name and target attribute value corresponding to the target network data through the rule loader; Match the target data indicator name, target attribute name and target attribute value with each of the three types of data conversion rules through the rule loader to obtain the target third type of data conversion rule that contains the target data indicator name, target attribute name and target attribute value; The target conversion attribute value matching the target third-category data conversion rule is obtained through the rule loader, and the target attribute value of the target network data is converted into the target conversion attribute value.

5. The method according to claim 1, characterized in that At least one first-category data conversion rule is generated by a rule generator according to the root node of each tree structure, including: Input each root node into a pre-trained word vector model for vectorization processing to obtain a root node vector corresponding to each root node; Calculate the Euclidean distance between two root node vectors, and determine the similarity between two root node vectors based on the Euclidean distance calculation result; Compare each similarity with a preset similarity threshold, and divide the root nodes corresponding to the root node vectors whose similarity is greater than the similarity threshold into at least one data index name set; Selecting a data indicator name from each data indicator name set as a conversion data indicator name corresponding to the data indicator name set, so as to obtain at least one first-category data conversion rule; Generate at least one second-category data conversion rule according to each first-level leaf node under the same root node through a rule generator, including: Input each first-level leaf node under the same root node into a pre-trained word vector model for vectorization processing to obtain a first-level leaf node vector; Calculate the Euclidean distance between the first-level leaf node vectors, and determine the similarity between the first-level leaf node vectors according to the Euclidean distance calculation result; Compare each similarity with a preset similarity threshold, and divide the first-level leaf nodes corresponding to the first-level leaf node vectors whose similarity is greater than the similarity threshold into the attribute name set under the root node to which each first-level leaf node belongs; Selecting one attribute name from each attribute name set as a conversion attribute name corresponding to the attribute name set, so as to obtain at least one second-category data conversion rule; At least one third-category data conversion rule is generated by a rule generator according to each second-level leaf node under the same first-level leaf node of the same root node, including: Get all the second-level leaf nodes under the same first-level leaf node of the same root node; Identify in the configuration management database the secondary leaf node information that matches all the secondary leaf nodes under the same primary leaf node of the same root node; In combination with the dependency relationship between the second-level leaf node information in the configuration management database, each second-level leaf node is divided into at least one attribute value set of each second-level leaf node under the same first-level leaf node of the same root node; An attribute value is selected from each attribute value set as a conversion attribute value corresponding to the attribute value set, so as to obtain at least one third-category data conversion rule.

6. The method according to claim 1, characterized in that The method further comprises: The rule generator sends the first type of data conversion rules, the second type of data conversion rules, and the third type of data conversion rules to the rule modifier for rule modification; The rule generator receives the first type of data conversion rules, the second type of data conversion rules and the third type of data conversion rules corrected by the rule corrector.

7. A data conversion device, characterized in that: include: A data reading module is used to read the process ID of the data processor through the rule loader and obtain the data conversion rules from the rule generator; The rule loader is built based on the extended Berkeley packet filter eBPF; The data reading module includes: a tree structure generating unit of a root node, which is used to collect observable historical data in a database through a rule generator, and generate at least one tree structure with an indicator data name as a root node according to the observable historical data, wherein in each tree structure, an attribute name is a first-level leaf node, and an attribute value is a second-level leaf node; A first-category data conversion rule generating unit, configured to generate at least one first-category data conversion rule according to a root node of each tree structure through a rule generator; A second-type data conversion rule generating unit, configured to generate at least one second-type data conversion rule according to each first-level leaf node under the same root node through a rule generator; A third-category data conversion rule generating unit, configured to generate at least one third-category data conversion rule according to each second-level leaf node under the same first-level leaf node of the same root node through a rule generator; The data writing module is used to write the process identifier and data conversion rules into the preset eBPF storage space through the rule loader; A target conversion data generation module is used to intercept the target network data whose data transmission endpoint points to the process identifier through the rule loader based on the eBPF storage space, and perform data conversion on the target network data using the data conversion rule to obtain the target conversion data; The data forwarding module is used to resend the target conversion data to the data processor so that the data processor performs a matching data processing operation on the target conversion data.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a data conversion method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a data conversion method according to any one of claims 1 to 6 when executed.

Images