Metadata type classification method, electronic device, and storage medium

Abstract

This application discloses a method for classifying metadata types, as well as an electronic device and a computer-readable storage medium. The solution provided by this application includes: acquiring field information of metadata in a target big data environment; constructing a target graph network based on the field information, wherein nodes in the target graph network represent metadata, and edges represent field information associated with two corresponding connected metadata; determining the weights of the field information of the metadata in the target graph network; training a node type discrimination model based on samples and labels determined by the node data corresponding to the nodes in the target graph network, wherein the labels include the application type of the metadata represented by the node corresponding to the target big data environment, and the sample features include the field information of the metadata represented by the node; and classifying the nodes in the target graph network into application types based on the type discrimination model.

Description

Technical Field

[0001] This application relates to the field of big data processing, and in particular to a method for classifying metadata types and an electronic device and computer-readable storage medium. Background Technology

[0002] With the widespread adoption of big data applications, effectively managing massive amounts of data in a big data environment has become a key focus for enhancing big data capabilities. Massive data management typically involves managing metadata, which is often also massive in scale. Therefore, accurately and efficiently classifying metadata automatically to improve the efficiency of data quality management in a big data environment is a crucial technical challenge that needs to be addressed. Summary of the Invention

[0003] The purpose of this application is to provide a metadata type classification method, an electronic device, and a computer-readable storage medium to solve the problem of how to accurately and efficiently classify metadata categories.

[0004] To solve the above-mentioned technical problems, this specification is implemented as follows:

[0005] Firstly, a method for classifying metadata types is provided, comprising: acquiring field information of metadata in a target big data environment; constructing a target graph network based on the field information, wherein nodes in the target graph network represent metadata, and edges represent field information associated with two corresponding connected metadata; determining the weights of the field information of the metadata in the target graph network; training a node type discrimination model based on samples and labels determined by node data corresponding to nodes in the target graph network, wherein the labels include the application type of the metadata represented by the node corresponding to the target big data environment, and the sample features include the field information of the metadata represented by the node; and classifying the nodes in the target graph network by application type based on the type discrimination model.

[0006] Optionally, the field information includes the creation time of the metadata, the description field of the metadata, the name of the table to which the metadata belongs, and the description field of the table to which the metadata belongs. Constructing the target graph network based on the field information includes:

[0007] Based on the creation time of the metadata, the description field of the metadata, the name of the table to which the metadata belongs, and the description field of the table to which the metadata belongs, determine the first type of metadata that has a contextual reference relationship with other metadata;

[0008] Construct a first-type graph network with each type of first-type metadata as a node and the description field of the table to which each type of first-type metadata belongs as an edge.

[0009] Optionally, the field information includes metadata data types, and constructing a target graph network based on the field information includes:

[0010] Based on the data type of the metadata, obtain the second type of metadata about the target topic in the target script, wherein the target script is used to record the analysis data or application data of the target topic;

[0011] Using each type of second-class metadata as a node, construct a fully connected network of each type of second-class metadata corresponding to the target script;

[0012] A second type of graph network is constructed using the connections between identical second metadata that appear simultaneously in the fully connected networks corresponding to different scripts as edges.

[0013] Optionally, the field information includes a name or description field of metadata, and constructing a target graph network based on the field information includes:

[0014] Identify a third type of metadata that has the same metadata name or description field;

[0015] A third-type graph network is constructed using each third-type metadata as a node and the same metadata name or description field corresponding to each third-type metadata as an edge.

[0016] Optionally, the weights of the field information of the metadata are determined in the target graph network, including:

[0017] Determine the sorting value of the target field information corresponding to the target metadata;

[0018] The weights of the target field information are obtained by performing latent factor encoding on the sorted values.

[0019] Optionally, the target graph network may include only a single type of graph network.

[0020] Based on the samples and labels determined by the node data corresponding to the nodes in the target graph network, a node type discrimination model is trained, including:

[0021] Based on the samples and labels determined by the node data corresponding to the nodes in the graph network of the single type, the type discrimination model is trained until the weights of the field information corresponding to the nodes in the graph network of the single type converge.

[0022] Optionally, the target graph network includes multiple types of subgraph networks.

[0023] Based on the samples and labels determined by the node data corresponding to the nodes in the target graph network, a node type discrimination model is trained, including:

[0024] Determine the network weights and subtype discrimination models for each type of subgraph network;

[0025] Based on the network weights corresponding to each type of subgraph network, a type graph network containing each type of subgraph network is generated. The type discrimination model corresponding to the type graph network includes each subtype discrimination model corresponding to each type of subgraph network.

[0026] Based on the samples and labels determined by the node data corresponding to the nodes in each type of subgraph network, the corresponding subtype discrimination model is trained until the weights of the field information corresponding to the nodes in each type of subgraph network and the weights of each network converge, thus obtaining the type discrimination model.

[0027] Optionally, before training the node type discrimination model, the following steps are also included:

[0028] Based on the weight of the target node in the target graph network and the weights of each neighbor node within N hops of the target node, the similarity coefficient between the target node and each neighbor node is calculated.

[0029] The weights between the target node and its neighboring nodes are calculated using the similarity coefficients.

[0030] The fusion field information of the target node is calculated based on the field information of each neighboring node and the weights between the target node and each neighboring node.

[0031] The fused field information of the target node is used as the field information of the metadata included in the corresponding sample features.

[0032] In a second aspect, an electronic device is provided, comprising: a memory and a processor electrically connected to the memory, the memory storing a computer program executable by the processor, the computer program, when executed by the processor, implementing the steps of the method described in the first aspect.

[0033] Thirdly, a computer-readable storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0034] In this embodiment, by acquiring the field information of metadata in the target big data environment, constructing a target graph network based on the field information, and determining the weights of the field information of metadata in the target graph network, a node type discrimination model is trained based on samples and labels determined by the node data corresponding to the nodes in the target graph network, and the nodes in the target graph network are classified by application type based on the type discrimination model. This allows for accurate and efficient automatic classification of metadata, improving the efficiency of data quality management in the big data environment. Attached Figure Description

[0035] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0036] Figure 1 This is a flowchart illustrating the metadata type classification method according to an embodiment of this application.

[0037] Figure 2 This is a network structure diagram of a graph network according to the first embodiment of this application.

[0038] Figure 3 This is a network structure diagram of a graph network according to the second embodiment of this application.

[0039] Figure 4 This is a network structure diagram of a graph network according to the third embodiment of this application.

[0040] Figure 5 This is an example flowchart of the metadata type classification method in an embodiment of this application.

[0041] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. The drawing numbers in this application are only used to distinguish the various steps in the solution and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0043] To address the problems existing in the prior art, embodiments of this application provide a method for classifying metadata types, such as... Figure 1 As shown, it includes the following steps:

[0044] Step 102: Obtain the field information of metadata in the target big data environment;

[0045] Step 104: Construct a target graph network based on the field information, wherein nodes in the target graph network represent metadata, and edges represent field information associated with two corresponding connected metadata.

[0046] Step 106: Determine the weights of the field information of the metadata in the target graph network;

[0047] Step 108: Based on the samples and labels determined by the node data corresponding to the nodes in the target graph network, train the node type discrimination model, wherein the label includes the application type of the target big data environment corresponding to the metadata represented by the node, and the sample features of the sample include the field information of the metadata represented by the node.

[0048] Step 110: Classify the nodes in the target graph network by application type based on the type discrimination model.

[0049] In step 102, the field information of the metadata, also known as the attribute information of the metadata or the metadata information of the metadata, is described. The attribute information of the metadata typically includes the name of the metadata, the description field of the metadata, the creation time of the metadata, the length of the metadata, the data type of the metadata, the name of the table to which the metadata belongs, the description field of the table to which the metadata belongs, and so on.

[0050] Metadata is extracted from the corresponding data in the big data environment to obtain an application-level metadata set. Then, the extracted metadata is preprocessed, that is, the field information of each metadata is extracted from the metadata set to obtain the metadata and its corresponding field information.

[0051] Depending on the application scenario, the metadata and field information of the data extracted from the corresponding big data environment will also be different. For example, in the field of mobile communication technology, the description fields of the corresponding field information may involve data traffic, data plans, voice, 4G, 5G and other related content.

[0052] In one embodiment, after obtaining the field information of the metadata, the relevant field information can be preprocessed, including at least one of the following: 1. If the field information of the metadata is empty, the field information of the table to which the metadata belongs is used to populate the field information of the metadata. The table contains metadata, and the attribute information of the empty metadata is represented by the attribute information of the table; 2. The descriptions of the table and the metadata can be divided using Chinese word segmentation, and stop words, etc., can be removed; 3. Some metadata used for indexing is removed. This type of metadata is invalid data and does not help with metadata management, so it can be removed.

[0053] In step 104, a target graph network is constructed based on the field information of the acquired metadata. Different methods can be used to construct the network depending on the different ways in which the metadata of the nodes are associated.

[0054] Optionally, the field information includes the creation time of the metadata, the description field of the metadata, the name of the table to which the metadata belongs, and the description field of the table to which the metadata belongs. Constructing a target graph network based on the field information includes: determining a first type of metadata that has contextual reference relationships with other metadata based on the creation time of the metadata, the description field of the metadata, the name of the table to which the metadata belongs, and the description field of the table to which the metadata belongs; and constructing a first type of graph network using each first type of metadata as a node and the description field of the table to which each first type of metadata belongs as an edge.

[0055] After constructing the network, the weights of each edge of the constructed graph network are obtained according to the number of times relationships are generated between nodes in the target graph network.

[0056] In this embodiment, this type of graph network determines the association between metadata based on the daily reference relationships of metadata and constructs the target graph network.

[0057] The structure is built by analyzing the generation method of the description fields in the metadata within the field information. For example, based on the creation time of each metadata record, its corresponding description field, and the name and description field of the table to which it belongs, other metadata that have reference relationships with the corresponding metadata are identified sequentially. Following a bottom-up, coarse-grained to fine-grained approach using description fields, metadata with contextual reference relationships are arranged as nodes to form a tree structure. For example... Figure 3 As shown, Figure 3 In this graph network, metadata represents the nodes, and tables represent the edges. The tables are connected through internal fields—fields not used by the upper-layer application but solely responsible for generating the fields used by the application. This context-reference-based approach allows for the extraction of deeper-level metadata relationships.

[0058] In this type of graph network, each node represents metadata, which corresponds to different field information. The field information protected by the metadata is not limited to the field information used to create the graph network. For example, in addition to the creation time of the metadata, the description field, the name of the table to which the metadata belongs, and the description field of the table, it can also include at least one of the following: the name of the metadata, the length of the metadata, and the data type of the metadata.

[0059] Optionally, the field information includes the data type of metadata. Constructing a target graph network based on the field information includes: obtaining second-type metadata about the target topic in the target script according to the data type of the metadata, wherein the target script is used to record analysis data or application data of the target topic; constructing a fully connected network of each second-type metadata corresponding to the target script, using each second-type metadata as a node; and constructing a second-type graph network using the connections between the same second-type metadata that appear simultaneously in the fully connected networks corresponding to different scripts as edges.

[0060] In this embodiment, the graph network of this type is constructed by determining the relationships between metadata based on the script creation method, and then building the target graph network.

[0061] For example, a graph network of the acquired metadata can be constructed using SQL scripts. Based on the data type of the metadata, such as the metadata corresponding to the Select or Join statements in the SQL scripts used to record analytical or application data for the target topic, metadata related to that target topic is extracted from the Select or Join statements. A fully connected internal network corresponding to that topic is then constructed according to the extracted metadata from each topic. Furthermore, if the metadata of the target topic in the Select or Join statements of the target SQL script also relates to other topics, such as the target metadata appearing in different topics recorded in different scripts, edges will be established between the target metadata appearing in different topics. A topic is an analysis performed on a specific domain or target; different topics have different analytical directions, such as 4G daily active user analysis, 5G daily active user analysis, etc.

[0062] like Figure 4 As shown, the Select statement in the topic analysis recorded in the target script contains metadata 1, metadata 2, ..., metadata n. A fully connected internal network is established between the metadata of topic 1 corresponding to the target script. Furthermore, metadata 3 in topic 1 also relates to topic 2, and metadata 4 in topic 1 also relates to topic n. Therefore, an edge is constructed between the corresponding metadata 3 in topic 1 and topic 2, and an edge is constructed between the corresponding metadata 4 in topic 1 and topic n.

[0063] This approach addresses situations where data belonging to similar application types is grouped into a single table, rather than generating new data. For example, metadata about data traffic and voice call duration might not be directly related, but since they belong to similar application types, grouping them together during thematic analysis can help describe customer profiles. This allows for an accurate and comprehensive capture of the relationships between metadata in such cases.

[0064] The graph network of the type described above will exhibit different topics at different sampling times t. Therefore, the constructed graph network will have different network forms at different times t. Representing different time points t i The following describes the corresponding graph network of this type. At this point, we can consider setting corresponding network weights for the graph networks built at different time points, and then weighting them to obtain the overall corresponding graph network.

[0065] By incorporating time-series considerations, we can follow the impact of events or periodic updates to dynamically and accurately capture the relationships in metadata, making the constructed network more realistically reflect the direct relationships between data.

[0066] After constructing the network, the weights of each edge in the constructed graph network are obtained according to the frequency of relationships generated between nodes in the target graph network. For example, the weights of the edges for metadata 1 and metadata 2 are set based on the frequency with which metadata 1 and metadata 2 appear simultaneously in two SQL scripts.

[0067] In this type of graph network, each node represents metadata, which corresponds to different field information. The field information protected by the metadata is not limited to the field information used to create the graph network. For example, in addition to the data type of the metadata, it can also include at least one of the following: the name of the metadata, the creation time, the length of the metadata, the description field of the metadata, the name of the table to which the metadata belongs, and the description field of the table.

[0068] Optionally, the field information includes the name or description field of metadata. Constructing a target graph network based on the field information includes: identifying a third type of metadata that has the same metadata name or description field; and constructing a third type graph network with each third type of metadata as a node and the same metadata name or description field corresponding to each third type of metadata as an edge.

[0069] In this embodiment, this type of graph network determines the relationships between metadata based on the name or description fields of the metadata and constructs the target graph network.

[0070] In this type of graph network, nodes represent metadata, while edges are formed by segmenting the names or field descriptions of the metadata. If different metadata nodes have the same segmentation for their names or field descriptions, an edge is created between the two corresponding metadata nodes.

[0071] like Figure 4 As shown, if metadata 1 and metadata 2 both contain the keyword "package" in their names or field descriptions, then metadata 1 and metadata 2 will form an edge. Similarly, if metadata 2 and metadata 3 both contain the keyword "4G" in their names or field descriptions, then metadata 2 and metadata 3 will form an edge.

[0072] After constructing the network, the weights of each edge in the constructed graph network are obtained according to the number of times relationships are generated between nodes in the target graph network. For example, in this embodiment, the weights are set based on the number of times the same word segment exists in the description fields of different metadata. If both metadata 1 and metadata 2 exist in the description fields of "revenue" and "traffic", then the edge weights of metadata 1 and metadata 2 are 2.

[0073] In this type of graph network, each node represents metadata, which corresponds to different field information. The field information protected by the metadata is not limited to the field information used to create the graph network. For example, in addition to the name and description fields of the metadata, it can also include at least one of the following: the creation time of the metadata, the length of the metadata, the name of the table to which the metadata belongs, the description field of the table, and the data type of the metadata.

[0074] After constructing the target type of graph network, the constructed graph network can be preprocessed to remove the incomplete nodes and isolated nodes.

[0075] In step 106, determining the weights of the field information of the metadata in the target graph network includes: determining the ranking value of the target field information corresponding to the target metadata; and performing latent factor encoding on the ranking value to obtain the weights of the target field information.

[0076] Based on the field information of the acquired or preprocessed metadata, the field information of each node in the constructed target graph network is sorted by computer to obtain the sorting value of the target field information. For example, the sorting value corresponding to the creation time of the field information is: 2015-05, 2015-06, 2021-07, etc. All metadata in the target graph network is sorted in ascending order of creation time, and then the sorting value is used for latent factor encoding. The encoding method can be to assign a latent factor, such as 128 bits, 512 bits, or 1024 bits, to a sorting value. That is, the weight of the field information of each node in the target graph network.

[0077] For the same type of field information, the latent factor encoding values ​​will differ under different metadata. For example, the latent factor encoding values ​​for the description field "flow" in metadata 1 are different from those for the description field "flow" in metadata 2.

[0078] Therefore, the latent factors of the nodes in the entire target graph network are the dimensions of a three-dimensional matrix [N1, N2, W1]. Here, N1 represents the number of nodes in the target graph network, i.e., the amount of metadata, and N2 represents the amount of field information contained in the metadata corresponding to the target node. For example, if the number of nodes in the target graph network is 10, and the field information of the metadata represented by the target node includes the name of the metadata, creation time, length of the metadata, description field of the metadata, name of the table to which the metadata belongs, and description field of the table, then the amount of field information contained in the metadata is 6. W1 represents the number of bits in the latent factor encoding value. Therefore, with 10 nodes, 6 bits of field information per node, and each field information having a 128-bit latent factor encoding value, the latent factor dimension of the nodes in this target graph network is [10, 10, 128].

[0079] In one embodiment, the target graph network includes only a single type of graph network. In step 108, the node type discrimination model is trained based on the samples and labels determined by the node data corresponding to the nodes in the target graph network, including: training the type discrimination model based on the samples and labels determined by the node data corresponding to the nodes in the single type of graph network until the weights of the field information corresponding to the nodes in the single type of graph network converge.

[0080] When training a type discrimination model, a semi-supervised learning approach can be used. In this approach, the application type of some nodes' metadata in the target graph network is known, while the application type of the remaining nodes' metadata is unknown. The latent factors of each node in the target graph network are trained using samples with known application types, and the parameters of the entire network converge, including the weights of the field information corresponding to the nodes in the graph network. This yields the final type discrimination model and reveals the application type of the unknown nodes.

[0081] In this embodiment, the type discrimination model can also be trained using a fully supervised learning approach. By using nodes with known metadata application types as samples for training, the parameters of the entire network eventually converge, resulting in the final type discrimination model. Subsequent input of field information from target metadata with unknown application types into this type discrimination model allows the determination of the target metadata's application type.

[0082] The application type of metadata is the application type of the target big data environment to which the metadata corresponds. For example, the application type is related to the actual scenario. In the field of mobile communications, it can be divided into user type, package type, and fee type.

[0083] Metadata for known application types can be obtained in advance through manual analysis using multiple types of field information corresponding to the metadata, such as the metadata name, creation time, and description field.

[0084] By using the field information of all metadata corresponding to all nodes of the target graph network as samples and the application type of some known metadata as labels, all nodes are input into the type discrimination model constructed for the target graph network at once for learning, until the weights of the field information corresponding to the nodes in the graph network converge.

[0085] In another embodiment, the target graph network includes multiple types of subgraph networks. In step 108, based on the samples and labels determined by the node data corresponding to the nodes in the target graph network, the node type discrimination model is trained, including: determining the network weights and subtype discrimination models corresponding to each type of subgraph network; generating a type graph network containing each type of subgraph network according to the network weights, wherein the type discrimination model corresponding to each type of subgraph network includes each subtype discrimination model corresponding to each type of subgraph network; training the corresponding subtype discrimination model based on the samples and labels determined by the node data corresponding to the nodes in each type of subgraph network until the weights of the field information corresponding to the nodes in each type of subgraph network and each network weight converge, thereby obtaining the type discrimination model.

[0086] In this embodiment, a graph network can be composed of multiple subgraph networks of different types. For example, a graph network can be composed of two different types of subgraph networks, or three different types of subgraph networks, and each subgraph network can be constructed in one of the three different ways mentioned above.

[0087] By constructing a network structure that integrates multiple types of sub-networks based on different types of relationships between metadata, the network can more accurately capture relationships such as lineage and correlation, greatly improving the network's perception of metadata.

[0088] The initial values ​​of the network weights for different types of subgraph networks can be obtained by assigning a value of, for example, 128 bits or 256 bits through latent factor encoding.

[0089] Based on the construction type of each subgraph network and the metadata application category of the nodes, a latent factor dimension [N3, N4, W2] is assigned to each subgraph network, where N3 represents the number of subgraph networks of different types, N4 represents the number of metadata application types in each subgraph network, and W2 represents the number of bits in the latent factor encoding value corresponding to the subgraph network. Therefore, when there are 3 subgraph networks, 3 metadata application types in each subgraph network, and 128-bit encoding values ​​for the latent factors of each subgraph network, the latent factor dimension of the entire graph network is [3, 3, 128].

[0090] In this embodiment, the subtype discrimination model can be trained using a semi-supervised learning approach. When training in a semi-supervised learning mode, the application type of the metadata for some nodes in the target graph network is known, while the application type of the metadata for the remaining nodes is unknown. The latent factors of each node in the target graph network are trained using samples with known application types.

[0091] A subgraph network is used to train a neural network corresponding to a subtype discrimination model. The neural networks corresponding to multiple subgraph networks are then mapped to a higher-order space to obtain the neural network corresponding to the entire graph network. The neural network corresponding to the entire graph network is trained until the parameters of the entire network converge, including the convergence of the weights of the field information corresponding to the nodes in each subgraph network and the convergence of the weights of each subnetwork, resulting in the final type discrimination model and the application type of the unknown node. Alternatively, a fully supervised learning approach can be used to obtain the corresponding type discrimination model, and this model can be used to determine the application type based on the field information of the target metadata with unknown application types.

[0092] To improve the recognition accuracy of the trained type discrimination model, before training the node type discrimination model, the method further includes: calculating the similarity coefficient between the target node and each neighbor node based on the weight of the target node in the target graph network and the weights of each neighbor node within N hops of the target node; calculating the weight between the target node and each neighbor node using the similarity coefficient; calculating the fused field information of the target node based on the field information of each neighbor node and the weight between the target node and each neighbor node; and using the fused field information of the target node as the field information of the metadata included in the corresponding sample features.

[0093] In this embodiment, the field information of each node in the target graph network is comprehensively calculated based on the neighbor information of the nodes in the target graph network. For node i in the target graph network, and its N hops, such as neighbor node j within two hops, j∈N i Learnable latent causes were generated according to different types of field information. and j∈N. For node i, calculate its neighbors N step by step. i similarity coefficient e ij Where A(...) is a mapping function that maps high-dimensional hidden factors based on field correlation. and Mapped to the real space β ij .

[0094]

[0095] in, The calculation is as follows:

[0096]

[0097] The weights of node i and its neighbors can be calculated using the similarity coefficient:

[0098]

[0099] Then, the field information of the metadata represented by neighboring nodes is aggregated. The weights α of different nodes obtained through the above steps are... ij The fusion field information of computing node i is aggregated. The calculation method is shown in the following formula, where... The field information is the metadata of the nodes within two hops of node i. Let A(...) be the fused field information for node i, and let A(...) be the mapping function.

[0100]

[0101] Based on the neighbor nodes within N hops, the hidden factors of the target node and the network hidden factors of each subgraph network are combined to calculate the field information of the nodes in the target graph network. The hidden factors generated by the field information are then used as weights for the type discrimination model, which can improve the recognition accuracy of the finally trained type discrimination model.

[0102] In this embodiment, by acquiring the field information of metadata in the target big data environment, constructing a target graph network based on the field information, and determining the weights of the field information of metadata in the target graph network, a node type discrimination model is trained based on samples and labels determined by the node data corresponding to the nodes in the target graph network, and the nodes in the target graph network are classified by application type based on the type discrimination model. This allows for accurate and efficient automatic classification of metadata, improving the efficiency of data quality management in the big data environment.

[0103] Figure 5 This is an example flowchart of the metadata type classification method according to an embodiment of this application, such as... Figure 5 As shown, firstly, a metadata set is extracted from the data in the target big data environment. Then, the metadata is preprocessed to obtain preprocessed metadata. Metadata field information, i.e., metadata of metadata, is extracted from the metadata. The field information of the metadata can be preprocessed, and then three types of networks can be constructed based on the preprocessed metadata and field information, resulting in Network 1, Network 2, and Network 3. A complex network is constructed based on these networks, and then preprocessed. The complex network can be constructed based on the metadata and field information acquired at different timest, and complex network management is performed according to the corresponding time series, including configuring different weights for the complex network at different time nodes and performing weighted processing to obtain the corresponding metadata complex network.

[0104] The field information of the metadata represented by each node in each of the constructed Networks 1, 2, and 3 is latently factor-encoded to determine the weights of each node. Latent factor encoding is also performed based on the proportion of each network within the complex network to determine its weights. The complex network is then trained until its parameters converge, resulting in the final type discrimination model, which is then stored. When training using semi-supervised learning, the type discrimination results for metadata with unknown types can be directly output at the end of the training. Subsequent acquisition of metadata and field information can be directly input into the stored type discrimination model to classify the metadata into application types, yielding the corresponding discrimination results.

[0105] Optionally, embodiments of this application also provide an electronic device. Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application.

[0106] like Figure 6 As shown, the electronic device 2000 includes a memory 2200 and a processor 2400 electrically connected to the memory 2200. The memory 2200 stores a computer program that can be run on the processor 2400. When the computer program is executed by the processor, it implements the various processes of any of the above-described metadata type classification method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0107] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of any of the above-described metadata type classification method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0108] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0110] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for classifying metadata types, characterized in that, include: The field information of metadata in the target big data environment is obtained. The field information includes the data type of the metadata. The data type of the metadata is the metadata of the Select or join statement of the corresponding SQL script. The metadata related to the target topic is extracted from the Select or join statement in the SQL script used to record the analysis data or application data of the target topic, and the fully connected internal network corresponding to the target topic is constructed. The direction of the target topic analysis is 4G daily active user analysis and 5G daily active user analysis. A target graph network is constructed based on the field information, wherein nodes in the target graph network represent metadata, and edges represent the field information associated with two corresponding connected metadata. Determine the weights of field information in the target graph network; Based on samples and labels determined by node data corresponding to nodes in the target graph network, a node type discrimination model is trained, wherein the label includes the application type of the target big data environment corresponding to the metadata represented by the node, and the sample features of the sample include the field information of the metadata represented by the node. Based on the type discrimination model, the nodes in the target graph network are classified according to their application types. The field information includes the creation time of the metadata, the description field of the metadata, the name of the table to which the metadata belongs, and the description field of the table to which the metadata belongs. Constructing a target graph network based on the field information includes: determining a first type of metadata that has a contextual reference relationship with other metadata based on the creation time of the metadata, the description field of the metadata, the name of the table to which the metadata belongs, and the description field of the table to which the metadata belongs; and constructing a first type of graph network with each first type of metadata as a node and the description field of the table to which each first type of metadata belongs as an edge.

2. The method as described in claim 1, characterized in that, Constructing a target graph network based on the field information includes: Based on the data type of the metadata, obtain the second type of metadata about the target topic in the target script, wherein the target script is used to record the analysis data or application data of the target topic; Using each type of second-class metadata as a node, construct a fully connected network of each type of second-class metadata corresponding to the target script; A second type of graph network is constructed using the connections between identical second metadata that appear simultaneously in the fully connected networks corresponding to different scripts as edges.

3. The method as described in claim 1, characterized in that, The field information includes the name or description field of metadata. Constructing the target graph network based on the field information includes: Identify a third type of metadata that has the same metadata name or description field; A third-type graph network is constructed using each third-type metadata as a node and the same metadata name or description field corresponding to each third-type metadata as an edge.

4. The method according to any one of claims 1 to 3, characterized in that, Determine the weights of field information in the metadata within the target graph network, including: Determine the sorting value of the target field information corresponding to the target metadata; The weights of the target field information are obtained by performing latent factor encoding on the sorted values.

5. The method according to any one of claims 1 to 3, characterized in that, The target graph network includes only a single type of graph network. Based on the samples and labels determined by the node data corresponding to the nodes in the target graph network, a node type discrimination model is trained, including: Based on the samples and labels determined by the node data corresponding to the nodes in the graph network of the single type, the type discrimination model is trained until the weights of the field information corresponding to the nodes in the graph network of the single type converge.

6. The method according to any one of claims 1 to 3, characterized in that, The target graph network includes multiple types of subgraph networks. Based on the samples and labels determined by the node data corresponding to the nodes in the target graph network, a node type discrimination model is trained, including: Determine the network weights and subtype discrimination models for each type of subgraph network; Based on the network weights corresponding to each type of subgraph network, a type graph network containing each type of subgraph network is generated. The type discrimination model corresponding to the type graph network includes each subtype discrimination model corresponding to each type of subgraph network. Based on the samples and labels determined by the node data corresponding to the nodes in each type of subgraph network, the corresponding subtype discrimination model is trained until the weights of the field information corresponding to the nodes in each type of subgraph network and the weights of each network converge, thus obtaining the type discrimination model.

7. The method as described in claim 1, characterized in that, Before training the node type discrimination model, the following is also included: Based on the weight of the target node in the target graph network and the weights of each neighbor node within N hops of the target node, the similarity coefficient between the target node and each neighbor node is calculated. The weights between the target node and its neighboring nodes are calculated using the similarity coefficients. The fusion field information of the target node is calculated based on the field information of each neighboring node and the weights between the target node and each neighboring node. The fused field information of the target node is used as the field information of the metadata included in the corresponding sample features.

8. An electronic device, characterized in that, include: A memory and a processor electrically connected to the memory, the memory storing a computer program executable by the processor, the computer program, when executed by the processor, implementing the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

Images