A metadata intelligent identification method and device based on semi-supervised learning

Abstract

The application provides a metadata intelligent identification method and device based on semi-supervised learning, and the method comprises the following steps: generating a metadata keyword identifier according to a conditional random field, wherein the metadata keyword identifier is used for extracting metadata features corresponding to any identification; acquiring an initial labeled data set, training the metadata keyword identifier according to the initial labeled data set to generate a metadata classifier; acquiring an unlabeled data set, predicting the unlabeled data set according to the metadata classifier to generate a prediction result; generating an intermediate training data set according to the prediction result; cyclically self-training the metadata keyword identifier and the metadata classifier according to the intermediate training data set to obtain a cyclically self-trained metadata classifier; and identifying metadata intelligently according to the cyclically self-trained metadata classifier. Through the application, the problem of low metadata identification construction efficiency in the related art is solved.

Description

Technical Field

[0001] This invention relates to the field of metadata identification, and in particular to a method and apparatus for intelligent metadata identification based on semi-supervised learning. Background Technology

[0002] With the rapid development of the information age, data platforms and IoT management platforms have taken initial shape. However, this informatization has generated a massive amount of complex data, posing significant challenges to power grid data management technology. On the one hand, with the emergence of various complex application scenarios, power grid data is growing exponentially. The sheer volume of data has rendered existing data processing methods inadequate for the demands of these complex scenarios. On the other hand, the data in the data platform originates from various sources, is not only abundant but also diverse in type, and is constantly being updated, making metadata management and maintenance extremely difficult. Currently, metadata management, classification, and labeling are still largely done manually, resulting in low efficiency and compromising the consistency and accuracy of metadata with the data in the data platform. Therefore, existing technologies suffer from low efficiency in metadata labeling and construction. Summary of the Invention

[0003] This invention provides a method and apparatus for intelligent metadata identification based on semi-supervised learning, which at least solves the problem of low efficiency in metadata identification construction in related technologies.

[0004] According to a first aspect of the present invention, a method for intelligent metadata labeling based on semi-supervised learning is provided. The method includes: generating a metadata keyword identifier based on a conditional random field (CRF), wherein the metadata keyword identifier is used to extract metadata features corresponding to any identifier; acquiring an initial labeled dataset, and training the metadata keyword identifier on the initial labeled dataset to generate a metadata classifier; acquiring an unlabeled dataset, and predicting the unlabeled dataset based on the metadata classifier to generate a prediction result, wherein the prediction result includes both the labeled dataset and the unlabeled dataset; generating an intermediate training dataset based on the prediction result; iteratively self-training the metadata keyword identifier and the metadata classifier on the intermediate training dataset to obtain a iteratively self-trained metadata classifier; and intelligently labeling metadata based on the iteratively self-trained metadata classifier.

[0005] Optionally, generating the metadata keyword identifier based on the conditional random field model includes: representing the metadata and the identifiers corresponding to the metadata as an undirected graph; determining an observation sequence and an identifier sequence based on the conditional random field, wherein the observation sequence corresponds to the metadata and the identifier sequence corresponds to the identifiers corresponding to the metadata; determining an objective function and a feature set based on the observation sequence and the identifier sequence, wherein the objective function is used to obtain the identifier sequence with the highest probability corresponding to the observation sequence, and the feature set is a set of metadata features; and generating the metadata keyword identifier based on the objective function and the feature set.

[0006] Optionally, the observation sequence includes a transition feature function, which is determined based on a conditional random field, wherein the transition feature function acts on the edges of an undirected graph and represents the relationship between the previous output state and the current output state.

[0007] Optionally, the step of obtaining the unlabeled dataset and predicting the unlabeled dataset based on the metadata classifier to generate a prediction result, wherein the prediction result includes the labeled dataset and the unlabeled dataset, includes: predicting the unlabeled data based on the metadata features corresponding to any of the identifiers, wherein the unlabeled data is any data in the unlabeled dataset; if the unlabeled data has similar metadata features, then an identifier corresponding to the similar metadata features is generated for the unlabeled data, and the unlabeled data with generated identifiers is stored in the labeled dataset; if the unlabeled data does not have similar metadata features, then no identifier is generated for the unlabeled data, and the unlabeled data without generated identifiers is stored in the unlabeled dataset.

[0008] Optionally, generating an intermediate training dataset based on the prediction result includes generating an intermediate training dataset based on data with high classification feature weights in the unlabeled dataset and the labeled dataset.

[0009] Optionally, the step of iteratively self-training the metadata keyword identifier and the metadata classifier based on the intermediate training dataset to obtain a self-trained metadata classifier includes: training the metadata keyword identifier to generate a metadata classifier based on the intermediate training dataset; predicting the unlabeled dataset based on the metadata classifier to generate prediction results; generating an intermediate training dataset based on the prediction results; and iteratively generating the metadata classifier, generating prediction results, and generating the intermediate training dataset until the metadata classifier converges to obtain a self-trained metadata classifier.

[0010] Optionally, the step of iteratively self-training the metadata keyword identifier and the metadata classifier based on the intermediate training dataset to obtain the iteratively self-trained metadata classifier further includes: determining the classification accuracy based on the labeled dataset in the prediction results; if the classification accuracy does not reach a preset value, iteratively generating the metadata classifier, generating prediction results, and generating the intermediate training dataset; if the classification accuracy reaches a preset value, ending the iterative self-training to obtain the iteratively self-trained metadata classifier.

[0011] According to a second aspect of the present invention, a metadata intelligent labeling device based on semi-supervised learning is also provided. The device includes: a first generation module, configured to generate a metadata keyword labeler based on a conditional random field (CRF), wherein the metadata keyword labeler is used to extract metadata features corresponding to any label; a second generation module, configured to acquire an initial labeled dataset and train the metadata keyword labeler on the initial labeled dataset to generate a metadata classifier; a third generation module, configured to acquire an unlabeled dataset and predict the unlabeled dataset based on the metadata classifier to generate a prediction result, wherein the prediction result includes both labeled and unlabeled datasets; a fourth generation module, configured to generate an intermediate training dataset based on the prediction result; a cyclic self-training module, configured to cyclically self-train the metadata keyword labeler and the metadata classifier on the intermediate training dataset to obtain a cyclically self-trained metadata classifier; and an labeling module, configured to intelligently label metadata based on the cyclically self-trained metadata classifier.

[0012] Optionally, the first generation module includes: a representation unit, used to represent the metadata and the identifiers corresponding to the metadata as an undirected graph; a first determination unit, used to determine an observation sequence and an identifier sequence based on a conditional random field, wherein the observation sequence corresponds to the metadata and the identifier sequence corresponds to the identifiers corresponding to the metadata; a second determination unit, used to determine an objective function and a feature set based on the observation sequence and the identifier sequence, wherein the objective function is used to obtain the identifier sequence with the highest probability corresponding to the observation sequence, and the feature set is a set of metadata features; and a generation unit, used to generate a metadata keyword identifier based on the objective function and the feature set.

[0013] Optionally, the observation sequence includes a transition feature function, which is determined based on a conditional random field, wherein the transition feature function acts on the edges of an undirected graph and represents the relationship between the previous output state and the current output state.

[0014] Optionally, the third generation module includes: a prediction unit, configured to predict unlabeled data based on metadata features corresponding to any of the identifiers, wherein the unlabeled data is any data in the unlabeled dataset; a first storage unit, configured to generate an identifier corresponding to the same metadata feature for the unlabeled data when the unlabeled data has the same metadata feature, and store the unlabeled data with the generated identifier in the labeled dataset; and a second storage unit, configured to not generate an identifier for the unlabeled data when the unlabeled data does not have the same metadata feature, and store the unlabeled data without the generated identifier in the unlabeled dataset.

[0015] Optionally, the fourth generation module includes a generation unit, used to generate an intermediate training dataset based on the data with high classification feature weights in the unlabeled dataset and the labeled dataset.

[0016] Optionally, the recurrent self-training module includes: a first generation unit, used to train the metadata keyword identifier to generate a metadata classifier based on the intermediate training dataset; a second generation unit, used to predict the unlabeled dataset based on the metadata classifier and generate prediction results; a third generation unit, used to generate an intermediate training dataset based on the prediction results; and a obtaining unit, used to repeatedly generate the metadata classifier, generate prediction results, and generate intermediate training datasets until the metadata classifier converges, thereby obtaining the recurrently self-trained metadata classifier.

[0017] Optionally, the obtaining unit includes: a determining submodule, used to determine the classification accuracy based on the labeled dataset in the prediction result; a generating submodule, used to iteratively generate a metadata classifier, generate prediction results, and generate an intermediate training dataset if the classification accuracy does not reach a preset value; and an obtaining submodule, used to end the iterative self-training when the classification accuracy reaches the preset value, and obtain the iteratively self-trained metadata classifier.

[0018] According to a third aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps of any of the above embodiments by running the computer program stored in the memory.

[0019] According to a fourth aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the method steps of any of the above embodiments when running.

[0020] In this embodiment of the invention, a metadata keyword identifier is generated based on a conditional random field; an initial labeled dataset is obtained, and the metadata keyword identifier is trained using the initial labeled dataset to generate a metadata classifier; an unlabeled dataset is obtained, and the metadata classifier is used to predict the unlabeled dataset to generate prediction results; an intermediate training dataset is generated based on the prediction results; the metadata keyword identifier and the metadata classifier are iteratively self-trained using the intermediate training dataset to obtain a iteratively self-trained metadata classifier; and metadata is intelligently labeled using the iteratively self-trained metadata classifier. Because metadata features are extracted through the metadata keyword identifier, the metadata classifier is trained using labeled data, and the metadata keyword identifier and the metadata classifier are iteratively self-trained using both labeled and unlabeled data, the purpose of labeling metadata through the metadata classifier can be achieved, thus improving the efficiency of metadata labeling and solving the problem of low metadata labeling construction efficiency in related technologies.

[0021] In this embodiment of the invention, by representing metadata and the corresponding identifiers as an undirected graph, determining the observation sequence and identifier sequence based on the conditional random field, and determining the objective function and feature set based on the observation sequence and identifier sequence, the purpose of constructing a model and using the conditional random field to achieve metadata identification is realized.

[0022] In this embodiment of the invention, an intermediate training dataset is generated based on data with high classification feature weights in both the unlabeled and labeled datasets. The metadata keyword identifier and metadata classifier are then used for iterative self-training. This utilizes the useful structural information contained in the unlabeled sample data, thereby improving data usability and solving the problem of insufficient labeled samples. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the hardware environment for an optional metadata intelligent identification method based on semi-supervised learning according to an embodiment of the present invention;

[0026] Figure 2This is a flowchart illustrating an optional metadata intelligent identification method based on semi-supervised learning according to an embodiment of the present invention.

[0027] Figure 3 This is a structural block diagram of an optional metadata intelligent identification device based on semi-supervised learning according to an embodiment of the present invention;

[0028] Figure 4 This is a structural block diagram of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] According to a first aspect of the present invention, a method for intelligent metadata identification based on semi-supervised learning is provided. Optionally, in this embodiment, the above-described method for intelligent metadata identification based on semi-supervised learning can be applied to, for example... Figure 1 In the hardware environment shown. For example... Figure 1 As shown, terminal 102 may include memory 104, processor 106, and display 108 (optional component). Terminal 102 can communicate with server 112 via network 110. Server 112 can provide services (such as application services) to the terminal or clients installed on the terminal. Database 114 can be set up on or independently of server 112 to provide data storage services to server 112. In addition, server 112 may run a processing engine 116, which can be used to execute the steps performed by server 112.

[0032] Optionally, terminal 102 may be, but is not limited to, a terminal capable of computing data, such as a mobile terminal (e.g., mobile phone, tablet computer), laptop computer, PC (Personal Computer), etc. The aforementioned network may include, but is not limited to, a wireless network or a wired network. The wireless network includes Bluetooth, Wi-Fi (Wireless Fidelity), and other networks that enable wireless communication. The aforementioned wired network may include, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), and a local area network (LAN). The aforementioned server 112 may include, but is not limited to, any hardware device capable of computing.

[0033] Furthermore, in this embodiment, the aforementioned metadata intelligent identification method based on semi-supervised learning can also be applied to, but is not limited to, independent processing devices with powerful processing capabilities, without the need for data interaction. For example, the processing device can be, but is not limited to, a terminal device with powerful processing capabilities; that is, the various operations in the aforementioned metadata intelligent identification method based on semi-supervised learning can be integrated into a single independent processing device. The above is merely an example, and no limitation is made in this embodiment.

[0034] Optionally, in this embodiment, the above-described metadata intelligent identification method based on semi-supervised learning can be executed by server 112, terminal 102, or jointly by server 112 and terminal 102. Alternatively, the execution of the metadata intelligent identification method based on semi-supervised learning by terminal 102 can also be performed by a client installed on it.

[0035] Taking the application of a semi-supervised learning-based intelligent metadata identification method to the central processing unit as an example, Figure 2 This is a flowchart illustrating an optional metadata intelligent identification method based on semi-supervised learning according to an embodiment of the present invention, as shown below. Figure 2 As shown, the process of this method may include the following steps:

[0036] Step S201: Generate a metadata keyword identifier based on the conditional random field, wherein the metadata keyword identifier is used to extract the metadata features corresponding to any identifier. Optionally, the metadata and the corresponding identifier are respectively mapped to the observation sequence and the identifier sequence in the conditional random field, and the metadata features corresponding to the metadata identifier are extracted.

[0037] Step S202: Obtain the initial labeled dataset, and train the metadata keyword identifier using the initial labeled dataset to generate a metadata classifier. Optionally, the metadata keyword identifier is trained using the initial labeled dataset to generate a metadata classifier, which can generate corresponding identifiers for metadata.

[0038] Step S203: Obtain the unlabeled dataset, predict the unlabeled dataset using the metadata classifier, and generate prediction results, whereby the prediction results include both labeled and unlabeled datasets. Optionally, the metadata classifier is used to predict the data in the unlabeled dataset. If the prediction generates a corresponding label, the corresponding data is stored in the labeled dataset; otherwise, the corresponding data is stored in the unlabeled dataset.

[0039] Step S204: Generate an intermediate training dataset based on the prediction results. Optionally, a portion of data can be selected from the labeled and unlabeled datasets to generate the intermediate training dataset for iterative training of the model.

[0040] Step S205: The metadata keyword identifier and metadata classifier are cyclically self-trained using the intermediate training dataset to obtain a cyclically self-trained metadata classifier. Optionally, the metadata keyword identifier and metadata classifier are automatically cyclically trained using the intermediate training dataset to obtain a converged metadata classifier after training.

[0041] Step S206: Intelligently label the metadata based on the recurrently self-trained metadata classifier. Optionally, use the converged metadata classifier after training to generate corresponding labels for the metadata.

[0042] In this embodiment of the invention, a metadata keyword identifier is generated based on a conditional random field; an initial labeled dataset is obtained, and the metadata keyword identifier is trained using the initial labeled dataset to generate a metadata classifier; an unlabeled dataset is obtained, and the metadata classifier is used to predict the unlabeled dataset to generate prediction results; an intermediate training dataset is generated based on the prediction results; the metadata keyword identifier and the metadata classifier are iteratively self-trained using the intermediate training dataset to obtain a iteratively self-trained metadata classifier; and metadata is intelligently labeled using the iteratively self-trained metadata classifier. Because metadata features are extracted through the metadata keyword identifier, the metadata classifier is trained using labeled data, and the metadata keyword identifier and the metadata classifier are iteratively self-trained using both labeled and unlabeled data, the purpose of labeling metadata through the metadata classifier can be achieved, thus improving the efficiency of metadata labeling and solving the problem of low metadata labeling construction efficiency in related technologies.

[0043] As an optional embodiment, generating a metadata keyword identifier based on a conditional random field model includes: representing metadata and its corresponding identifiers as an undirected graph; determining an observation sequence and an identifier sequence based on the conditional random field, wherein the observation sequence corresponds to metadata and the identifier sequence corresponds to the identifiers corresponding to the metadata; determining an objective function and a feature set based on the observation sequence and the identifier sequence, wherein the objective function is used to obtain the identifier sequence with the highest probability corresponding to the observation sequence, and the feature set is a set of metadata features; and generating a metadata keyword identifier based on the objective function and the feature set.

[0044] Optionally, the metadata and its corresponding identifiers can be represented as an undirected graph G = (E, F), where G represents the undirected graph, E represents the nodes of the undirected graph (corresponding to metadata), and F represents the edges of the undirected graph (corresponding to the identifiers of the metadata). The observation sequence in a conditional random field is defined as U = (u1, u2, u3...u...). n The corresponding metadata defines the identifier sequence in a conditional random field as V = (v1, v2, v3...v...). n The probability of determining the identifier sequence from features and observation sequences, corresponding to the metadata identifier, can be expressed as follows, according to conditional random field theory:

[0045]

[0046] Where λ represents the feature set, k is the number of features, and x k (v i-1 ,v i y(u,i) is the transition feature function of the observation sequence at positions i-1 to i. k (v i ,v i ,u,i) is the state feature function of the observation sequence at position i.

[0047] Assuming the number of features is k, what is the conditional probability P of the identifier sequence? θ (v|u) is:

[0048]

[0049] Where, λ k and μ k Z(u) represents the feature weights for different features. Z(u) is a normalization factor independent of the observed values, which acts on the entire sequence and serves as a normalization factor.

[0050]

[0051] The objective function is to label the original data, i.e., given the observation sequence, find the label sequence with the highest probability.

[0052]

[0053]

[0054] in, For the identifier sequence with the highest probability, P λ (v|u) is the conditional probability of the maximum probability identifier sequence, and the feature set λ is estimated using the inductive iterative scaling method.

[0055] In this embodiment of the invention, by representing metadata and the corresponding identifiers as an undirected graph, determining the observation sequence and identifier sequence based on the conditional random field, and determining the objective function and feature set based on the observation sequence and identifier sequence, the purpose of constructing a model and using the conditional random field to achieve metadata identification is realized.

[0056] As an optional embodiment, the observation sequence includes a transition feature function, which is determined based on a conditional random field (CRF). The transition feature function acts on the edges of an undirected graph and represents the relationship between the previous output state and the current output state. Optionally, according to CRF theory, the observation sequence includes a transition feature function x. k (v i-1 ,v i The expression (u,i) acts on the edges of an undirected graph, representing the relationship between the previous output state and the current output state, i.e., the relationship between the identifier at position i-1 and the identifier at position i. In this embodiment of the invention, the transition feature function of the observed sequence is determined by a conditional random field, and the relationship between the metadata identifiers of the sequence's hidden state is considered in the model.

[0057] As an optional embodiment, an unlabeled dataset is obtained, and predictions are made on the unlabeled dataset using a metadata classifier to generate prediction results. The prediction results include both labeled and unlabeled datasets. This involves: predicting unlabeled data based on metadata features corresponding to any identifier, where unlabeled data is any data point in the unlabeled dataset; if the unlabeled data has similar metadata features, an identifier corresponding to that feature is generated for the unlabeled data, and the unlabeled data with the generated identifier is stored in the labeled dataset; if the unlabeled data does not have similar metadata features, no identifier is generated for the unlabeled data, and the unlabeled data without an identifier is stored in the unlabeled dataset. This yields prediction results containing both labeled and unlabeled datasets.

[0058] Optionally, a metadata classifier is used to predict the unlabeled dataset. The metadata classifier predicts based on metadata features in the feature set. Specifically, for any unlabeled data in the unlabeled dataset, if the unlabeled data has similar metadata features, an identifier corresponding to those features is generated for the unlabeled data, and the unlabeled data with the generated identifier is stored in the labeled dataset. If the unlabeled data does not have similar metadata features, no identifier is generated for the unlabeled data, and the unlabeled data without an identifier is stored in the unlabeled dataset. In this embodiment of the invention, determining the corresponding identifier based on metadata features achieves intelligent and efficient metadata identification.

[0059] As an optional embodiment, generating an intermediate training dataset based on the prediction results includes: generating an intermediate training dataset based on data with high classification feature weights from both the unlabeled and labeled datasets. Optionally, data with high classification feature weights from both the unlabeled and labeled datasets in the prediction results are selected to generate the intermediate training dataset, and the model is further trained using the intermediate training dataset. In this embodiment of the invention, generating an intermediate training dataset based on data with high classification feature weights from both the unlabeled and labeled datasets, and using the intermediate training dataset for iterative self-training of the metadata keyword identifier and metadata classifier, utilizes the useful structural information contained in the unlabeled sample data, thereby improving data usability and solving the problem of insufficient labeled samples.

[0060] As an optional embodiment, the metadata keyword identifier and metadata classifier are iteratively self-trained based on the intermediate training dataset to obtain the iteratively self-trained metadata classifier. This includes: training the metadata keyword identifier based on the intermediate training dataset to generate the metadata classifier; predicting the unlabeled dataset based on the metadata classifier to generate prediction results; generating the intermediate training dataset based on the prediction results; iteratively generating the metadata classifier, generating prediction results, and generating the intermediate training dataset until the metadata classifier converges to obtain the iteratively self-trained metadata classifier.

[0061] Optionally, the model is automatically and iteratively trained using an intermediate training dataset. This process involves repeatedly training a metadata keyword identifier based on the intermediate training dataset to generate a metadata classifier; predicting the unlabeled dataset using the metadata classifier to generate prediction results; generating an intermediate training dataset based on the prediction results; and training the metadata keyword identifier based on the intermediate training dataset to generate a metadata classifier again until the metadata classifier converges, resulting in a recurrently self-trained metadata classifier. In this embodiment of the invention, recurrent self-training allows the model to learn more metadata features, achieving better classification results.

[0062] As an optional embodiment, the process of iteratively self-training the metadata keyword identifier and metadata classifier based on the intermediate training dataset to obtain the iteratively self-trained metadata classifier further includes: determining the classification accuracy based on the labeled dataset in the prediction results; if the classification accuracy does not reach a preset value, iteratively generating the metadata classifier, generating prediction results, and generating the intermediate training dataset; if the classification accuracy reaches the preset value, ending the iterative self-training, and obtaining the iteratively self-trained metadata classifier. Optionally, the classification accuracy of the prediction data is calculated; if the classification accuracy does not reach a preset value, the model continues to undergo iterative self-training; if the classification accuracy reaches the preset value, the iterative self-training stops. In this embodiment of the invention, determining when to stop the iterative self-training of the model based on the classification accuracy, compared to the model convergence condition, can yield a model with better classification performance.

[0063] According to another aspect of the present invention, a metadata intelligent identification device based on semi-supervised learning is also provided for implementing the above-described metadata intelligent identification method based on semi-supervised learning. Figure 3 This is a structural block diagram of an optional metadata intelligent identification device based on semi-supervised learning according to an embodiment of the present invention, such as... Figure 3 As shown, the device may include: a first generation module 301, used to generate a metadata keyword identifier based on a conditional random field, wherein the metadata keyword identifier is used to extract metadata features corresponding to any identifier; a second generation module 302, used to acquire an initial labeled dataset, and train the metadata keyword identifier based on the initial labeled dataset to generate a metadata classifier; a third generation module 303, used to acquire an unlabeled dataset, predict the unlabeled dataset based on the metadata classifier, and generate a prediction result, wherein the prediction result includes the labeled dataset and the unlabeled dataset; a fourth generation module 304, used to generate an intermediate training dataset based on the prediction result; a cyclic self-training module 305, used to cyclically self-train the metadata keyword identifier and the metadata classifier based on the intermediate training dataset to obtain a cyclically self-trained metadata classifier; and an identifier module 306, used to intelligently identify metadata based on the cyclically self-trained metadata classifier.

[0064] It should be noted that the first generation module 301 in this embodiment can be used to execute the above step S201, the second generation module 302 in this embodiment can be used to execute the above step S202, the third generation module 303 in this embodiment can be used to execute the above step S203, the fourth generation module 304 in this embodiment can be used to execute the above step S204, the cyclic self-training module 305 in this embodiment can be used to execute the above step S205, and the identification module 306 in this embodiment can be used to execute the above step S206.

[0065] Through the above modules, metadata features are extracted based on the metadata keyword identifier. The metadata keyword identifier is trained using labeled data to generate a metadata classifier. The metadata keyword identifier and the metadata classifier are then repeatedly trained using a combination of labeled and unlabeled data. This achieves the goal of labeling metadata through the metadata classifier, thereby improving the efficiency of metadata labeling and solving the problem of low efficiency in metadata identifier construction in related technologies.

[0066] As an optional embodiment, the first generation module includes: a representation unit, used to represent metadata and the identifiers corresponding to the metadata as an undirected graph; a first determination unit, used to determine an observation sequence and an identifier sequence based on a conditional random field, wherein the observation sequence corresponds to the metadata and the identifier sequence corresponds to the identifiers corresponding to the metadata; a second determination unit, used to determine an objective function and a feature set based on the observation sequence and the identifier sequence, wherein the objective function is used to obtain the identifier sequence with the highest probability corresponding to the observation sequence, and the feature set is a set of metadata features; and a generation unit, used to generate a metadata keyword identifier based on the objective function and the feature set.

[0067] As an optional embodiment, the observation sequence includes a transition feature function, which is determined based on a conditional random field. The transition feature function acts on the edges of the undirected graph and represents the relationship between the previous output state and the current output state.

[0068] As an optional embodiment, the third generation module includes: a prediction unit, configured to predict unlabeled data based on metadata features corresponding to any identifier, wherein the unlabeled data is any data in the unlabeled dataset; a first storage unit, configured to generate an identifier corresponding to the same metadata feature for the unlabeled data when the unlabeled data has the same metadata feature, and store the unlabeled data with generated identifiers into the labeled dataset; and a second storage unit, configured to not generate an identifier for the unlabeled data when the unlabeled data does not have the same metadata feature, and store the unlabeled data without generated identifiers into the unlabeled dataset.

[0069] As an optional embodiment, the fourth generation module includes: a generation unit for generating an intermediate training dataset based on data with high classification feature weights in the unlabeled dataset and the labeled dataset.

[0070] As an optional embodiment, the recurrent self-training module includes: a first generation unit, used to train a metadata keyword identifier based on an intermediate training dataset to generate a metadata classifier; a second generation unit, used to predict the unlabeled dataset based on the metadata classifier and generate prediction results; a third generation unit, used to generate an intermediate training dataset based on the prediction results; and a obtaining unit, used to recurrently generate the metadata classifier, generate prediction results, and generate an intermediate training dataset until the metadata classifier converges, thereby obtaining the recurrently self-trained metadata classifier.

[0071] As an optional embodiment, the obtaining unit includes: a determining submodule, used to determine the classification accuracy based on the labeled dataset in the prediction results; a generating submodule, used to iteratively generate a metadata classifier, generate prediction results, and generate an intermediate training dataset if the classification accuracy does not reach a preset value; and an obtaining submodule, used to end the iterative self-training when the classification accuracy reaches the preset value, and obtain the iteratively self-trained metadata classifier.

[0072] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0073] According to another aspect of the present invention, an electronic device for implementing the above-described semi-supervised learning-based metadata intelligent identification method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0074] Figure 4 This is a structural block diagram of an optional electronic device according to an embodiment of the present invention, such as... Figure 4 As shown, the system includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404. The processor 401, communication interface 402, and memory 403 communicate with each other via the communication bus 404. The memory 403 stores computer programs. When the processor 401 executes the computer program stored in the memory 403, it performs the following steps:

[0075] A metadata keyword identifier is generated based on a conditional random field (CRF), whereby the metadata keyword identifier is used to extract metadata features corresponding to any identifier. An initial labeled dataset is obtained, and the metadata keyword identifier is trained on the initial labeled dataset to generate a metadata classifier. An unlabeled dataset is obtained, and the metadata classifier is used to predict the unlabeled dataset to generate prediction results, whereby the prediction results include both labeled and unlabeled datasets. An intermediate training dataset is generated based on the prediction results. The metadata keyword identifier and metadata classifier are iteratively self-trained on the intermediate training dataset to obtain a recursively self-trained metadata classifier. The metadata is intelligently identified based on the recursively self-trained metadata classifier.

[0076] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0077] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0078] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0079] As an example, such as Figure 4 As shown, the memory 403 may include, but is not limited to, the first generation module 301, the second generation module 302, the third generation module 303, the fourth generation module 304, the cyclic self-training module 305, and the identification module 306 in the aforementioned semi-supervised learning-based metadata intelligent identification device. Furthermore, it may include, but is not limited to, other module units in the aforementioned semi-supervised learning-based metadata intelligent identification device, which will not be elaborated upon in this example.

[0080] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0081] In addition, the aforementioned electronic device also includes a display for displaying the results of metadata intelligent identification based on semi-supervised learning.

[0082] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0083] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. The device implementing the above-mentioned metadata intelligent identification method based on semi-supervised learning can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, handheld computer, mobile internet device (MID), PAD, etc. Figure 4 This does not limit the structure of the aforementioned electronic devices. For example, the terminal device may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0084] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0085] According to another aspect of the present invention, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute program code for a metadata intelligent identification method based on semi-supervised learning.

[0086] Optionally, in this embodiment, the storage medium may be located on at least one of the multiple network devices in the network shown in the above embodiment.

[0087] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0088] A metadata keyword identifier is generated based on a conditional random field (CRF), whereby the metadata keyword identifier is used to extract metadata features corresponding to any identifier. An initial labeled dataset is obtained, and the metadata keyword identifier is trained on this dataset to generate a metadata classifier. An unlabeled dataset is obtained, and the metadata classifier is used to predict the results on the unlabeled dataset, generating prediction results that include both labeled and unlabeled datasets. An intermediate training dataset is generated based on the prediction results. The metadata keyword identifier and metadata classifier are iteratively self-trained on the intermediate training dataset to obtain a self-trained metadata classifier. Finally, metadata is intelligently identified based on the self-trained metadata classifier.

[0089] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0090] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0091] According to another aspect of the present invention, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps of the metadata intelligent identification method based on semi-supervised learning in any of the above embodiments.

[0092] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0093] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the metadata intelligent identification method based on semi-supervised learning in the various embodiments of the present invention.

[0094] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] In the several embodiments provided by this invention, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0097] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0098] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent metadata identification based on semi-supervised learning, characterized in that, The method includes: A metadata keyword identifier is generated based on a conditional random field, wherein the metadata keyword identifier is used to extract metadata features corresponding to any identifier; Obtain an initial labeled dataset, and train the metadata keyword identifier to generate a metadata classifier based on the initial labeled dataset; Obtain an unlabeled dataset, predict the unlabeled dataset based on the metadata classifier, and generate a prediction result, wherein the prediction result includes labeled datasets and unlabeled datasets; An intermediate training dataset is generated based on the prediction results; The metadata keyword identifier and the metadata classifier are cyclically self-trained based on the intermediate training dataset to obtain the cyclically self-trained metadata classifier. The metadata is intelligently identified based on the recurrently self-trained metadata classifier. The metadata keyword identifier generated based on the conditional random field includes: The metadata and the corresponding identifiers are represented as an undirected graph. An observation sequence and an identifier sequence are determined based on a conditional random field, wherein the observation sequence corresponds to the metadata and the identifier sequence corresponds to the identifier corresponding to the metadata; An objective function and a feature set are determined based on the observation sequence and the identifier sequence, wherein the objective function is used to obtain the identifier sequence with the highest probability corresponding to the observation sequence, and the feature set is a set of metadata features; Generate a metadata keyword identifier based on the objective function and the feature set.

2. The metadata intelligent identification method based on semi-supervised learning according to claim 1, characterized in that, The observation sequence includes a transition feature function, which is determined based on a conditional random field. The transition feature function acts on the edges of an undirected graph and represents the relationship between the previous output state and the current output state.

3. The metadata intelligent identification method based on semi-supervised learning according to claim 1, characterized in that, The step of obtaining the unlabeled dataset involves predicting the unlabeled dataset based on the metadata classifier and generating a prediction result, wherein the prediction result includes both labeled and unlabeled datasets. Predict unlabeled data based on the metadata features corresponding to any of the identifiers, wherein the unlabeled data is any data in the unlabeled dataset; If the unlabeled data has the same type of metadata features, then generate an identifier corresponding to the same type of metadata features for the unlabeled data, and store the unlabeled data with the generated identifier into the labeled dataset; If the unlabeled data does not have the same type of metadata features, then no identifier will be generated for the unlabeled data, and the unlabeled data without generated identifiers will be stored in the unlabeled dataset.

4. The metadata intelligent identification method based on semi-supervised learning according to claim 3, characterized in that, The step of generating the intermediate training dataset based on the prediction results includes: An intermediate training dataset is generated based on the data with high classification feature weights in the unlabeled dataset and the labeled dataset.

5. The metadata intelligent identification method based on semi-supervised learning according to claim 1, characterized in that, The step of iteratively self-training the metadata keyword identifier and the metadata classifier based on the intermediate training dataset to obtain the iteratively self-trained metadata classifier includes: The metadata keyword identifier is trained based on the intermediate training dataset to generate a metadata classifier; Based on the metadata classifier, prediction results are generated for the unlabeled dataset. An intermediate training dataset is generated based on the prediction results; The process involves iteratively generating a metadata classifier, generating prediction results, and generating intermediate training datasets until the metadata classifier converges, resulting in a self-trained metadata classifier.

6. The metadata intelligent identification method based on semi-supervised learning according to claim 5, characterized in that, The step of iteratively self-training the metadata keyword identifier and the metadata classifier based on the intermediate training dataset to obtain the iteratively self-trained metadata classifier further includes: The classification accuracy is determined based on the labeled dataset in the prediction results; If the classification accuracy does not reach the preset value, the metadata classifier, prediction results and intermediate training dataset are generated in a loop. If the classification accuracy reaches the preset value, the loop self-training ends, and the metadata classifier after loop self-training is obtained.

7. A metadata intelligent identification device based on semi-supervised learning, characterized in that, The device includes: The first generation module is used to generate a metadata keyword identifier based on a conditional random field, wherein the metadata keyword identifier is used to extract metadata features corresponding to any identifier; The second generation module is used to obtain an initial labeled dataset and train the metadata keyword identifier to generate a metadata classifier based on the initial labeled dataset. The third generation module is used to obtain the unlabeled dataset, predict the unlabeled dataset based on the metadata classifier, and generate prediction results, wherein the prediction results include labeled datasets and unlabeled datasets; The fourth generation module is used to generate an intermediate training dataset based on the prediction results; The loop self-training module is used to loop self-train the metadata keyword identifier and the metadata classifier based on the intermediate training dataset to obtain the loop self-trained metadata classifier. The identification module is used to intelligently identify metadata based on the recurrently self-trained metadata classifier; Specifically, the first generation module is used for: The metadata and the corresponding identifiers are represented as an undirected graph. An observation sequence and an identifier sequence are determined based on a conditional random field, wherein the observation sequence corresponds to the metadata and the identifier sequence corresponds to the identifier corresponding to the metadata; An objective function and a feature set are determined based on the observation sequence and the identifier sequence, wherein the objective function is used to obtain the identifier sequence with the highest probability corresponding to the observation sequence, and the feature set is a set of metadata features; Generate a metadata keyword identifier based on the objective function and the feature set.

8. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory communicate with each other via the communication bus, characterized in that... The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1 to 6 by running the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images