Labeling method and device based on reinforcement learning and computer program product

Abstract

The invention relates to an annotation method and device based on reinforcement learning, computer equipment, a storage medium and a computer program product. The method comprises the steps of defining a labeling environment based on feature data corresponding to initial data; obtaining target feature data from the feature data, initializing an annotation reinforcement learning model according to the target feature data and a triple annotation result corresponding to the target feature data, enabling the initialized annotation reinforcement learning model to interact with the annotation environment, and outputting to obtain a final annotation strategy; and annotating data to be annotated through the final annotation strategy to obtain a triple annotation result of the data to be annotated. By adopting the method, the implicit relationship between the data can be identified, and the precision and efficiency of triple labeling are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a labeling method, apparatus, computer equipment, storage medium, and computer program product based on reinforcement learning. Background Art

[0002] Knowledge graph technology enhances the interpretability of expertise in the information and communications field. The construction of knowledge graphs primarily relies on triple annotations based on strong domain expertise. Triple annotation, as an effective information extraction method, can represent entities and relationships in unstructured text in a structured form.

[0003] There are two methods for triple labeling in related technologies: one is to manually label the input text according to the defined entity and relationship types; the other is entity labeling based on Chinese word segmentation, which uses manually labeled entity labeling text training data to train the target pre-training model, obtain the required entity labeling model, input the text to be labeled into the entity labeling model, and output the entity labeling results.

[0004] Both of the above methods cannot automatically discover the implicit relationships between data and cannot meet the needs of efficient, complete and accurate knowledge graph construction. Summary of the Invention

[0005] The embodiments of the present application provide a reinforcement learning-based labeling method, apparatus, computer device, storage medium, and computer program product, which can improve the accuracy and efficiency of triple labeling.

[0006] A labeling method based on reinforcement learning, the method comprising:

[0007] Define the annotation environment based on the feature data corresponding to the initial data;

[0008] Obtaining target feature data from the feature data, initializing a labeling reinforcement learning model based on the target feature data and a triplet labeling result corresponding to the target feature data, and allowing the initialized labeling reinforcement learning model to interact with the labeling environment to output a final labeling strategy;

[0009] The data to be labeled is labeled using the final labeling strategy to obtain a triplet labeling result of the data to be labeled.

[0010] In one embodiment, the method further comprises:

[0011] Based on the data to be labeled and the triplet labeling results of the data to be labeled, the optimization learning of the labeling reinforcement learning model is iteratively performed to output an updated final labeling strategy.

[0012] In one embodiment, the annotation environment includes: a state space, an action space and a transition function, wherein the state space includes historically annotated triples and the currently input text sequence; the action space includes selecting the next annotation action or transferring to the next feature data, and the next annotation action includes annotating a head entity, annotating a relationship, and annotating a tail entity; the transition function includes transferring to the next state, transferring to the next position of the feature data, and updating the annotation history.

[0013] In one embodiment, the reinforcement learning model includes a policy network and a value network, and the initialized annotation reinforcement learning model interacts with the annotation environment to output a final annotation strategy, including:

[0014] After multiple iterative learning and optimization processes, the final labeling strategy output by the strategy network is obtained;

[0015] The learning and optimization process includes:

[0016] Determining a target action selected by the strategy network based on the current labeling strategy and the current state;

[0017] Observe the new state that the labeled environment enters in response to the target action, and the reward given by the reward function for the target action;

[0018] Obtaining the value of the current state, the value of the target action, and the value of the new state calculated by the value network;

[0019] The policy network is tuned based on the value of the current state and the value of the target action, and the value network is tuned based on the value of the new state and the reward.

[0020] In one embodiment, the reward includes positive rewards and negative rewards, and the positive reward includes at least one of a basic reward, a completeness reward, and an exploration reward. The basic reward represents that the labeling action is correct, the completeness reward represents that the overall accuracy of the labeling meets the labeling requirements, and the exploration reward represents that a new implicit relationship is explored.

[0021] In one embodiment, obtaining target feature data from the feature data includes:

[0022] Clustering the feature data using an unsupervised algorithm to obtain at least one feature category, wherein the feature category includes at least one feature data;

[0023] Mark the implicit relationship corresponding to each feature category;

[0024] For any feature category, some feature data are selected from the feature category as target feature data, and triples are labeled on the target feature data based on the implicit relationship corresponding to the feature category to obtain triples labeling results corresponding to the target feature data.

[0025] In one embodiment, defining the annotation environment based on the feature data corresponding to the initial data includes:

[0026] Performing data preprocessing on the initial data, and extracting feature data from the preprocessed initial data to obtain feature data of the initial data;

[0027] A labeling environment is defined according to the feature data.

[0028] In one embodiment, the initial data is unstructured data, and extracting feature data from the preprocessed initial data to obtain feature data of the initial data includes:

[0029] Performing text conversion on the preprocessed initial data to obtain a high-dimensional vector space representation corresponding to the initial data;

[0030] Feature data is extracted based on the high-dimensional vector space representation to obtain feature data of the initial data, where the feature data is a feature related to the triplet annotation.

[0031] In one embodiment, the initial data is structured data, and extracting feature data from the preprocessed initial data to obtain feature data of the initial data includes:

[0032] Extracting key features from the preprocessed initial data;

[0033] According to the data processing strategy corresponding to the key feature, the key feature is processed to obtain feature data of the initial data.

[0034] A labeling device based on reinforcement learning, comprising:

[0035] The annotation environment definition module is used to define the annotation environment based on the feature data corresponding to the initial data;

[0036] a labeling reinforcement learning model module, configured to initialize a labeling reinforcement learning model based on target feature data and triplet labeling results corresponding to the target feature data, and to enable the initialized labeling reinforcement learning model to interact with the labeling environment to output experience data;

[0037] A labeling training and optimization module, configured to perform optimization training on the labeling reinforcement learning model based on the empirical data provided by the labeling reinforcement learning model module;

[0038] The annotation reinforcement learning model module is further used to output a final annotation strategy based on the optimized and trained annotation reinforcement learning model;

[0039] The implicit relationship discovery module is used to label the data to be labeled using the final labeling strategy to obtain triple labeling results of the data to be labeled.

[0040] In one embodiment, the annotation enhancement learning model module is further used to iteratively perform optimization learning of the annotation enhancement learning model based on the data to be labeled and the triplet labeling results of the data to be labeled.

[0041] In one embodiment, the apparatus further comprises:

[0042] The labeling reward function module is used to give corresponding rewards for the labeling results of the labeling reinforcement learning model. The rewards include positive rewards and negative rewards. The positive rewards include at least one of basic rewards, integrity rewards, and exploration rewards. The basic reward indicates that the labeling action is correct, the integrity reward indicates that the overall accuracy of the labeling meets the labeling requirements, and the exploration reward indicates that a new implicit relationship is explored.

[0043] In one embodiment, the apparatus further comprises:

[0044] The labeling module is used to learn the category of the implicit relationship from the feature data, and label the target feature data based on the category of the implicit relationship to obtain a triple labeling result of the target feature data.

[0045] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0046] Define the annotation environment based on the feature data corresponding to the initial data;

[0047] Obtaining target feature data from the feature data, initializing a labeling reinforcement learning model based on the target feature data and a triplet labeling result corresponding to the target feature data, and allowing the initialized labeling reinforcement learning model to interact with the labeling environment to output a final labeling strategy;

[0048] The data to be labeled is labeled using the final labeling strategy to obtain a triplet labeling result of the data to be labeled.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0050] Define the annotation environment based on the feature data corresponding to the initial data;

[0051] Obtaining target feature data from the feature data, initializing a labeling reinforcement learning model based on the target feature data and a triplet labeling result corresponding to the target feature data, and allowing the initialized labeling reinforcement learning model to interact with the labeling environment to output a final labeling strategy;

[0052] The data to be labeled is labeled using the final labeling strategy to obtain a triplet labeling result of the data to be labeled.

[0053] A computer program product, including a computer program, is characterized in that when the computer program is executed by a processor, it implements the reinforcement learning-based labeling method provided in an embodiment of the present application, which method can be:

[0054] Define the annotation environment based on the feature data corresponding to the initial data;

[0055] Obtaining target feature data from the feature data, initializing a labeling reinforcement learning model based on the target feature data and a triplet labeling result corresponding to the target feature data, and allowing the initialized labeling reinforcement learning model to interact with the labeling environment to output a final labeling strategy;

[0056] The data to be labeled is labeled using the final labeling strategy to obtain a triplet labeling result of the data to be labeled.

[0057] The above-mentioned reinforcement learning-based labeling method, device, communication equipment, storage medium and computer program product can define a labeling environment based on the feature data corresponding to the initial data, and obtain target feature data from the feature data, initialize the labeling reinforcement learning model according to the target feature data and the triple labeling result corresponding to the target feature data, and make the initialized labeling reinforcement learning model interact with the labeling environment to output a final labeling strategy, and then label the data to be labeled using the final labeling strategy to obtain the triple labeling result of the data to be labeled. Using the reinforcement learning-based labeling method, device, communication equipment, storage medium and computer program product provided by the embodiment of the present disclosure, the triple labeling strategy is learned and optimized through the reinforcement learning algorithm. The learned final labeling strategy can not only identify the implicit relationship between the data, but also significantly improve the accuracy and efficiency of the triple labeling, thereby meeting the requirements of efficient, complete and accurate knowledge graph construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 1 is a flow chart of a reinforcement learning-based labeling method in one embodiment;

[0059] Figure 2 1 is a flow chart of a reinforcement learning-based labeling method in one embodiment;

[0060] Figure 3 is a schematic diagram of a labeling method based on reinforcement learning in one embodiment;

[0061] Figure 4 1 is a flow chart of a reinforcement learning-based labeling method in one embodiment;

[0062] Figure 5 is a schematic diagram of a labeling method based on reinforcement learning in one embodiment;

[0063] Figure 6 is a structural block diagram of a labeling device based on reinforcement learning in one embodiment;

[0064] Figure 7 4 is a structural block diagram of a labeling device based on reinforcement learning in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0066] Knowledge graph technology enhances the interpretability of professional knowledge in the information and communications field. Knowledge computing based on knowledge graphs has significantly improved the intelligence level of information and communications networks, but it also brings new challenges to the construction of information and communications-related knowledge graphs. Specifically, in addition to business-unrelated technologies such as data collection and graph data storage, knowledge graph construction relies heavily on strong domain expertise for triple annotation. Triple annotation, as an effective information extraction method, can represent entities and relationships in unstructured text in a structured form, providing critical data support for applications such as knowledge graph construction, knowledge question answering, and semantic search.

[0067] Currently, there are two approaches to triple annotation for knowledge graphs. One involves purely manual annotation of input text based on defined entity and relationship types. This method requires extensive human involvement with extensive business experience, resulting in low efficiency and high cost. The other involves entity annotation based on Chinese word segmentation. This involves obtaining manually annotated entity annotation text training data, fine-tuning the target pre-trained model to obtain the required entity annotation model, inputting the text to be annotated into the entity annotation model, and outputting the entity annotation results. However, neither of these approaches can automatically discover implicit relationships and can only be applied to the annotation of unstructured text data, failing to meet the requirements for efficient, complete, and accurate knowledge graph construction.

[0068] Taking network operations as an example, in addition to the large amount of unstructured knowledge documents generated during network operations, such as operation and maintenance manuals, case studies, and technical specifications, there is also a massive amount of structured operational data, including alarm logs, performance monitoring data, and network topology information. These documents and data contain many valuable implicit relationships, which are crucial for network troubleshooting, performance optimization, and preventive maintenance. Efficiently and accurately extracting triples from this massive amount of text and operational data to construct a knowledge graph has become a challenging issue in the network operations field.

[0069] Based on the above traditional technology, the embodiment of the present application provides a tagging method based on reinforcement learning, which adopts the reinforcement learning algorithm to learn and optimize the triple tagging strategy. The final tagging strategy learned can identify the implicit relationship between the data. The final tagging strategy is used for triple tagging, which can significantly improve the accuracy and efficiency of triple tagging, thereby meeting the requirements of efficient, complete and accurate knowledge graph construction.

[0070] It should be noted that the following will use the network operation scenario as an example to illustrate the embodiments of the present disclosure, but it should be understood that the reinforcement learning-based labeling method provided by the embodiments of the present disclosure is not limited to triple labeling in the network operation scenario. The method provided by the embodiments of the present disclosure is applicable to any scenario where triple labeling is performed.

[0071] It should be noted that the beneficial effects or technical problems solved by the embodiments of the present application are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0072] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0073] In one embodiment, Figure 1 As shown, a reinforcement learning-based annotation method is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. The following steps are included:

[0074] Step 102: defining a labeling environment based on feature data corresponding to the initial data.

[0075] In the embodiments of the present disclosure, the initial data may be training data or data to be annotated. For example, the initial data may be obtained from various data sources offline or online. The initial data may include structured data and unstructured data. The embodiments of the present disclosure do not specifically limit the method for obtaining the initial data and the type of data.

[0076] Taking the network operation scenario as an example, the initial data may include unstructured data such as network operation and maintenance manuals, cases, technical specifications, etc. obtained from the network operation management system or operator knowledge base system, or obtained through manual upload, as well as massive structured operation data, including alarm logs, performance monitoring data, network topology information, etc.

[0077] Exemplarily, feature data of the initial data may be extracted, and the extracted feature data may be used to define a labeling environment, including defining a state space, defining an action space, and defining a transfer function.

[0078] In an exemplary embodiment, in step 102, defining a labeling environment based on feature data corresponding to the initial data may include:

[0079] The initial data is preprocessed, and feature data is extracted from the preprocessed initial data to obtain feature data of the initial data; and a labeling environment is defined according to the feature data.

[0080] In embodiments of the present disclosure, the initial data can be preprocessed, including cleaning and preprocessing operations on the initial data. For example, the initial data can be cleaned (including deduplication, error removal or repair, filtering inconsistent or invalid data, etc.), formatted, word segmented, part-of-speech tagged, terminology identified, and structured information extracted. Furthermore, feature data can be extracted from the preprocessed initial data using methods such as word embedding to obtain feature data of the initial data, and a labeling environment can be defined based on the feature data. This can further improve the learning efficiency and accuracy of the labeling strategy, thereby improving the accuracy and efficiency of triple labeling.

[0081] In an exemplary embodiment, the initial data includes unstructured data, and feature data extraction is performed on the preprocessed initial data to obtain feature data of the initial data, which may include:

[0082] The preprocessed initial data is subjected to text conversion processing to obtain a high-dimensional vector space representation corresponding to the initial data; feature data is extracted based on the high-dimensional vector space representation to obtain feature data of the initial data, where the feature data is features related to triple annotation.

[0083] In the disclosed embodiments, when the initial data is unstructured, word embedding technology can be used to convert the preprocessed text into a high-dimensional vector space representation. Based on this high-dimensional vector space representation, features related to triple annotations can be extracted to obtain feature data for the initial data. For example, the feature data extracted from network operations manuals can include operating procedures, device names, configuration parameters, and so on.

[0084] In another exemplary embodiment, the initial data is structured data, and feature data extraction is performed on the preprocessed initial data to obtain feature data of the initial data, which may include:

[0085] Extract key features from the preprocessed initial data; perform data processing on the key features according to the data processing strategy corresponding to the key features to obtain feature data of the initial data.

[0086] In the embodiments of the present disclosure, when the initial data is structured data, key features can be directly extracted from the preprocessed initial data. Taking the network operations field as an example, when the initial data is alarm data and performance data, extracting key features from the alarm data and performance data can include: extracting key features from the alarm data, such as alarm type, alarm level, alarm time, and alarm location (device / link); and extracting key features from the performance data, such as performance indicator names (such as throughput, latency, packet loss rate, CPU / memory utilization), performance indicator values, etc.

[0087] After extracting key features, a data processing strategy for the key features can be obtained. Based on the data processing strategy corresponding to the key features, the key data can be processed to obtain feature data for the initial data. Data processing strategies include strategies for assigning entities to key data and converting data formats. For example, using alarm data and performance data as the initial data, the data processing strategy could include assigning relevant network elements (such as switches, routers, and servers) to each alarm and performance metric based on network topology information, or performing necessary encoding conversions on non-numeric features, such as alarm type.

[0088] The reinforcement learning-based labeling method provided in the embodiments of the present disclosure can realize triple labeling of structured data and unstructured data, automatically learn the implicit relationships between entities in unstructured data and structured data, and apply them to the knowledge graph triple labeling task, thereby improving the accuracy and completeness of the labeling.

[0089] Step 104: obtain target feature data from the feature data, initialize the labeling reinforcement learning model according to the target feature data and the triplet labeling results corresponding to the target feature data, and allow the initialized labeling reinforcement learning model to interact with the labeling environment to output the final labeling strategy.

[0090] In the disclosed embodiment, the target feature data may be a small portion of the feature data that has been triple-labeled. Training data may be constructed based on the target feature data and the triple-labeled results corresponding to the target feature data. A labeling reinforcement learning model may be initialized based on the training data, and the initialized labeling reinforcement learning model may be caused to interact with a labeling environment. This allows the labeling reinforcement learning model to perform reinforcement learning and optimization of the labeling strategy in conjunction with a reward function, resulting in a final labeling strategy. This final labeling strategy is used to identify implicit relationships between data and perform triple labeling based on the identified implicit relationships.

[0091] In an exemplary embodiment, referring to Figure 2 As shown, in step 104, obtaining target feature data from feature data may include:

[0092] Step 202: clustering the feature data using an unsupervised algorithm to obtain at least one feature category, each of which includes at least one feature data;

[0093] Step 204: mark the implicit relationship corresponding to each feature category;

[0094] Step 206 , for any feature category, select part of the feature data from the feature category as target feature data, and perform triple labeling on the target feature data based on the implicit relationship corresponding to the feature category to obtain a triple labeling result corresponding to the target feature data.

[0095] In an embodiment of the present disclosure, after obtaining the feature categories of the initial data, an unsupervised algorithm can be used to cluster the feature data to obtain at least one feature category. For any feature category, the feature category can include at least one feature data, and at least one feature data in the feature category corresponds to a consistent implicit relationship, that is, the feature category can actually be regarded as an implicit relationship category.

[0096] Therefore, after clustering to obtain at least one feature category, the feature category can be labeled based on the implicit relationship corresponding to the feature category. Taking the network operation field as an example, feature categories can correspond to implicit relationships such as: include, cover, cause, generate, influence, and associate.

[0097] After labeling each feature category, a small amount of feature data from each feature category can be selected as target feature data. Based on the implicit relationships between the feature categories, the target feature data can be triple-labeled to obtain the corresponding triple-labeled results. For example, in the feature category with the implicit relationship "cause", the target feature data is selected: When the CPU utilization of the switch exceeds 80%, it may cause increased network latency. After manually or using a pre-trained entity labeling model to perform triple-labeling on this target feature data, the triple-labeled results for the target feature data can be obtained: switch, cause, increased network latency.

[0098] The target feature data and the triplet labeling results of the target feature data are obtained through cluster labeling and other means as training data to initialize the labeling reinforcement learning model to build a labeling reinforcement learning model for triple labeling. Then, the labeling strategy is learned through the reinforcement learning algorithm. The final labeling strategy can be learned that can not only identify the implicit relationship between the data, but also significantly improve the accuracy and efficiency of triple labeling, thereby meeting the needs of efficient, complete and accurate knowledge graph construction.

[0099] Step 106 : annotate the data to be annotated using the final annotation strategy to obtain triplet annotation results of the data to be annotated.

[0100] In the disclosed embodiments, the data to be annotated may include data that is not annotated in the initial data, or data that is distinct from the initial data. The disclosed embodiments do not specifically limit the data to be annotated. After obtaining a final annotation strategy, the final annotation strategy can be applied to identify implicit relationships in the data to be annotated, thereby performing triple annotation based on the implicit relationships, obtaining triple annotation results for the data to be annotated. Based on the triple annotation results, a more complete and accurate knowledge graph can be constructed.

[0101] The above-mentioned reinforcement learning-based labeling method can define a labeling environment based on the feature data corresponding to the initial data, obtain target feature data from the feature data, initialize the labeling reinforcement learning model according to the target feature data and the triple labeling results corresponding to the target feature data, and make the initialized labeling reinforcement learning model interact with the labeling environment to output a final labeling strategy, and then label the data to be labeled using the final labeling strategy to obtain the triple labeling results of the data to be labeled. The reinforcement learning-based labeling method provided by the embodiment of the present disclosure is used to learn and optimize the triple labeling strategy through the reinforcement learning algorithm. The learned final labeling strategy can not only identify the implicit relationship between the data, but also significantly improve the accuracy and efficiency of the triple labeling, thereby meeting the requirements of efficient, complete and accurate knowledge graph construction.

[0102] In an exemplary embodiment, the above method may further include:

[0103] Based on the data to be labeled and the triplet labeling results of the data to be labeled, the optimization learning of the labeling reinforcement learning model is iteratively performed to output the updated final labeling strategy.

[0104] In the disclosed embodiment, after triple labeling of the data to be labeled, the obtained triple labeling results can be fed back to the labeling reinforcement learning model, so that the reinforcement learning model can be iteratively trained through the triple labeling results in the application process, so as to realize the iterative learning and optimization of the labeling strategy by the labeling reinforcement learning model, obtain a more accurate final labeling strategy, and use the final labeling strategy obtained by reinforcement learning to perform triple labeling of the data to be labeled.

[0105] By adopting the reinforcement learning-based labeling method provided by the embodiment of the present disclosure, the labeling strategy is adaptively adjusted using reinforcement learning. The labeling reinforcement learning model can dynamically optimize its subsequent labeling strategy based on the current triplet labeling results and the feedback of the reward function, thereby improving the accuracy and efficiency of triplet labeling.

[0106] In an exemplary embodiment, the annotation environment includes: a state space, an action space, and a transition function. The state space includes historically annotated triples and the currently input text sequence. The action space includes selecting the next annotation action or transferring to the next feature data. The next annotation action includes annotating a head entity, annotating a relationship, and annotating a tail entity. The transition function includes transferring to the next state, transferring to the next position of the feature data, and updating the annotation history.

[0107] In this disclosed embodiment, a labeling environment is defined based on feature data, including a state space, an action space, and a transition function. This provides an interactive environment interface for the labeling reinforcement learning model. The triple labeling task is modeled as a sequential decision problem, where each decision step corresponds to selecting a relation to label. The labeling environment provides information about the current state of the text (e.g., labeled triples, remaining unlabeled text, etc.) and a possible action space.

[0108] Reference Figure 3 As shown in FIG, the state space is defined to be composed of a text sequence and an annotation history (such as annotated triples), where the text sequence treats the input text as a state sequence, where each word or character in the text sequence can be regarded as a state, and the annotation history can include not only the current text position but also the previous triple annotation results to provide contextual information through the triple annotation results.

[0109] Defining the action space consists of a labeling strategy and a movement strategy. The labeling strategy allows the reinforcement learning model (agent) to choose the next action to perform in each state, such as labeling the head entity, labeling a relationship or irrelevant entity, or labeling the tail entity. The movement strategy involves choosing the next position to move to in the text. Defining the transition function involves defining state transitions.

[0110] The transition function is defined as a state transition. The annotation reinforcement learning model (agent) will transition to the next state based on the current state and the selected action. For example, if the annotation reinforcement learning model (agent) selects a labeling action, it may transition to the next position in the text and update the labeling history based on the labeling action.

[0111] After completing the definition of the labeling environment, the labeling reinforcement learning model can achieve enhanced learning for labeling strategies by interacting with the standard environment, thereby learning labeling strategies that can discover implicit relationships between data. This labeling strategy can be used to perform triple labeling, significantly improving the accuracy and efficiency of triple labeling, thereby meeting the needs of efficient, complete and accurate knowledge graph construction.

[0112] In an exemplary embodiment, the reinforcement learning model includes a policy network and a value network. In step 104, the initialized annotation reinforcement learning model interacts with the annotation environment to output a final annotation strategy, which may include:

[0113] After multiple iterations of learning and optimization, the final labeling strategy output by the policy network is obtained;

[0114] Among them, reference Figure 4 As shown, the learning and optimization process may include:

[0115] Step 402, determining the target action selected by the strategy network based on the current labeling strategy and the current state;

[0116] Step 404: observe and annotate the new state that the environment enters in response to the target action, and the reward given by the reward function for the target action;

[0117] Step 406, obtaining the value of the current state, the value of the target action, and the value of the new state calculated by the value network;

[0118] Step 408: Adjust the parameters of the policy network based on the value of the current state and the value of the target action, and adjust the parameters of the value network based on the value of the new state and the reward.

[0119] In the disclosed embodiment, the text sequence and the historical triplet annotation results can be encoded into a complete state representation and then input into the policy network. The policy network outputs the action probability distribution corresponding to each annotation action based on the current state corresponding to the current text position in the state representation and the current annotation strategy. The policy network uses the ε-greedy strategy to select the target action as the next action, while observing the new states that the annotation environment can enter in response to the target action. The reward function can provide a corresponding reward based on the target action for the current state, for example, providing a positive reward signal when the target action is accurately labeled and a negative reward signal when the target action is incorrectly labeled.

[0120] The value network evaluates the value of the current state, the value of the target action, and the value of the new state. It uses the advantage function to calculate the difference between the value of the current state and the value of the target action. Based on this difference, the policy gradient is calculated. The policy network parameters are adjusted based on the policy gradient to maximize the expected long-term reward, completing a single learning cycle for the policy network. Simultaneously, based on the value of the new state and the observed reward signal, the TD (Temporal Difference) algorithm is used to adjust the value network parameters to minimize the error in the state value estimation, completing a single learning cycle for the value network.

[0121] Repeat the above process to iteratively learn the annotation reinforcement learning model until the number of iterations reaches the preset iteration threshold, and then output the final annotation strategy.

[0122] In an embodiment of an embodiment, the reward includes positive rewards and negative rewards. The positive reward includes at least one of a basic reward, a completeness reward, and an exploration reward. The basic reward represents that the labeling action is correct, the completeness reward represents that the overall accuracy of the labeling meets the labeling requirements, and the exploration reward represents that a new implicit relationship is explored.

[0123] In the disclosed embodiment, a reward function is defined to improve the practicality and accuracy of the implicit relationship explored by the labeling strategy, and the rewards are determined to include positive rewards and negative rewards, wherein positive rewards are given when the labeling is accurate and negative rewards are given when the labeling is incorrect.

[0124] Among them, the positive reward includes at least one of the basic reward, integrity reward, and exploration reward. Figure 3 As shown, after the reward function is defined, it can provide corresponding rewards based on the annotation action. If the annotation action is accurate, the reward function can provide a basic reward signal. After the entire text is annotated and the overall accuracy meets the annotation requirements, the reward function can provide a completeness reward. If the explored relationship is an implicit relationship that has never appeared in history and / or the implicit relationship is practical, the reward function will provide an exploration reward. During the reinforcement learning process, the annotation reinforcement learning model interacts with the annotation environment to learn and execute the annotation strategy, which is used to map states to annotation actions.

[0125] The reinforcement learning-based labeling method provided by the embodiments of the present disclosure designs a reward function for the specific scenario of triple labeling. The designed reward function will drive the labeling reinforcement learning model to focus on learning implicit relationships that are beneficial to the triple labeling results. This reward-driven learning method enables the model to focus more on the task objectives, avoids unnecessary waste of computing resources, and improves the learning efficiency and accuracy of the labeling strategy.

[0126] In order to enable those skilled in the art to better understand the embodiments of the present disclosure, the embodiments of the present disclosure will be described below through specific examples.

[0127] Reference Figure 5 As shown, initial data, such as network operation and maintenance manuals, can be obtained from network operation management systems or operator knowledge base systems, or manually uploaded. After preprocessing the initial data, including text cleaning, term recognition, and structured information extraction, embedding techniques can be used to convert the preprocessed text into a high-dimensional vector space representation. Feature data related to triple annotations can then be extracted based on this high-dimensional vector space representation. Unsupervised clustering algorithms are used to classify the feature data, generating feature categories containing implicit relationships. These feature categories are then manually refined and defined, for example, by correcting the data within the feature categories and defining implicit relationships.

[0128] The extracted feature data is used to define the annotation environment, including defining the state space, action space, and transition function. A reward function is defined based on the practicality and accuracy requirements of the triplet annotation task, and the accuracy and practicality of the triplet annotation results are rewarded accordingly. A small amount of data from the feature categories is selected for triplet relationship annotation and used as pre-training data. This pre-training data is used to initialize the annotation reinforcement learning model (agent) based on the A2C (Advantage-Actor-Criti) algorithm. The annotation reinforcement learning model is constructed from a policy network and a value network.

[0129] A training loop is constructed. In each training step, the annotation reinforcement learning model interacts with the annotation environment, selects annotation actions, and adjusts network parameters based on the reward function's feedback for these annotation actions. During training, the annotation reinforcement learning model discovers implicit relationships in the initial data. Based on the feedback from the reward function, the annotation reinforcement learning model's annotation strategy is adjusted, gradually learning these implicit relationships and obtaining the final annotation strategy. The final annotation strategy is then used to annotate the unlabeled data, generating triplet annotations that represent the implicit relationships in the unlabeled data. These triplet annotations are then fed back to the annotation reinforcement learning model for a new round of optimization. The output triplet annotations are then post-processed and output in a structured format.

[0130] The following uses the network operation field as an example to illustrate the embodiments of the present disclosure. In the network operation field, the data to be annotated may include structured data and unstructured data. The following describes the process of annotating structured data and unstructured data respectively.

[0131] In one example, when the initial data is unstructured data, such as a network operation and maintenance manual, implicit triple relationships are extracted from the manual to assist in network troubleshooting and configuration management. The manual contains network device configuration information, troubleshooting guides, performance optimization suggestions, etc. The specific implementation process is as follows:

[0132] Obtain network operation and maintenance manuals from the network operation management system, operator knowledge base, or manually upload them. Perform data preprocessing on the manuals, including text cleaning, term recognition, and structured information extraction to remove irrelevant characters, perform word segmentation, and perform part-of-speech tagging. This produces processed text data, which is then stored in a database. Using network operation-specific word embedding techniques, the preprocessed text is converted into a high-dimensional vector space representation. Features related to triplet annotations, such as operating procedures, device names, and configuration parameters, are extracted to produce feature data.

[0133] An unsupervised clustering algorithm is used to classify feature data, generating initial implicit relationship categories. These are then manually refined and defined to yield relationship categories such as include, cover, cause, generate, influence, and associate. These relationship categories are then used to label a small amount of feature data as triples. The triple labeling output serves as pre-training data to initialize the labeling reinforcement learning model. The labeling reinforcement learning model is then supervised using the triple labeling results of the feature data as a supervisory signal. For example, the feature data is a passage from a network operation and maintenance manual: "When the CPU utilization of a switch exceeds 80%, it may cause increased network latency." This text does not directly mention the relationship between the switch and increased network latency. However, through contextual reasoning, a triple labeling structure can be derived: switch, cause, increased network latency, where "cause" is the implicit relationship between the switch and increased network latency.

[0134] The annotation environment is defined using feature data, including the state space, action space, and transition function. The state is defined as the triple annotation results of the current text sequence and historical annotations; the annotation action is to annotate the head entity, annotate the relationship, or annotate the tail entity, or to choose to skip the current position and continue processing the next text segment. The state transition is defined. Based on the current state and the selected annotation action, the annotation reinforcement learning model will transition to the next state, the next position in the text, and update the annotation history based on the selected annotation action. The reward function is defined based on the practicality and accuracy requirements of the triple annotation task, and rewards are given for the accuracy of the annotation results and their practicality for network operation and maintenance. This includes basic rewards, completeness rewards, exploration rewards, and negative rewards.

[0135] The A2C-based labeled reinforcement learning model (agent) is initialized using pre-trained data. The labeled reinforcement learning model consists of a policy network and a value network. The policy network generates triple-labeled actions, while the value network evaluates the value of states and actions. The policy network generates a probability distribution over actions based on the current state, outputs the probability of each possible action, and uses an ε-greedy strategy to select the target action as the next action. A recurrent neural network is used to encode the text, generating a fixed-length vector as the state representation. The triple-labeled results of the historical annotations are then encoded, and together with the text encoding, form the complete state representation.

[0136] The value network evaluates the expected long-term reward from the current state. It takes the state as input and outputs a scalar value representing the estimated value of the current state. The value network helps the model consider possible future rewards when selecting actions. The accuracy and usefulness of newly discovered implicit relationships are defined, for example, by whether the relationship can help quickly locate faults, reveal new correlations between performance indicators, or reveal new correlations between fault phenomena and performance indicators. The advantage function is used to calculate the difference between the action value of the target action and the state value of the current state. The policy gradient is then calculated based on this difference, and policy extraction is used to update the parameters of the policy network. Simultaneously, the reward given by the observed reward function for the target action and the state value of the new state are used to update the parameters of the value network using the time-delayed learning method.

[0137] A training loop is constructed. During each training step, the annotation reinforcement learning model interacts with the environment, generates annotation actions, and adjusts parameters based on the feedback from the reward function. During training, the reinforcement learning model discovers implicit relationships within the network operations manual, such as dependencies between operation steps and the relationship between device configuration and troubleshooting. The annotation strategy of the annotation reinforcement learning model is adjusted based on the feedback from the reward function, gradually learning the implicit relationships within the network operations manual.

[0138] The trained final annotations are used to annotate the unlabeled data to obtain triple annotation results that represent the implicit relationships between the data. The triple annotation results are fed back to the annotation reinforcement learning model for a new round of optimization. The output annotation results are post-processed, such as formatted output, relationship induction and organization, etc. The processed triple annotation results are output in a structured format for network operation and maintenance personnel to use in building knowledge graphs to assist in troubleshooting and configuration management.

[0139] In another example, when the initial data is structured data, for example, network alarm data and performance data, implicit triple relationships can be extracted from the network alarm data and performance data to assist in network troubleshooting and performance optimization.

[0140] Multidisciplinary alarm and performance data are collected from the integrated alarm system, integrated performance system, and network management system as a dataset, which serves as the initial data. The initial data is cleaned to remove invalid and duplicate data, formatted and normalized to ensure data quality and suitability for model processing, and then stored in the database. Key features are extracted from the alarm and performance data, such as alarm type, alarm level, alarm time, and alarm location (device / link) from the alarm data, and key data such as performance indicator names (such as throughput, latency, packet loss rate, CPU / memory utilization), and performance indicator values ​​from the performance data. Incorporating network topology information, each key data item is assigned a relevant network element (such as a switch, router, server, etc.). For non-numeric features, such as alarm type, necessary encoding conversion is performed to obtain feature data.

[0141] An unsupervised clustering algorithm is used to classify feature data to obtain initial implicit relationship categories, which are then manually refined and defined to obtain relationship categories such as: cause, generate, influence, and associate. A small amount of feature data is labeled as triples based on the relationship categories. The triple labeling results are used as pre-training data to initialize the labeling reinforcement learning model. For example, the performance indicator name field is "CPU utilization" and the fault phenomenon field is "increased network delay". The relationship between CPU utilization and increased network delay is not directly mentioned, but through historical data mining, the implicit relationship between the two can be obtained. Since the network element corresponding to CPU utilization is the switch, the labeling obtains the triplet: switch, cause, increased network delay, where "cause" is the implicit relationship between the switch and increased network delay.

[0142] The annotation environment is defined using feature data, including the state space, action space, and transition function. The state is defined as the triple annotation results of the current text sequence and historical annotations; the annotation action is to annotate the head entity, annotate the relationship, or annotate the tail entity, or to choose to skip the current position and continue processing the next text segment. The state transition is defined. Based on the current state and the selected annotation action, the annotation reinforcement learning model will transition to the next state, the next position in the text, and update the annotation history based on the selected annotation action. The reward function is defined based on the practicality and accuracy requirements of the triple annotation task, and rewards are given for the accuracy of the annotation results and their practicality for network operation and maintenance. This includes basic rewards, completeness rewards, exploration rewards, and negative rewards.

[0143] A labeled reinforcement learning model is constructed to learn implicit triple relationships from structured data. Based on the A2C architecture, the policy network generates triple-labeled actions, selects alerts or performance indicators to explore, and predicts possible relationships between them. The policy network takes as input a state representation that encodes the currently processed alerts and performance indicators, discovered implicit relationships, and network topology information. Its output is an action probability distribution, which is used to select the next alert or performance indicator to explore and predict possible relationships between them. The value network is responsible for evaluating the value of the current state, that is, predicting the long-term reward. The advantage function is used to calculate the difference between the action value of the target action and the state value of the current state. Based on this difference, the policy gradient is calculated, and policy extraction is used to update the parameters of the policy network. Simultaneously, the reward given by the observed reward function for the target action and the state value of the new state are used to update the parameters of the value network using the time-delayed learning method.

[0144] A training loop is constructed. During each training step, the annotation reinforcement learning model interacts with the environment, generates annotation actions, and adjusts parameters based on the feedback of the reward function. Leveraging the exploration mechanism of reinforcement learning, it continuously discovers implicit triple relationships from new alarm and performance data. These relationships may include associations between alarms and performance indicators, or temporal relationships between alarms, generating triplet annotation results. The triplet annotation results are fed back to the annotation reinforcement learning model for a new round of optimization. Based on the feedback from the reward function, the model's annotation strategy is adjusted to gradually learn the implicit relationships between alarm and performance data.

[0145] The output triplet annotation results are post-processed, such as removing redundant annotations and performing consistency checks. The processed triplet annotation results are generated into a knowledge graph output or directly output as triples for use by network operation and maintenance personnel to assist in troubleshooting and performance optimization.

[0146] The reinforcement learning-based labeling method provided by the embodiment of the present disclosure is used to dynamically identify implicit relationships through the exploration mechanism of reinforcement learning, so that the labeling reinforcement learning model can continuously discover new relationship types in the interaction with the defined labeling environment. Traditional technologies are often limited to predefined sets of explicit relationships, while the embodiment of the present disclosure can dynamically identify and learn implicit relationships in text. And unlike traditional rule-based labeling methods, the reinforcement learning-based labeling method provided by the embodiment of the present disclosure uses reinforcement learning to adaptively adjust the labeling strategy, so that the model can dynamically optimize its subsequent labeling strategy based on the current labeling results and feedback, thereby improving the accuracy and efficiency of labeling. The reinforcement learning-based labeling method provided by the embodiment of the present disclosure realizes the joint optimization of the two tasks of triple labeling and implicit relationship recognition. By sharing the underlying text representation and jointly training the strategy network and value network, the two tasks are mutually promoted, further improving the overall performance.

[0147] It should be understood that although Figure 1-5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0148] In one embodiment, Figure 6 As shown, a labeling device based on reinforcement learning is provided, wherein the labeling device based on reinforcement learning includes: a labeling environment definition module 602, a labeling reinforcement learning model module 604, a labeling training and optimization module 606, and an implicit relationship discovery module 608, wherein,

[0149] A labeling environment definition module 602 is used to define a labeling environment based on feature data corresponding to the initial data;

[0150] The labeling reinforcement learning model module 604 is used to initialize the labeling reinforcement learning model based on the target feature data and the triplet labeling results corresponding to the target feature data, and enable the initialized labeling reinforcement learning model to interact with the labeling environment to output experience data;

[0151] The annotation training and optimization module 606 is used to optimize the annotation reinforcement learning model based on the empirical data provided by the annotation reinforcement learning model module;

[0152] The labeling reinforcement learning model module 604 is further used to output a final labeling strategy based on the optimized and trained labeling reinforcement learning model;

[0153] The implicit relationship discovery module 608 is used to label the data to be labeled using the final labeling strategy to obtain triple labeling results of the data to be labeled.

[0154] In an exemplary embodiment, the annotation enhancement learning model module 604 is further configured to iteratively perform optimization learning of the annotation enhancement learning model based on the data to be annotated and the triplet annotation results of the data to be annotated.

[0155] In an exemplary embodiment, the reinforcement learning-based labeling apparatus further includes:

[0156] The labeling reward function module is used to give corresponding rewards for the labeling results of the labeling reinforcement learning model. The rewards include positive rewards and negative rewards. Positive rewards include at least one of basic rewards, completeness rewards, and exploration rewards. Basic rewards represent that the labeling action is correct, the completeness reward represents that the overall accuracy of the labeling meets the labeling requirements, and the exploration reward represents that new implicit relationships are explored.

[0157] In an exemplary embodiment, the reinforcement learning-based labeling apparatus further includes:

[0158] The labeling module is used to learn the categories of implicit relations from the feature data, and label the target feature data based on the categories of implicit relations to obtain the triple labeling results of the target feature data.

[0159] The operations performed by the functional modules in the reinforcement learning-based labeling device in the embodiment of the present disclosure may refer to the relevant descriptions of the aforementioned embodiment, which will not be described in detail in the embodiment of the present disclosure.

[0160] The following is a complete example of the present disclosure. Figure 7 As shown, the reinforcement learning-based labeling device can be composed of a data acquisition module, a data preprocessing module, a data storage module, a feature extraction and representation learning module, an implicit relationship discovery module, a labeling environment definition module, a labeling reinforcement learning model module, a labeling reward function module, a labeling training and optimization module, and a post-processing and result output module, wherein:

[0161] Data acquisition module: responsible for acquiring initial data from multiple data sources offline or online, including structured data and unstructured data, and outputting the initial data to the data preprocessing module.

[0162] Data preprocessing module: performs cleaning and preprocessing operations on the initial data, such as deduplication, removal or repair of errors, inconsistent or invalid data, formatting, word segmentation, part-of-speech tagging, etc. The processed data is stored in the data storage module.

[0163] Data storage module: stores preprocessed data in the database, supports structured, unstructured and vector data storage, provides data for feature extraction and representation learning modules, and also stores intermediate data and annotation result data generated by other modules.

[0164] The Feature Extraction and Representation Learning Module reads preprocessed initial data from the Data Storage Module, converts the unstructured text into a high-dimensional vector space representation, and extracts feature data related to triple annotations based on the high-dimensional vector space representation, such as entity type, syntactic structure, and contextual keywords. It also directly extracts fields related to triple annotations from the structured data as feature data. This module provides input feature data for the Environment Definition Module and the Implicit Relationship Discovery Module.

[0165] The labeling module defines the categories and labeling specifications for implicit relationships, automatically discovers potential relationship categories from feature data using unsupervised learning methods, and then manually refines and defines them. A small amount of feature data is initially labeled as triplets, which serve as training samples for the reinforcement learning model and as supervisory signals for the labeling training and optimization modules.

[0166] The annotation environment definition module defines the annotation reinforcement learning environment based on the feature data, including the state space, action space, and transition function. This provides an interactive environment interface for the annotation reinforcement learning module. The triplet annotation task is modeled as a sequential decision problem, where each decision step corresponds to selecting a relation to annotate. The annotation environment provides information about the current state of the text (e.g., annotated triples, remaining unannotated text, etc.) and the possible action space (e.g., the selected relation category).

[0167] Annotation Reinforcement Learning Model Module (Annotation Agent): Initializes the annotation reinforcement learning module (agent), which includes a policy network and a value network. The policy network is used to generate annotation actions, and the value network is used to evaluate the value of the state. In the disclosed embodiment, the A2C algorithm is used, combining the two methods of policy gradient and value function approximation to generate triple-annotated actions and evaluate the value of the current state. This module learns and optimizes through interaction with the annotation environment, and outputs the learned annotation strategy as empirical data to the annotation training and optimization module to provide empirical data, and outputs it as the final annotation strategy to the implicit relationship discovery module.

[0168] Labeling Reward Function Module: Based on the characteristics of the triple labeling task, a labeling reward function is designed to evaluate the consistency between the model-generated labeling results and the actual labeling. This module provides reward signals to the labeling reinforcement learning model. When the model successfully labels an implicit relationship, it provides positive rewards, including base rewards, completeness rewards, and exploration rewards. When the labeling is incorrect, it provides negative rewards.

[0169] Labeling training and optimization module: Uses the empirical data generated by the labeling reinforcement learning model module, performs iterative training and continuous optimization, outputs the optimized parameters to the labeling reinforcement learning model module, updates the parameters of the policy network and value network of the labeling reinforcement learning model, and is responsible for the training and performance improvement of the labeling reinforcement learning model.

[0170] Implicit Relationship Discovery Module: During model training, it discovers implicit relationships within text or structured data and uses the annotation strategy of the annotation reinforcement learning model to test and adjust them. During the inference application phase, the module processes new unlabeled data based on the final annotation strategy provided by the annotation reinforcement learning module and outputs the triplet annotation results to the post-processing and result output module.

[0171] Post-processing and result output module: This module post-processes the triple annotation results output by the implicit relationship discovery module and outputs them in a structured format. This module is responsible for providing the processed results for subsequent knowledge graph construction and application use.

[0172] The above-mentioned reinforcement learning-based labeling device learns and optimizes the triple labeling strategy through the reinforcement learning algorithm. The final labeling strategy learned can not only identify the implicit relationship between data, but also significantly improve the accuracy and efficiency of triple labeling, thereby meeting the needs of efficient, complete and accurate knowledge graph construction.

[0173] For the specific definition of the reinforcement learning-based annotation device, please refer to the definition of the reinforcement learning-based annotation method above, which will not be repeated here. The various modules in the above-mentioned reinforcement learning-based annotation device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0175] Define the annotation environment based on the feature data corresponding to the initial data;

[0176] Obtaining target feature data from the feature data, initializing a labeling reinforcement learning model based on the target feature data and a triplet labeling result corresponding to the target feature data, and allowing the initialized labeling reinforcement learning model to interact with the labeling environment to output a final labeling strategy;

[0177] The data to be labeled is labeled using the final labeling strategy to obtain a triplet labeling result of the data to be labeled.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0179] Based on the data to be labeled and the triplet labeling results of the data to be labeled, the optimization learning of the labeling reinforcement learning model is iteratively performed to output an updated final labeling strategy.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0181] After multiple iterative learning and optimization processes, the final labeling strategy output by the strategy network is obtained;

[0182] The learning and optimization process includes:

[0183] Determining a target action selected by the strategy network based on the current labeling strategy and the current state;

[0184] Observe the new state that the labeled environment enters in response to the target action, and the reward given by the reward function for the target action;

[0185] Obtaining the value of the current state, the value of the target action, and the value of the new state calculated by the value network;

[0186] The policy network is tuned based on the value of the current state and the value of the target action, and the value network is tuned based on the value of the new state and the reward.

[0187] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0188] Clustering the feature data using an unsupervised algorithm to obtain at least one feature category, wherein the feature category includes at least one feature data;

[0189] Mark the implicit relationship corresponding to each feature category;

[0190] For any feature category, some feature data are selected from the feature category as target feature data, and triples are labeled on the target feature data based on the implicit relationship corresponding to the feature category to obtain triples labeling results corresponding to the target feature data.

[0191] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0192] Performing data preprocessing on the initial data, and extracting feature data from the preprocessed initial data to obtain feature data of the initial data;

[0193] A labeling environment is defined according to the feature data.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0195] Performing text conversion on the preprocessed initial data to obtain a high-dimensional vector space representation corresponding to the initial data;

[0196] Feature data is extracted based on the high-dimensional vector space representation to obtain feature data of the initial data, where the feature data is a feature related to the triplet annotation.

[0197] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0198] Extracting key features from the preprocessed initial data;

[0199] According to the data processing strategy corresponding to the key feature, the key feature is processed to obtain feature data of the initial data.

[0200] The present application also provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to perform the following steps:

[0201] Define the annotation environment based on the feature data corresponding to the initial data;

[0202] Obtaining target feature data from the feature data, initializing a labeling reinforcement learning model based on the target feature data and a triplet labeling result corresponding to the target feature data, and allowing the initialized labeling reinforcement learning model to interact with the labeling environment to output a final labeling strategy;

[0203] The data to be labeled is labeled using the final labeling strategy to obtain a triplet labeling result of the data to be labeled.

[0204] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0205] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A labeling method based on reinforcement learning, characterized in that: The method comprises: Define the annotation environment based on the feature data corresponding to the initial data; Obtaining target feature data from the feature data, initializing a labeling reinforcement learning model based on the target feature data and a triplet labeling result corresponding to the target feature data, and allowing the initialized labeling reinforcement learning model to interact with the labeling environment to output a final labeling strategy; The data to be labeled is labeled using the final labeling strategy to obtain a triplet labeling result of the data to be labeled.

2. The method according to claim 1, characterized in that The method further comprises: Based on the data to be labeled and the triplet labeling results of the data to be labeled, the optimization learning of the labeling reinforcement learning model is iteratively performed to output an updated final labeling strategy.

3. The method according to claim 1, characterized in that The annotation environment includes: a state space, an action space and a transition function. The state space includes historically annotated triples and the currently input text sequence; the action space includes selecting the next annotation action or transferring to the next feature data, and the next annotation action includes annotating a head entity, annotating a relationship, and annotating a tail entity; the transition function includes transferring to the next state, transferring to the next position of the feature data, and updating the annotation history.

4. The method according to claim 1, wherein The reinforcement learning model includes a policy network and a value network. The initialized annotation reinforcement learning model interacts with the annotation environment to output a final annotation strategy, including: After multiple iterative learning and optimization processes, the final labeling strategy output by the strategy network is obtained; The learning and optimization process includes: Determining a target action selected by the strategy network based on the current labeling strategy and the current state; Observe the new state that the labeled environment enters in response to the target action, and the reward given by the reward function for the target action; Obtaining the value of the current state, the value of the target action, and the value of the new state calculated by the value network; The policy network is tuned based on the value of the current state and the value of the target action, and the value network is tuned based on the value of the new state and the reward.

5. The method according to claim 4, characterized in that The reward includes positive rewards and negative rewards. The positive reward includes at least one of a basic reward, a completeness reward, and an exploration reward. The basic reward represents that the labeling action is correct, the completeness reward represents that the overall accuracy of the labeling meets the labeling requirements, and the exploration reward represents that a new implicit relationship is explored.

6. The method according to claim 1, characterized in that The acquiring target feature data from the feature data includes: Clustering the feature data using an unsupervised algorithm to obtain at least one feature category, wherein the feature category includes at least one feature data; Mark the implicit relationship corresponding to each feature category; For any feature category, some feature data are selected from the feature category as target feature data, and triples are labeled on the target feature data based on the implicit relationship corresponding to the feature category to obtain triples labeling results corresponding to the target feature data.

7. The method according to claim 1, characterized in that The defining of the annotation environment based on the feature data corresponding to the initial data includes: Performing data preprocessing on the initial data, and extracting feature data from the preprocessed initial data to obtain feature data of the initial data; A labeling environment is defined according to the feature data.

8. The method according to claim 7, characterized in that The initial data is unstructured data, and the extracting of feature data from the preprocessed initial data to obtain feature data of the initial data includes: Performing text conversion on the preprocessed initial data to obtain a high-dimensional vector space representation corresponding to the initial data; Feature data is extracted based on the high-dimensional vector space representation to obtain feature data of the initial data, where the feature data is a feature related to the triplet annotation.

9. The method according to claim 7, characterized in that The initial data is structured data, and the extracting of feature data from the pre-processed initial data to obtain feature data of the initial data includes: Extracting key features from the preprocessed initial data; According to the data processing strategy corresponding to the key feature, the key feature is processed to obtain feature data of the initial data.

10. A labeling device based on reinforcement learning, characterized in that: The device comprises: The annotation environment definition module is used to define the annotation environment based on the feature data corresponding to the initial data; a labeling reinforcement learning model module, configured to initialize a labeling reinforcement learning model based on target feature data and triplet labeling results corresponding to the target feature data, and to enable the initialized labeling reinforcement learning model to interact with the labeling environment to output experience data; A labeling training and optimization module, configured to perform optimization training on the labeling reinforcement learning model based on the empirical data provided by the labeling reinforcement learning model module; The annotation reinforcement learning model module is further used to output a final annotation strategy based on the optimized and trained annotation reinforcement learning model; The implicit relationship discovery module is used to label the data to be labeled using the final labeling strategy to obtain triple labeling results of the data to be labeled.

11. The device according to claim 10, characterized in that The labeling enhancement learning model module is further used to iteratively perform optimization learning of the labeling enhancement learning model based on the data to be labeled and the triple labeling results of the data to be labeled.

12. The device according to claim 10, characterized in that The device further comprises: The labeling reward function module is used to give corresponding rewards for the labeling results of the labeling reinforcement learning model. The rewards include positive rewards and negative rewards. The positive rewards include at least one of basic rewards, integrity rewards, and exploration rewards. The basic reward indicates that the labeling action is correct, the integrity reward indicates that the overall accuracy of the labeling meets the labeling requirements, and the exploration reward indicates that a new implicit relationship is explored.

13. The device according to claim 10, characterized in that The device further comprises: The labeling module is used to learn the category of the implicit relationship from the feature data, and label the target feature data based on the category of the implicit relationship to obtain a triple labeling result of the target feature data.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images