Intelligent labeling method and system based on reinforcement learning agent

Abstract

The invention provides an intelligent labeling method and system based on a reinforcement learning agent, and belongs to the technical field of natural language processing. Comprising the following steps: performing data preprocessing and labeling on multi-source text data to obtain a label data set, and training a BGE semantic retrieval model; recalling a preset number of candidate tags from a tag library by using a BGE semantic retrieval model; combining the candidate tags and the multi-source text data into Prompt data according to a Prompt format, performing probability prediction on each tag by using a large model, and outputting a probability value; dynamically adjusting a label threshold based on a reinforcement learning agent method; and comparing the label threshold value with the probability value to determine a final label output result. According to the method, on the basis of adapting to a complex business scene, rapid, efficient and accurate label labeling of the text content can be realized, so that the quality and efficiency of text content management and distribution are improved.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing technology, and in particular relates to an intelligent labeling method and system based on reinforcement learning agents. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In today's era of information explosion, the internet has accumulated a vast amount of text data, such as questions and answers, articles, and ideas in question-and-answer communities like Zhihu. Effectively understanding and characterizing this textual content, and thus achieving high-quality distribution, is key to improving user experience. Content tagging, a crucial step in text understanding, enables refined content operations and precise characterization of user interests by adding accurate, appropriate, and rich semantic tags to text, thereby promoting efficient matching of content with users.

[0004] Currently, mainstream text multi-label classification methods are usually based on pre-trained models with fine-tuning of parameters. However, when faced with massive label systems, traditional methods expose many problems, such as:

[0005] (1) The training cycle is lengthy. Due to the huge number of labels, a large number of labeled samples are required, which leads to slow model convergence and a sharp increase in sample labeling and training costs. At the same time, there is a lag after parameter adjustment, which cannot meet the real-time processing requirements of massive dynamic label systems.

[0006] (2) The generalization performance of the model is poor. It has a good fitting effect on the head label, but the accuracy of the middle and tail labels is low and difficult to improve effectively; and it cannot adapt to complex business scenarios.

[0007] (3) It is difficult to iterate the labeling system. Every time a new batch of labels is added, a large number of samples need to be re-labeled and the model needs to be trained, and the model performance on the new labels cannot be guaranteed. At the same time, traditional methods often encode labels in a one-hot form, which cannot fully explore the semantic information of the labels, resulting in insufficient accuracy. Summary of the Invention

[0008] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention provides an intelligent labeling method and system based on reinforcement learning agents, which can realize fast, efficient and accurate labeling of text content on the basis of adapting to complex business scenarios, thereby improving the quality and efficiency of text content management and distribution.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] The first aspect of the present invention provides an intelligent labeling method based on reinforcement learning agent.

[0011] An intelligent labeling method based on reinforcement learning agent, comprising:

[0012] Collect multi-source text data;

[0013] Perform data preprocessing and annotation on the obtained multi-source text data to obtain a labeled dataset;

[0014] The obtained label dataset is used to train the BGE semantic retrieval model; and the trained BGE semantic retrieval model is used to recall a predetermined number of candidate labels from the label library;

[0015] Combining the candidate tags and multi-source text data into Prompt data in Prompt format, and inputting the Prompt data into a large model to perform probability prediction on each tag and output a probability value;

[0016] The label threshold is dynamically adjusted based on the reinforcement learning agent method; the obtained label threshold is compared with the probability value to determine the final label output result.

[0017] Furthermore, the data preprocessing includes data cleaning, format conversion and data screening.

[0018] Furthermore, before using the labeled dataset to train the BGE semantic retrieval model, the labeled dataset is first divided into a training set and a validation set; and stratified sampling technology is combined to maintain the similarity of the distribution of data in the training set and the validation set on various labels.

[0019] Furthermore, the recall number K of the candidate tags is determined based on the business scenario, and the top K tags with the highest similarity to the multi-source text data are selected from all tags in the tag library as candidate tags.

[0020] Furthermore, the Prompt format combination is scenario-dependent, that is, different Prompt format combinations are used in different scenarios. Specifically:

[0021] In e-commerce scenarios, the prompt format is: {Product Description: [Text Content], Candidate Tags: [Tag 1, Tag 2, ..., Tag 10], Please select the most appropriate tag, Tag Explanation: [Brief explanation of each tag]};

[0022] In the news scenario, the prompt format is: {news content: [text content], candidate tags: [topic tag 1, topic tag 2, ...], please determine the topic of the news, brief description of the news event: [a brief description of the core event of the news]}.

[0023] Furthermore, the label threshold is dynamically adjusted based on the reinforcement learning agent method, including: first, constructing a spatial environment, that is, defining the state space and action space, and designing a reward function; then, using the Q-learning algorithm for optimization and setting optimization rules; under the set optimization rules, the label threshold is dynamically adjusted.

[0024] Furthermore, the reward function includes three levels of rewards, namely core goal rewards, long-term stability rewards and final rewards.

[0025] A second aspect of the present invention provides an intelligent labeling system based on reinforcement learning agents.

[0026] An intelligent labeling system based on reinforcement learning agents, comprising:

[0027] The data collection module is configured to: collect multi-source text data;

[0028] The data preprocessing module is configured to: perform data preprocessing and labeling on the obtained multi-source text data to obtain a labeled data set;

[0029] The recall module is configured to: train the BGE semantic retrieval model using the obtained label dataset; and recall a predetermined number of candidate labels from the label library using the trained BGE semantic retrieval model;

[0030] The large model prediction module is configured to: combine the candidate tags and multi-source text data into Prompt data according to the Prompt format, and input the Prompt data into the large model to perform probability prediction for each tag and output a probability value;

[0031] The label selection module is configured to: dynamically adjust the label threshold based on the reinforcement learning agent method; compare the obtained label threshold with the probability value to determine the final label output result.

[0032] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the intelligent labeling method based on reinforcement learning agents as described in the first aspect of the present invention.

[0033] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, it implements the steps of the intelligent labeling method based on reinforcement learning agent as described in the first aspect of the present invention.

[0034] One or more of the above technical solutions have the following beneficial effects:

[0035] (1) The present invention combines the semantic retrieval model with the big model. The semantic retrieval model is fine-tuned on the annotated dataset to achieve semantic alignment of text and labels, greatly reducing the required number of annotated samples, thereby shortening the model convergence time and reducing the sample annotation and training costs. At the same time, the input text and the recalled candidate labels are combined using Prompt engineering technology and then input into the big model, which can accurately screen out the correct label that best matches the input text in real time, effectively meeting the real-time processing requirements of the massive dynamic label system and solving the problem of lagging parameter adjustment in traditional methods. In addition, the present invention dynamically adjusts the label threshold based on the reinforcement learning agent method, and can also respond to changes in data distribution in real time, further avoiding parameter adjustment lag.

[0036] (2) The present invention uses a label dataset to train the BGE semantic retrieval model, and uses the trained BGE semantic retrieval model to recall a predetermined number of candidate labels from the label library. During the recall process, the number of candidate labels to be recalled is determined based on the business scenario. At the same time, to address the problem of false negatives within a batch, a mask matrix with the same shape as the sample matrix is constructed to improve the loss function. This not only improves the fitting ability for mid-tail labels, but also ensures that the large model can efficiently and accurately screen out correct labels, allowing the model to exhibit good generalization performance in different business scenarios.

[0037] (3) The present invention proposes a solution for enriching the semantic information of labels and supports the dynamic update of the label system. On the one hand, adding detailed explanatory information to the labels solves the problem of inaccurate recall caused by semantic ambiguity of labels and improves the accuracy of label annotation. On the other hand, when a batch of new labels are added, there is no need to re-annotate a large number of samples and train the model. Only the semantic information of the new labels needs to be input into the system, so that the model can quickly adapt to the new labels, ensuring the model performance on the new labels.

[0038] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0040] Figure 1 This is a flowchart of an intelligent labeling method based on reinforcement learning agent in Example 1 of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0042] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0043] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0044] Example 1

[0045] This embodiment discloses an intelligent labeling method based on reinforcement learning agents.

[0046] like Figure 1 As shown in FIG, an intelligent labeling method based on reinforcement learning agent includes:

[0047] Step S1, collecting multi-source text data;

[0048] Step S2: preprocess and label the obtained multi-source text data to obtain a labeled data set;

[0049] Step S3: using the obtained tag dataset to train the BGE semantic retrieval model; and using the trained BGE semantic retrieval model to recall a predetermined number of candidate tags from the tag library;

[0050] Step S4: combining the candidate tags and multi-source text data into Prompt data according to the Prompt format, and inputting the Prompt data into the large model to perform probability prediction on each tag and output the probability value;

[0051] Step S5: Dynamically adjust the label threshold based on the reinforcement learning agent method; compare the obtained label threshold with the probability value to determine the final label output result.

[0052] Based on the above process, the present invention can achieve rapid, efficient, and accurate tagging of text content while adapting to complex business scenarios, thereby improving the quality and efficiency of text content management and distribution. To facilitate understanding of the technical solution of the present invention, the specific implementation method of the technical solution of the present invention is further explained and illustrated below.

[0053] In step S1, multi-source text data is collected.

[0054] Acquire text data from multiple data sources (i.e., multi-source text data). For example, for e-commerce companies, in addition to querying product details and user review tables in the company's internal database, they can also integrate customer service chat records, after-sales feedback documents, etc.

[0055] As an optional embodiment, for e-commerce platform business scenarios, you can write SQL query statements to filter out product-related text data with sales exceeding 100 units and more than 5 user reviews in the past three months to ensure data richness and representativeness.

[0056] As an optional implementation, news media organizations can use compliant web crawler technology to collect news reports from designated "technology" and "finance" channels on news websites. They can also subscribe to professional news information databases to obtain authoritative and exclusive news content. Furthermore, they can monitor news discussions under trending topics on social media platforms, broadening their news data collection channels and capturing news updates from all angles.

[0057] In step S2, data preprocessing and annotation are performed on the multi-source text data to obtain a labeled dataset. Data preprocessing includes data cleaning, format conversion, and data screening, which can be achieved through the following methods.

[0058] A. Data cleaning: Use multi-dimensional technical means to deeply clean the collected data.

[0059] As an optional embodiment, Python's re library can be used to write regular expressions, which can not only remove promotional symbols (such as "￥", "!", etc.) in product descriptions and special characters such as copyright statements in news reports, but also accurately identify and remove interference information such as emoticons and out-of-order characters in the text.

[0060] As an optional embodiment, the powerful BeautifulSoup library can be used to perform in-depth parsing of HTML tags in the text to ensure that the plain text content is completely extracted. <div class＝"article">This is a news article containing <em> Key Vocabulary< / em> } accurately converts it to {"This is a news article containing key words"}. To address the issue of garbled text, we can introduce the encoding detection library chardet to automatically identify the text encoding format and try decoding different encoding formats, such as UTF-8 and GB2312, to ensure data readability and integrity.

[0061] B. Format conversion: If the collected data is in complex formats, such as product data that may be in CSV, JSON, XLSX, and news releases that may be in PDF, XML, DOCX, etc., you can use professional tools for precise format conversion.

[0062] As an optional embodiment, for commodity data in CSV format, Python's pandas library can be used to read the data, and then the flexible to_csv method can be used to convert it into a plain text format TXT that is easy for the system to process, and each text data is stored according to the row specification. For news releases in PDF format, the PyPDF2 library can be combined with the pdfminer.six library to optimize the text extraction process to improve the extraction accuracy and save it as a TXT file. For complex DOCX format news releases, the python-docx library can be used to parse the document structure, extract the text content and convert it into TXT format, laying a solid foundation for subsequent processing.

[0063] C. Data screening: Perform refined data screening based on business needs.

[0064] For example, in the business scenario of e-commerce companies, database query statements such as SELECT * FROM products WHERE category = 'electronics' AND brand IN ('Apple', 'Samsung') can be used to accurately filter product text data for a specific brand within the electronics category, excluding data from irrelevant categories and brands.

[0065] Taking the business scenario of news media organizations as an example, they can classify news by topic, using deep learning-based text classification algorithms such as convolutional neural networks (CNN) or recurrent neural networks (RNN), combined with word vector representations of news text, to filter out news text on specific topics (such as "politics," "culture," and "entertainment," etc.) and remove text that does not meet the topic requirements. At the same time, data quality can be improved by setting up keyword blacklists, such as excluding news text related to advertising promotions, repetitive news, and low-quality news.

[0066] In step S3, the BGE semantic retrieval model is trained using the tag dataset; and a predetermined number of candidate tags are recalled from the tag library using the trained BGE semantic retrieval model.

[0067] This embodiment uses the pre-trained BGE semantic retrieval model to perform targeted fine-tuning on a carefully constructed label dataset. During the fine-tuning process, in order to optimize the loss between the input text and the label, the Zhihu high-quality corpus was used, and the negative sampling method within the batch was applied. This method can effectively increase the diversity of training data, allowing the model to better learn the semantic relationship between text and labels, thereby achieving semantic alignment of text and labels. On this basis, the model can accurately recall candidate labels that are semantically related to the input text from a massive label library. Among them, the specific type of semantic retrieval model can be selected according to actual needs. It can be specifically achieved by the following methods:

[0068] Step S3-1: Data set division.

[0069] Before training the BGE semantic retrieval model using a labeled dataset, the labeled dataset must be divided into a training set and a validation set. Stratified sampling techniques are then used to maintain similarity in the distribution of labels between the training and validation sets. Specifically, the labeled dataset is first split into a training set and a validation set in an 8:2 ratio. Then, the train_test_split function in the Python sklearn library is used, combined with stratified sampling techniques, to ensure that the distribution of labels between the training and validation sets is similar, thus preventing data bias from affecting model training. This can be achieved using the stratify parameter of the train_test_split function in the Python sklearn library. First, the parameter settings are entered, using the label column y in the labeled dataset as the stratification basis. Next, the stratification logic is set. By setting test_size = 0.2 (20% for the validation set), the function proportionally samples the labels in the training and validation sets based on the category distribution of the label column, ensuring that the proportion of each category label in the two sets is consistent with that in the original dataset.

[0070] For example, for a labeled dataset containing 10,000 text data and an uneven label distribution, stratified sampling is used to ensure that the proportion of each type of label in the training set and validation set is close to that of the original dataset. After division, the training set has approximately 8,000 data items and the validation set has approximately 2,000 data items.

[0071] Step S3-2: Model initialization.

[0072] In this embodiment, the BGE model (BAAI General Embedding vector model) with outstanding performance in the field of natural language processing is selected. The BGE model is based on the pre-trained Transformer architecture and uses InfoNCE Loss or MarginLoss to carry out comparative learning and multi-task learning. By loading the weights and configuration files of the pre-trained model, the parameters and structure of the model are carefully initialized. For example, in Python's transformers library, the pre-trained weights of the BGE model can be loaded using the AutoModel.from_pretrained method, such as model = AutoModel.from_pretrained('bge-large-en'), and the model architecture can be fine-tuned according to actual needs, such as adjusting the number of hidden layers, the number of neurons, etc., to fully prepare for subsequent training.

[0073] Step S3-3: model training.

[0074] As an optional embodiment, the key training parameters can be set as follows: the learning rate is initially set to 0.001, and it is passed as a parameter to the optimizer constructor, such as optimizer = AdamW(model.parameters(), lr = 0.001, betas = (0.9, 0.999), eps = 1e-08, weight_decay = 0.01), to fine-tune the optimizer hyperparameters; wherein, this embodiment uses AdamW as the optimizer. The number of iterations is first set to 50 times. During the training process, the number of iterations is adjusted in time according to the performance of the model on the validation set, such as accuracy, loss value, F1 value and other multi-dimensional indicators. If the accuracy of the validation set stabilizes after 30 iterations and does not meet expectations, the number of iterations can be increased to 80 times, and the learning rate can be appropriately reduced, such as adjusted to 0.0001, to balance the model convergence speed and accuracy. The batch size is set to 32 and can be dynamically adjusted based on the server's memory and computing resources. If the server has sufficient memory and strong computing performance, the batch size can be appropriately increased to 64 or even 128 to improve training efficiency. However, close attention should be paid to model training stability to prevent problems such as memory overflow.

[0075] During the training process, the semantic retrieval model is deeply iteratively trained using the training set data. As an optional embodiment, during each training, data is taken out from the training set in batches. For example, each batch of data contains 32 texts and their corresponding labels. The loss is calculated using the intra-batch negative sampling method. In each batch, some labels that do not match the current text are randomly selected as negative samples, and participate in the loss calculation together with the positive samples to enhance the model's ability to distinguish between positive and negative samples. To address the problem of false negatives within the batch, a mask matrix of the same shape as the sample matrix is constructed. When calculating the loss, the positions (i, j and j, i) where sample i and sample j have the same label are set to 0; then, the loss matrix and the mask matrix are multiplied to obtain the final masked loss matrix for loss calculation. The automatic differentiation function of the deep learning framework (such as PyTorch) is used to adjust the model parameters, such as weights and biases, based on the loss value backpropagation. After each parameter update, it is determined whether the current iteration is completed. Once the iteration is complete, the model performance is evaluated on the validation set, comparing the model's predictions with the true labels to calculate metrics such as accuracy, recall, and F1 score. If the model performance meets the expected goals, such as validation set accuracy exceeding 90% and an F1 score exceeding 0.85, the trained semantic retrieval model is saved. If the goals are not met, the training parameters are readjusted, such as adjusting optimizer hyperparameters, changing activation functions, and increasing the amount of training data, and training continues until the model performance meets the requirements.

[0076] Step S3-4: recall candidate labels.

[0077] The number K of candidate tags to be recalled is determined based on the business scenario, that is, the top K tags with the highest similarity to the multi-source text data are selected from all the tags in the tag library as candidate tags. Specifically, when new text data is input into the system, it is passed into the trained semantic retrieval model. The model calculates the similarity between the text (i.e., multi-source text data) and all the tags in the tag library, and recalls K candidate tags in order of similarity from high to low. According to different business scenarios and preliminary experimental tests, the appropriate K value is determined. For example, in this embodiment, in the e-commerce product tag generation scenario, it was found through multiple experiments that the best effect is achieved when the K value is 10, that is, the 10 candidate tags that are most semantically relevant to the product description text are recalled.

[0078] The similarity between the text and the labels in the tag library is calculated based on the semantic retrieval model. Specifically: first, vector representation of the text and the label is performed, that is, text encoding and label encoding. Among them, text encoding is: the input text data (such as product description, news content) is encoded through the BGE semantic retrieval model to generate a text vector of fixed length; label encoding is: each label in the tag library (such as "thin notebook" and "technology news") is also encoded into a label vector through the BGE model. Subsequently, the similarity calculation can be performed. In this embodiment, the cosine similarity is calculated between the text vector and the label vector. Finally, all the labels in the tag library are sorted in descending order according to the similarity value, and the top K labels with the highest similarity are selected as candidate labels.

[0079] In step S4, the candidate tags and multi-source text data are combined into Prompt data in the Prompt format, and the Prompt data is input into the large model to perform probability prediction for each tag and output the probability value. This can be achieved through the following process:

[0080] Step S4-1: Prompt format combination.

[0081] This invention uses advanced Prompt engineering technology to combine the input text and the recalled candidate tags in a specific format and logic as the input content of the large model (such as the Zhihaitu AI large model). In this way, the large model is fine-tuned so that it can accurately filter out the correct tag that best matches the input text from the candidate tag set based on the input information. In addition, in order to further improve the performance of the model, the present invention has been optimized, specifically:

[0082] In order to improve the accuracy of label recall, the present invention also provides a method for enriching label semantic information. By adding detailed explanatory information to the label, the problem of inaccurate recall caused by label semantic ambiguity is effectively solved. Moreover, this method can also well support the dynamic update of the label system. When the label system changes, only the newly added or modified label explanation information needs to be adjusted accordingly, and the model can quickly adapt to the new label system without the need for large-scale retraining. When adding detailed explanatory information to the label, first, a large model (such as the Zhihaitu AI large model) is used to automatically generate a summary of the label. For example, enter the label name into the large model, and generate a preliminary explanation through Prompt (such as "Please briefly explain the definition and applicable scenarios of the label 'thin notebook'"), and then manually review and correct it. Subsequently, the label is stored and it is allowed to add / modify the label explanation field through the API interface or management interface to ensure that it is updated in sync with business needs.

[0083] Furthermore, the prompt format combination is scenario-dependent, that is, different prompt format combinations are used in different scenarios. Specifically:

[0084] In e-commerce scenarios, the prompt format is: {Product Description: [Text Content], Candidate Tags: [Tag 1, Tag 2, ..., Tag 10], Please select the most appropriate tag, Tag Explanation: [Brief explanation of each tag]}. For example, for the product description "A thin and portable laptop computer suitable for office work and daily entertainment," the candidate tags are "thin and portable notebook," "office computer," "entertainment notebook," etc. The combined prompt would be "Product Description: A thin and portable laptop computer suitable for office work and daily entertainment, Candidate Tags: [thin and portable notebook, office computer, entertainment notebook, ...], Please select the most appropriate tag, Tag Explanation: [Thin and portable notebook: A lightweight, portable laptop computer; office computer: A computer primarily used in office scenarios; entertainment notebook: A computer with a focus on entertainment functions]."

[0085] In the news scenario, the prompt format is: {News content: [text content], Candidate tags: [Topic tag 1, Topic tag 2, ...], Please determine the topic of the news, News event summary: [A brief description of the core event of the news]}. By enriching the prompt content, the accuracy of the large model's tag screening can be improved.

[0086] Step S4-2: Large model processing.

[0087] The combined Prompt content is input into the big model. In this embodiment, the Zhihaitu AI big model is selected as the big model. The training of the Zhihaitu AI big model is based on the CPM enterprise-level big model and ModelForce big model system independently developed by Mianbi Intelligence. It can quickly extract elements, sort out opinions and aggregate content, and finally present the outline of all answers to the user. Specifically, the big model makes a probability prediction for each label, and obtains the probability value of each label by comprehensively calculating the predicted probability of each token in the label. For example, the big model outputs results such as "thin notebook, probability: 0.8; office computer, probability: 0.65; entertainment notebook, probability: 0.7".

[0088] In step S5, the label threshold is dynamically adjusted based on the reinforcement learning agent method; the label threshold is compared with the probability value to determine the final label output result. This can be achieved through the following process:

[0089] Step S5-1, dynamically adjust the label threshold based on the reinforcement learning agent method.

[0090] First, we construct a spatial environment, defining the state space and action space, and designing a reward function. Next, we use the Q-learning algorithm for optimization and set optimization rules. Under these optimization rules, we dynamically adjust the label threshold. The reward function consists of three levels of rewards: core objective rewards, long-term stability rewards, and final rewards. Specifically:

[0091] ①State space definition

[0092] The state space is represented by states and state encodings; states include basic indicators, dynamic indicators, and environmental feedback. Furthermore, the accuracy, recall, and F1 score of the current batch of labels serve as basic indicators; the label distribution entropy (a measure of label diversity) and the real-time ratio of the number and quality of candidate labels serve as dynamic indicators; and the user click rate on historical labels and the correction rate of manual review serve as environmental feedback.

[0093] Basic indicators, dynamic indicators and environmental feedback are normalized into multi-dimensional vectors through state encoding to construct the state space, namely:

[0094]

[0095] Among them, s t Represents the current state under normalized situation, P t represents the accuracy, R t represents the recall rate, F1 represents the F1 value, H dist represents the label distribution entropy, N valid Indicates the number of tags that meet the preset quality standards, N totalIndicates the total number of labels in the current batch. t 、R t , F1, H dist All of them can be calculated using existing technologies, so this embodiment will not elaborate on them in detail. In actual implementation, they need to be encoded into multi-dimensional vectors using predefined rules.

[0096] ②Action space definition

[0097] The action space is represented by the threshold adjustment direction and dynamic step size control; the threshold adjustment direction includes the adjustment direction of discrete actions and continuous actions; the dynamic step size control is adaptively adjusted according to the state entropy, for example, the adjustment amplitude can be increased in high entropy states (label confusion).

[0098] ③Reward function design

[0099] The reward function includes three levels of rewards, namely core target rewards, long-term stability rewards, and final rewards.

[0100] A. Core target rewards are:

[0101] R core =α*ΔP+β*ΔR-γ*Redundancy;

[0102] Among them, α represents the weight coefficient used to adjust the contribution of the precision change ΔP, β represents the weight coefficient used to adjust the contribution of the recall change ΔR, and γ represents the weight coefficient used to adjust the suppression strength of the long-term stability penalty term Redundancy.

[0103] B. Long-term stability rewards prevent threshold fluctuations by introducing a penalty term for indicator variance within the sliding window. Specifically, the long-term stability reward is:

[0104] Redundancy=-μ*Var(P t-10:t ,R t-10:t );

[0105] Among them, μ represents the weight coefficient used to adjust the strength of the variance penalty term, P t-10:t and R t-10:t Represent the accuracy and recall rate in the past t time step respectively; Var() represents the joint variance of the accuracy and recall rate in the past 10 time steps. t-10:t ,R t-10:t ), by default, the variance is calculated using data from the past 10 time steps. In practical applications, the window size can be adjusted through hyperparameters.

[0106] C. The final reward is:

[0107] R t=R core +R stable ;

[0108] Among them, R stable Represents the long-term stability reward value, which is the opposite of Redundancy.

[0109] ④Q-learning algorithm optimization

[0110] The Q-learning algorithm uses a deep Q network (DQN) to solve high-dimensional state space problems. Its network architecture includes a three-layer fully connected neural network, with the state vector as input and the Q value of each action as output; by storing historical transfers (s t ,a t ,R t ,s t+1 ) to the buffer pool, randomly sample to break data correlation; and use an independent target network to calculate maxQ(s t+1 ,a ′ ), to prevent the Q value from being over-estimated, that is:

[0111]

[0112] Among them, Q(s t ,a t ) represents the Q value of the current state-action pair, that is, the current state s t Next, perform action a t The expected long-term cumulative reward is used to guide the agent to choose the optimal action (select the action with the largest Q value); t+1 represents the next state. η represents the learning rate, which ranges from [0,1] and is used to control the influence of new information on the Q value update. Represents the discount factor, with a value range of [0,1], which is used to attenuate the weight of future rewards; Indicates the maximum Q value of the next state. During the training process, the Q-learning algorithm selects the optimal action (i.e., the optimal ΔT) for each state (such as the accuracy, recall rate, label distribution entropy, etc. of the current state), that is:

[0113]

[0114] Among them, sign() indicates that the adjustment direction is determined according to the positive or negative maximum Q value of the current state (positive → increase, negative → decrease); step_size(H state ) represents the state entropy H state Dynamic step size for calculations.

[0115] On this basis, the label threshold can be updated in each iteration according to the selected action, that is:

[0116] T new =T current +ΔT;

[0117] Among them, T current Indicates the current threshold. If there is no current threshold, you can also set it yourself. T new Indicates the updated label threshold.

[0118] After obtaining a more valuable updated label threshold, the resulting label threshold is compared with the actual probability value to determine the final label output result. For example, when the resulting label threshold is 0.5, labels with probabilities greater than the label threshold of 0.5 are filtered out and used as the final output label. In the above example, if the probabilities of "thin notebook" and "entertainment notebook" are greater than 0.5, they will be filtered out as the final labels, completing the label generation process.

[0119] Example 2

[0120] This embodiment discloses an intelligent labeling system based on reinforcement learning agents.

[0121] An intelligent labeling system based on reinforcement learning agents, comprising:

[0122] The data collection module is configured to: collect multi-source text data;

[0123] The data preprocessing module is configured to: perform data preprocessing and labeling on the obtained multi-source text data to obtain a labeled data set;

[0124] The recall module is configured to: train the BGE semantic retrieval model using the obtained label dataset; and recall a predetermined number of candidate labels from the label library using the trained BGE semantic retrieval model;

[0125] The large model prediction module is configured to: combine the candidate tags and multi-source text data into Prompt data according to the Prompt format, and input the Prompt data into the large model to perform probability prediction for each tag and output a probability value;

[0126] The label selection module is configured to: dynamically adjust the label threshold based on the reinforcement learning agent method; compare the obtained label threshold with the probability value to determine the final label output result.

[0127] Example 3

[0128] The purpose of this embodiment is to provide a computer-readable storage medium.

[0129] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an intelligent labeling method based on a reinforcement learning agent as described in the first embodiment of the present disclosure.

[0130] Example 4

[0131] The purpose of this embodiment is to provide an electronic device.

[0132] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the intelligent labeling method based on reinforcement learning agent as described in the first embodiment of the present disclosure are implemented.

[0133] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0134] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0135] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. An intelligent labeling method based on reinforcement learning agent, characterized in that: include: Collect multi-source text data; Perform data preprocessing and annotation on the obtained multi-source text data to obtain a labeled dataset; The obtained label dataset is used to train the BGE semantic retrieval model; and the trained BGE semantic retrieval model is used to recall a predetermined number of candidate labels from the label library; Combining the candidate tags and multi-source text data into Prompt data in Prompt format, and inputting the Prompt data into a large model to perform probability prediction on each tag and output a probability value; The label threshold is dynamically adjusted based on the reinforcement learning agent method; the obtained label threshold is compared with the probability value to determine the final label output result.

2. The intelligent labeling method based on reinforcement learning agent according to claim 1, characterized in that: The data preprocessing includes data cleaning, format conversion and data screening.

3. The intelligent labeling method based on reinforcement learning agent according to claim 1, characterized in that: Before using the labeled dataset to train the BGE semantic retrieval model, the labeled dataset is first divided into a training set and a validation set; It is combined with stratified sampling technology to maintain the similarity of the distribution of data in the training set and the validation set on various labels.

4. The intelligent labeling method based on reinforcement learning agent according to claim 1, characterized in that: The recall number K of the candidate tags is determined based on the business scenario, and the top K tags with the highest similarity to the multi-source text data are selected from all the tags in the tag library as candidate tags.

5. The intelligent tagging method based on reinforcement learning agent according to claim 1, characterized in that: The Prompt format combination is scenario-dependent, that is, different Prompt format combinations are used in different scenarios. Specifically: In e-commerce scenarios, the prompt format is: {Product Description: [Text Content], Candidate Tags: [Tag 1, Tag 2, ..., Tag 10], Please select the most appropriate tag, Tag Explanation: [Brief explanation of each tag]}; In the news scenario, the prompt format is: {news content: [text content], candidate tags: [topic tag 1, topic tag 2, ...], please determine the topic of the news, brief description of the news event: [a brief description of the core event of the news]}.

6. The intelligent labeling method based on reinforcement learning agent according to claim 1, characterized in that: Dynamically adjust the label threshold based on the reinforcement learning agent method, It includes: first, building a spatial environment, that is, defining the state space and action space, and designing the reward function; Subsequently, the Q-learning algorithm is used for optimization and optimization rules are set; under the set optimization rules, the label threshold is dynamically adjusted.

7. The intelligent tagging method based on reinforcement learning agent according to claim 6, characterized in that: The reward function includes three levels of rewards, namely core target rewards, long-term stability rewards and final rewards.

8. An intelligent labeling system based on reinforcement learning agent, characterized in that: include: The data collection module is configured to: collect multi-source text data; The data preprocessing module is configured to: perform data preprocessing and labeling on the obtained multi-source text data to obtain a labeled data set; The recall module is configured to: train the BGE semantic retrieval model using the obtained label dataset; and recall a predetermined number of candidate labels from the label library using the trained BGE semantic retrieval model; The large model prediction module is configured to: combine the candidate tags and multi-source text data into Prompt data according to the Prompt format, and input the Prompt data into the large model to perform probability prediction for each tag and output a probability value; The label selection module is configured to: dynamically adjust the label threshold based on the reinforcement learning agent method; compare the obtained label threshold with the probability value to determine the final label output result.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the intelligent labeling method based on reinforcement learning agent as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the intelligent labeling method based on reinforcement learning agent as described in any one of claims 1 to 7 are implemented.

Images