Label marking method, device, equipment, storage medium and computer program product

Abstract

The application relates to the technical field of big data, and discloses a label labeling method, device, equipment, storage medium and computer program product, the method comprising the following steps: in response to an input target to be labeled, a preset labeling model is called, wherein a multi-path recall module and a preset large language model are arranged in the preset labeling model; a multi-path recall is performed on the preset label library based on the target to be labeled through the multi-path recall module, a candidate label set corresponding to the target to be labeled is obtained, the target to be labeled and the candidate label set are input into the preset large language model, a target label set output by the preset large language model is obtained, the target to be labeled is labeled based on the target label set, and a labeled target is obtained; since the multi-path recall module is used to perform a multi-path recall on the preset label library, a small-scale candidate set is efficiently screened from a full-scale label library, and the large language model is used to make accurate generative decisions on the candidate set, so that the efficiency and precision of label labeling are improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a labeling method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] In the era of massive information, data is growing explosively. To effectively manage, retrieve, and analyze this data, it needs to be structured, with tagging being a core component. Tagging aims to associate unstructured content with predefined, semantically clear tags. However, existing tagging methods suffer from low efficiency and inaccuracy when dealing with large-scale tags. Summary of the Invention

[0003] The main objective of this application is to provide a labeling method, apparatus, device, storage medium, and computer program product, which addresses the technical problems of low labeling efficiency and inaccurate labeling in existing labeling methods when dealing with large-scale labels.

[0004] To achieve the above objectives, this application provides a labeling method, the labeling method comprising:

[0005] In response to the input target to be labeled, a preset labeling model is invoked, wherein the preset labeling model is equipped with a multi-path recall module and a preset large language model;

[0006] Based on the target to be labeled, the multi-way recall module performs multi-way recall in the preset tag library to obtain the candidate tag set corresponding to the target to be labeled;

[0007] Input the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model;

[0008] The target to be labeled is labeled based on the target label set to obtain the labeled target.

[0009] Optionally, the step of inputting the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model includes:

[0010] Based on the target to be labeled, the candidate tag set, and the preset prompt word template, a tag filtering prompt word is constructed, wherein the candidate tag set includes candidate tags and the tag definitions corresponding to the candidate tags;

[0011] Input the tag filtering prompts into the preset large language model to obtain the target tag set output by the preset large language model.

[0012] Optionally, the step of obtaining a candidate tag set corresponding to the target to be labeled by performing multi-way recall in a preset tag library based on the target to be labeled through the multi-way recall module includes:

[0013] Based on the target to be labeled, the multi-channel recall module performs multi-channel recall in the preset tag library to obtain multi-channel recall results;

[0014] The multi-path recall results are merged to obtain the candidate tag set corresponding to the target to be labeled.

[0015] Optionally, the multi-path recall results include: keyword recall results, similarity recall results, and split recall results; the step of performing multi-path recall in a preset tag library based on the target to be labeled, using the multi-path recall module to obtain multi-path recall results, includes:

[0016] The multi-path recall module extracts key information from the target to be labeled and matches the key information with a preset tag library to obtain keyword recall results.

[0017] The target to be labeled and the preset tag library are transformed into vectors to obtain a first vector and a second vector, and the similarity index between the first vector and the second vector is calculated to obtain the similarity recall result.

[0018] The preset tag library is split into multiple sub-tag sets according to preset rules, and the tags corresponding to the target to be labeled are recalled in the multiple sub-tag sets based on the preset large language model to obtain the splitting and recall results.

[0019] Optionally, the step of merging the multi-path recall results to obtain the candidate tag set corresponding to the target to be labeled includes:

[0020] Calculate the number of times each tag appears repeatedly in the multi-path recall results;

[0021] Based on the number of recalls, the multi-path recall results are merged to obtain the candidate tag set corresponding to the target to be labeled.

[0022] Optionally, the preset annotation model further includes a preliminary screening module. Before inputting the target to be annotated and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model, the method further includes:

[0023] The candidate tag set is initially filtered through the preliminary filtering module to obtain the filtered tag set;

[0024] Accordingly, the step of inputting the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model includes:

[0025] The target to be labeled and the filtered label set are input into the preset large language model to obtain the target label set output by the preset large language model.

[0026] Optionally, the step of performing preliminary screening on the candidate tag set through the preliminary screening module to obtain a filtered tag set includes:

[0027] The preliminary screening module extracts quantitative indicators for each tag from the candidate tag set. The quantitative indicators include the number of times each tag is retrieved in the multi-way recall, the recall score corresponding to each recall strategy, and the confidence value output by the strategy.

[0028] The candidate tag set is initially screened based on the quantitative indicators to obtain the screened tag set.

[0029] Optionally, before invoking the preset annotation model in response to the input target to be annotated, the method further includes:

[0030] Analyze the key hyperparameters that affect the performance of the annotation model, and define the value range of the key hyperparameters;

[0031] Within the range of values, a target parameter combination is searched using a preset hyperparameter search algorithm;

[0032] Based on the target parameter combination, the initial annotation model is configured to obtain the preset annotation model.

[0033] Optionally, the step of searching for target parameter combinations within the value range using a preset hyperparameter search algorithm includes:

[0034] Within the range of values, candidate parameter combinations are searched using a preset hyperparameter search algorithm;

[0035] Based on the candidate parameter combinations, the initial annotation model is configured to obtain a candidate annotation model;

[0036] The target parameter combination is determined based on the candidate annotation model.

[0037] Optionally, determining the target parameter combination based on the candidate annotation model includes:

[0038] Input the test set into the candidate annotation model to obtain the test annotation results;

[0039] The test annotation results are compared with the standard annotation results of the test set to obtain annotation performance indicators;

[0040] If the labeled performance index meets the preset performance conditions, or the computational resources consumed by the search have exceeded the preset resource threshold, then the candidate parameter combination will be used as the target parameter combination.

[0041] Optionally, after comparing the test annotation results with the standard annotation results of the test set to obtain the annotation performance index, the method further includes:

[0042] If the annotation performance index does not meet the preset performance conditions, and the computational resources consumed in the search do not exceed the preset resource threshold, then return to the step of searching for candidate parameter combinations within the range of the values ​​using the preset hyperparameter search algorithm, until the annotation performance index meets the preset performance conditions, or the computational resources consumed in the search have exceeded the preset resource threshold.

[0043] Furthermore, to achieve the above objectives, this application also proposes a labeling device, which includes:

[0044] The model invocation module is used to invoke a preset annotation model in response to the input target to be labeled. The preset annotation model is equipped with a multi-way recall module and a preset large language model.

[0045] The first-level filtering module is used to perform multi-way recall in a preset tag library based on the target to be labeled through the multi-way recall module to obtain the candidate tag set corresponding to the target to be labeled.

[0046] The secondary filtering module is used to input the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model;

[0047] The labeling module is used to label the target to be labeled based on the target label set, so as to obtain the labeled target.

[0048] Optionally, the secondary filtering module is further configured to construct tag filtering prompts based on the target to be labeled, the candidate tag set, and the preset prompt word template, wherein the candidate tag set includes candidate tags and the tag definitions corresponding to the candidate tags; and input the tag filtering prompts into the preset large language model to obtain the target tag set output by the preset large language model.

[0049] Optionally, the primary filtering module is further configured to perform multi-way recall in a preset tag library based on the target to be labeled through the multi-way recall module to obtain multi-way recall results; and to merge the multi-way recall results to obtain a candidate tag set corresponding to the target to be labeled.

[0050] Optionally, the multi-path recall results include: keyword recall results, similarity recall results, and split recall results; the first-level screening module is further configured to extract key information from the target to be labeled through the multi-path recall module, and match the key information with a preset tag library to obtain keyword recall results; to perform vector transformation on the target to be labeled and the preset tag library to obtain a first vector and a second vector, and calculate the similarity index between the first vector and the second vector to obtain similarity recall results; to split the preset tag library into multiple sub-tag sets according to preset rules, and recall the tags corresponding to the target to be labeled in the multiple sub-tag sets based on a preset large language model to obtain split recall results.

[0051] Optionally, the primary filtering module is further configured to calculate the number of times each label appears repeatedly in the multi-path recall results; and to perform recall merging on the multi-path recall results based on the number of occurrences to obtain a candidate label set corresponding to the target to be labeled.

[0052] Optionally, the label marking device further includes:

[0053] The hyperparameter optimization module is used to analyze the key hyperparameters that affect the performance of the annotation model and define the value range of the key hyperparameters; within the value range, it searches for target parameter combinations using a preset hyperparameter search algorithm; and configures the parameters of the initial annotation model based on the target parameter combinations to obtain the preset annotation model.

[0054] In addition, to achieve the above objectives, this application also proposes a label labeling device, which includes a memory, a processor, and a label labeling program stored in the memory and executable on the processor, the label labeling program being configured to implement the label labeling method as described above.

[0055] In addition, to achieve the above objectives, this application also proposes a storage medium storing a labeling program, which, when executed by a processor, implements the labeling method as described above.

[0056] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a label labeling program, and when the label labeling program is executed by a processor, it implements the label labeling method as described above.

[0057] One or more technical solutions proposed in this application have at least the following technical effects:

[0058] This application discloses a method for responding to an input target to be labeled by calling a preset labeling model. The preset labeling model includes a multi-path recall module and a preset large language model. Based on the target to be labeled, the multi-path recall module performs multi-path recall within a preset tag library to obtain a candidate tag set corresponding to the target. The target to be labeled and the candidate tag set are then input into the preset large language model to obtain a target tag set output by the preset large language model. The target to be labeled is then labeled based on the target tag set to obtain the labeled target. Because this application first performs multi-path recall within the preset tag library through the multi-path recall module to efficiently filter a small-scale candidate set from the full tag library, and then utilizes the deep reasoning capabilities of the large language model to make accurate generative decisions on the candidate set, the efficiency and accuracy of tag labeling are improved, thereby increasing the efficiency of data processing. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating the first embodiment of the labeling method of this application;

[0062] Figure 2 This is a flowchart illustrating the second embodiment of the labeling method of this application;

[0063] Figure 3 This is a schematic diagram of a preset labeling model for one embodiment of the labeling method of this application;

[0064] Figure 4 This is a flowchart illustrating the third embodiment of the labeling method of this application;

[0065] Figure 5 This is a schematic diagram of hyperparameter optimization in one embodiment of the labeling method of this application;

[0066] Figure 6 This is a schematic diagram of the module structure of the label marking device according to an embodiment of this application;

[0067] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the labeling method in this application embodiment.

[0068] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0069] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0070] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0071] In the era of massive information, data (such as text, image, and audio data, with text data being used as an example below) is growing explosively. To effectively manage, retrieve, and analyze this data, it needs to be structured, with text annotation (or text tagging, text classification) being a core component. It aims to associate unstructured text content with predefined, semantically clear tags. For example, processing massive amounts of news and social media text by tagging each piece of information with multiple tags (such as the parties involved, event type, and scope of impact) enables precise monitoring and early warning. Multi-dimensional annotation of news, videos, and social media content (such as theme, style, and sentiment) enhances search, recommendation, and classification management capabilities.

[0072] However, existing labeling methods face the following problems: 1. The label set is huge: the number of labels in a normal scenario is no more than a few dozen, but in complex domains there are often hundreds to thousands, or even tens of thousands of labels; 2. The application bottleneck of large language models: although large language models (LLM) have shown powerful text understanding and generation capabilities, directly asking them to select labels for text from a huge label library not only exceeds the limitations of the model context window, but also faces problems such as low efficiency, high inference cost, and difficulty in stably outputting predefined label formats.

[0073] Therefore, to overcome the above-mentioned shortcomings, this application provides a solution, which includes: responding to the input target to be labeled, invoking a preset labeling model, wherein the preset labeling model is equipped with a multi-path recall module and a preset large language model; performing multi-path recall in a preset tag library based on the target to be labeled through the multi-path recall module to obtain a candidate tag set corresponding to the target to be labeled; inputting the target to be labeled and the candidate tag set into the preset large language model to obtain a target tag set output by the preset large language model; and labeling the target to be labeled based on the target tag set to obtain the labeled target. Since this application first performs multi-path recall in the preset tag library through the multi-path recall module to efficiently filter out a small-scale candidate set from the full tag library, and then uses the deep reasoning capability of the large language model to make accurate generative decisions on the candidate set, the efficiency and accuracy of tag labeling are improved, thereby improving the efficiency of data processing.

[0074] It should be noted that the executing entity in this embodiment can be a tag labeling device with data processing, network communication and program running functions, such as a computer, server, or other electronic devices that can achieve the same or similar functions. This embodiment does not limit this.

[0075] Based on this, embodiments of this application provide a labeling method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the labeling method of this application.

[0076] In the first embodiment, the labeling method includes:

[0077] Step S10: In response to the input target to be labeled, a preset labeling model is invoked, wherein the preset labeling model is equipped with a multi-path recall module and a preset large language model.

[0078] It should be understood that the target to be labeled can refer to the original information carrier that needs to be labeled. The target to be labeled can be at least one of the following: text to be labeled, image to be labeled, audio to be labeled, and other multimodal data. For ease of understanding, in this embodiment and other embodiments, the target to be labeled is illustrated using text to be labeled as an example. The preset labeling model can refer to a labeling model built based on the "divide and conquer" approach, including a multi-path recall module and a preset large language model, used to complete the fully automated labeling process from candidate tag recall to underlying tag generation. The "divide and conquer" approach means not directly throwing the complex task of "selecting the correct one from thousands of tags" to the large language model. Instead, it is broken down into two stages: Recall stage: using a variety of efficient and low-cost strategies, a small set of candidate tags that are highly relevant to the document to be labeled is quickly "recalled" from a massive tag library, quickly identifying potential targets. Generation stage: within the small candidate set, the large language model is used as a "domain expert" to make the final decision, performing detailed screening and judgment.

[0079] It is understandable that the multi-path recall module can refer to one of the core components of the pre-defined labeling model. Through various efficient retrieval strategies (such as keyword matching and semantic similarity calculation), it retrieves candidate labels related to the target to be labeled from a large-scale label library in parallel, achieving the function of "quickly identifying potential targets." The pre-defined large language model can refer to a pre-trained language model (such as the GPT series, Qwen series, etc.) with deep semantic understanding and reasoning capabilities. In this embodiment, it acts as a "domain expert," making the final decision based on candidate labels and definitions, and outputting accurate target labels.

[0080] In the specific implementation, when the target to be labeled (such as a news text) is input, the preset labeling model is automatically activated. The preset labeling model can integrate a multi-way recall module and the main components of the preset large language model, as well as auxiliary components such as prompt word generation and preliminary screening modules. The overall architecture follows a two-level architecture of "recall first and then generate".

[0081] Step S20: Based on the target to be labeled, perform multi-way recall in the preset tag library through the multi-way recall module to obtain the candidate tag set corresponding to the target to be labeled.

[0082] It should be understood that, in order to quickly identify highly relevant candidate tags from a massive tag library, significantly narrow the processing scope of subsequent LLM (e.g., reducing from 10,000 tags to 50), reduce LLM inference costs (reducing input tokens), and ensure the comprehensiveness of the candidate set through multi-strategy parallel recall (avoiding missed detections), this embodiment uses a multi-path recall module to perform multi-path recall in a preset tag library based on the target to be labeled, obtaining the candidate tag set corresponding to the target to be labeled. The preset tag library can refer to a database storing all predefined tags and their official "definitions." Tags must have clear semantics (e.g., "entity involved," "technology product release," etc.), and definitions are used to explain the connotation and boundaries of tags (e.g., "technology product release: refers to an event in which an enterprise or institution publicly launches a new technology product"). The candidate tag set can refer to potential tags highly relevant to the target to be labeled selected from a massive tag library, and its size can be much smaller than the original tag library (e.g., selecting 50 tags from 10,000 tags).

[0083] In its implementation, the multi-path recall module retrieves candidate tags in parallel from a pre-defined tag library using various methods. These recall methods can include keyword recall, semantic similarity recall, and at least one of other methods. Keyword recall refers to identifying candidate tags in the text that match a tag or its synonyms based on keyword matching (such as the BM25 algorithm). Semantic similarity recall involves converting the document to be labeled and all tags / definitions into vector representations (e.g., using models like Sentence-BERT), calculating semantic similarity, and recalling the top-K most similar tags.

[0084] Step S30: Input the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model.

[0085] Understandably, this approach aims to leverage a Large Language Model (LLM) as a "domain expert" to make the final decision within a small candidate set, performing meticulous screening and judgment. In this embodiment, the target to be labeled and the candidate label set are input into a preset Large Language Model to obtain the target label set output by the preset Large Language Model. The target label set can refer to the final label set output by the preset Large Language Model after meticulous screening of the candidate label set; it represents the label combination that best matches the target to be labeled (e.g., the target labels for a technology news article could be "technology product release," "company news," and "artificial intelligence").

[0086] In the specific implementation, a structured input instruction (Prompt) is dynamically constructed based on a preset Prompt template. This Prompt can contain the following parts: 1. Task description: Clearly informing the LLM of the task to be completed (e.g., "Please select all suitable tags for the given text from the following candidate tags"). 2. Text to be labeled: The original input text content. 3. Candidate tag list: Listing the coarsely filtered candidate tags in a formatted manner. The constructed Prompt is then input into a preset large language model (such as the GPT series, Qwen series, etc.). The LLM can comprehensively understand the entire text content and the meaning of each candidate tag, reasoning to determine which tags are appropriate, and outputting the final selected target tag set according to the specified format.

[0087] Furthermore, in order to further reduce the size of candidate labels, significantly reduce the input burden of the preset large language model, improve inference efficiency, and reduce computational costs, the preset labeling model is also equipped with a preliminary screening module. Before step S30, the model further includes: performing preliminary screening on the candidate label set through the preliminary screening module to obtain a filtered label set; correspondingly, step S30 further includes: inputting the target to be labeled and the filtered label set into the preset large language model to obtain the target label set output by the preset large language model.

[0088] It should be understood that the preliminary screening module can refer to one of the components in the pre-defined labeling model, used to further filter the candidate label set after recall merging. By setting thresholds (such as repetition count, recall score, etc.), low-relevance labels are eliminated, reducing the size of the candidate label set and easing the burden on subsequent large language model processing. The filtered label set can refer to the label set obtained after processing by the preliminary screening module. It is a smaller set after removing low-relevance labels from the candidate label set, focusing more on high-relevance labels.

[0089] In practical implementation, the core criteria are the "number of times each tag appears" (i.e., the number of times it is hit by different recall strategies, which can be regarded as "votes") or "recall score" (such as keyword recall matching score, semantic similarity score) of each tag in the candidate tag set. Thresholds are set according to scenario requirements (e.g., "retain tags with a repetition count ≥ 2" or "tags with a recall score ≥ 0.6") to ensure that obviously low-relevance tags are removed. For example, for the candidate tag set, "repetition count ≥ 2" is set as the threshold. Tags in the candidate tag set that do not meet the threshold are removed, and tags that meet the criteria are retained, forming the filtered tag set.

[0090] Furthermore, in order to transform the "relevance" of tags into quantifiable values ​​(frequency, score, confidence level), providing an objective and repeatable basis for screening and avoiding bias caused by subjective judgment, the preliminary screening module performs preliminary screening on the candidate tag set to obtain a filtered tag set. This includes: extracting quantitative indicators for each tag from the candidate tag set using the preliminary screening module, wherein the quantitative indicators include the number of times each tag is retrieved in multi-path recall, the recall score corresponding to each recall strategy, and the confidence value output by the strategy; and performing preliminary screening on the candidate tag set based on the quantitative indicators to obtain a filtered tag set.

[0091] Understandably, quantitative metrics refer to quantifiable numerical values ​​used to measure the strength of the association between a tag and the target to be labeled. These include the number of times a tag is retrieved in multi-path recall, the recall score corresponding to each recall strategy, and the confidence value output by the strategy. These are the core criteria for initial screening. The number of times a tag is retrieved refers to the number of times the same tag is hit by different recall strategies in multi-path recall results, reflecting the tag's cross-strategy acceptance. The recall score refers to the relevance score assigned to the tag by each recall strategy (such as the BM25 matching score for keyword recall and the cosine similarity score for semantic similarity recall). A higher value indicates a higher degree of association between the tag and the target to be labeled. The confidence value refers to the indicator output by each recall strategy that reflects the reliability of the tag retrieval results (such as a value between 0 and 1, where 1 indicates complete reliability), reflecting the strategy's level of trust in the tag's relevance.

[0092] In the specific implementation, based on the merged records of multi-path recall results, the number of times each tag was hit by different recall strategies is directly obtained. Recall scores can include keyword recall score, semantic similarity recall score, and split recall score. The methods for obtaining each recall score are as follows: Keyword recall score: BM25 matching score calculated for the tag by the keyword strategy. Semantic similarity recall score: Cosine similarity score calculated for the tag by the semantic strategy. Split recall score: LLM relevance score of the tag in the split strategy. The confidence value of the strategy output is obtained by evaluating the reliability of each strategy's output tag.

[0093] Step S40: Label the target to be labeled based on the target label set to obtain the labeled target.

[0094] It should be understood that the labeled target can refer to the target to be labeled that is associated with the target tag set, that is, the information carrier that has completed the structured processing (such as news text with the tag "technology product release"), which can be directly used for subsequent management, retrieval or analysis.

[0095] In practice, the target tag set is associated with the target to be labeled and stored to form a structured labeled target (such as adding a tag field to the metadata of news text), which facilitates subsequent applications such as classification retrieval and data analysis (such as quickly filtering related news through the "Technology Product Release" tag).

[0096] In this embodiment, a multi-path recall module is first used to perform multi-path recall in a preset tag library to efficiently filter out a small-scale candidate set from the full tag library. Then, the deep reasoning capability of the large language model is used to make accurate generative decisions on the candidate set, thereby improving the efficiency and accuracy of tag labeling and thus improving the efficiency of data processing.

[0097] Reference Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the labeling method of this application, based on the above. Figure 1 The first embodiment shown presents a second embodiment of the labeling method of this application.

[0098] In the second embodiment, step S20 includes:

[0099] Step S201: Based on the target to be labeled, perform multi-channel recall in the preset tag library through the multi-channel recall module to obtain multi-channel recall results.

[0100] It should be understood that, in order to integrate the results of multiple recalls, remove redundancies, strengthen highly relevant tags, and form a preliminary candidate tag set that is controllable in scale and comprehensive, in this embodiment, multiple recalls are first performed in a preset tag library based on the target to be labeled using a multi-path recall module to obtain multiple recall results. Then, the recall results of multiple paths are merged to obtain the candidate tag set corresponding to the target to be labeled. Here, the multi-path recall results can refer to the tag set obtained after each recall strategy in the multi-path recall module runs independently, that is, the tag list related to the target to be labeled output by different strategies (such as the tag set obtained by keyword recall, the tag set obtained by semantic similarity recall, etc.).

[0101] In its implementation, the multi-path recall module operates in parallel using multiple independent retrieval strategies to recall tags related to the target from a pre-defined tag library, forming multiple sets of independent recall results. These independent retrieval strategies may include at least one of keyword recall, similarity recall, split recall, and other recall methods.

[0102] Furthermore, to improve the accuracy of the multi-path recall results, the multi-path recall results include: keyword recall results, similarity recall results, and split recall results; step S201 includes: extracting key information from the target to be labeled through the multi-path recall module, and matching the key information with a preset tag library to obtain keyword recall results; performing vector transformation on the target to be labeled and the preset tag library to obtain a first vector and a second vector, and calculating the similarity index between the first vector and the second vector to obtain similarity recall results; splitting the preset tag library into multiple sub-tag sets according to preset rules, and recalling the tags corresponding to the target to be labeled in the multiple sub-tag sets based on a preset large language model to obtain split recall results.

[0103] Understandably, for the sake of efficiency and low cost, and to quickly match explicit related tags (such as "release" directly matching "technology product release"), this approach is suitable for targets with clearly defined key information. In this embodiment, key information is extracted from the target to be labeled, and this key information is matched with a preset tag library to obtain keyword recall results. The keyword recall results can refer to the set of tags recalled from the preset tag library through a keyword matching strategy, generated based on the matching degree between the key information in the target and the tags (or their synonyms). Key information can refer to core words, phrases, or concepts (such as events, subjects, attributes, etc.) extracted from the target to be labeled, and is the basis for keyword matching (such as "release" in technology news, "quantum computing chip," and "artificial intelligence training").

[0104] In practice, core words or phrases (such as "release", "quantum computing chip", and "artificial intelligence training") are extracted from the target to be labeled using word segmentation and keyword extraction tools (such as TF-IDF and TextRank). The extracted key information is compared with the tag names and synonyms in a pre-set tag library, and algorithms such as BM25 are used to calculate the matching score (the higher the score, the higher the matching degree). The top-K tags (such as the top-20) are selected according to the matching score to form the keyword recall results.

[0105] It should be understood that, in order to capture implicit semantic associations (such as "quantum computing chip" and "chip technology," which, although not having completely identical keywords, are semantically related), and to overcome the limitations of keyword recall, this embodiment performs vector transformation on the target to be labeled and a preset tag library to obtain a first vector and a second vector, and calculates the similarity index between the first vector and the second vector to obtain a similarity recall result. The similarity recall result can refer to the set of tags recalled from the preset tag library through a semantic similarity calculation strategy, generated based on the semantic vector similarity between the target to be labeled and the tags (or their definitions). Vector transformation can refer to the process of converting unstructured text (target to be labeled, tags / definitions) into numerical vectors, achieved through a pre-trained model (such as Sentence-BERT), so that the semantic information of the text can be quantified through vector operations. The first vector can refer to the semantic vector obtained after vector transformation of the target to be labeled, used to represent its core semantics. The second vector can refer to the semantic vector obtained after vector transformation of tags (and / or their definitions) in the preset tag library, used to represent the semantic connotation of the tags. Similarity metrics can refer to quantitative indicators (such as cosine similarity) used to measure the degree of semantic association between a first vector and a second vector. The value range can be [-1, 1], and the closer the value is to 1, the more similar the semantics are.

[0106] In the specific implementation, a pre-trained semantic model (such as Sentence-BERT) is used to transform the target to be labeled into a first vector, and the definition of each tag in the pre-set tag library is transformed into a second vector (the tag definition can more accurately represent the tag semantics). The similarity between the first vector and each second vector is calculated using the cosine similarity formula (e.g., the similarity between the text vector and the definition vector of "quantum computing" is 0.85). The top-M tags (e.g., Top-20) are selected according to the similarity index to form the similarity recall results.

[0107] It should be understood that, in order to address the efficiency bottleneck of direct retrieval of ultra-large-scale tag libraries (such as 100,000 tags), this embodiment reduces the LLM processing scope by narrowing the sub-tag sets, lowering the context window pressure, and improving recall comprehensiveness (avoiding LLM omissions due to excessive tags). In this embodiment, a preset tag library is divided into multiple sub-tag sets according to preset rules, and tags corresponding to the target to be labeled are recalled in each sub-tag set based on a preset large language model, resulting in a split recall result. The split recall result refers to the set of tags recalled in each sub-tag set after the preset tag library is divided into sub-tag sets according to preset rules, suitable for efficient retrieval of ultra-large-scale tag libraries. Preset rules refer to the basis for splitting the preset tag library (such as by the tag's domain, level, application scenario, etc.), ensuring that the size of the sub-tag sets is controllable and semantically focused (e.g., splitting "technical field" into sub-tag sets such as "quantum computing" and "artificial intelligence"). A sub-tag set refers to a small-scale tag set obtained after splitting the preset tag library according to preset rules, with each sub-tag set containing tags of a certain category or dimension (e.g., "subset of quantum computing-related tags").

[0108] In the specific implementation, the pre-defined tag library is split into sub-tag sets according to preset rules (such as "technical field" or "event type"). For example, by "technical field," it is split into sub-tag sets such as "quantum computing," "artificial intelligence," and "chip technology," with each sub-tag set containing 10-50 tags. For each sub-tag set, a prompt word containing the target to be labeled and the sub-tag set (including tags and definitions) is constructed. This prompt word is input into a pre-defined large language model (such as GPT), which is required to recall relevant tags from the sub-tag set (e.g., in the "quantum computing" sub-tag set, LLM recalls the "quantum computing" tag based on the text "quantum computing chip"). The recalled tags from each sub-tag set are collected to form the split recall results.

[0109] Step S202: Merge the multi-path recall results to obtain the candidate tag set corresponding to the target to be labeled.

[0110] It is understandable that recall merging can refer to the process of integrating multiple recall results, including merging tags from different strategies and counting the number of times tags appear repeatedly, in order to form a more comprehensive preliminary set of candidate tags.

[0111] In practice, multiple recall results are integrated, redundancy is removed, and highly relevant labels are strengthened to form a preliminary candidate label set that is controllable in scale and comprehensive.

[0112] Furthermore, in order to retain highly relevant labels and control the size of the candidate set, laying the foundation for accurate screening by the large language model in the future, step S202 includes: calculating the number of times each label appears repeatedly in the multi-way recall results; and performing recall merging on the multi-way recall results according to the number of times to obtain the candidate label set corresponding to the target to be labeled.

[0113] It should be understood that the number of recurrences can refer to the number of times the same tag is hit by different recall strategies in multiple recall results (e.g., if the tag "technology product release" is hit by both keyword recall and similarity recall, the number of times is 2), used to measure the reliability of the association between the tag and the target to be labeled. Recall merging can refer to the process of integrating multiple recall results, including counting the number of times tags are repeated, deduplication, and filtering by frequency, ultimately forming a controllable and highly relevant set of candidate tags.

[0114] In the specific implementation, all tags from the keyword recall results, similarity recall results, and split recall results are integrated into a single overall list (including duplicate tags). For example, the three recall results are: Keyword Recall: ["Technology Product Release", "Quantum Computing", "Chip Technology", "Artificial Intelligence Applications"], Similarity Recall: ["Technology Product Release", "Quantum Computing", "Artificial Intelligence Applications", "Technology News"], and Split Recall: ["Technology Product Release", "Quantum Computing", "Artificial Intelligence Applications"]. The resulting overall list is: ["Technology Product Release", "Quantum Computing", "Chip Technology", "Artificial Intelligence Applications", "Technology Product Release", "Quantum Computing", "Artificial Intelligence Applications", "Technology News", "Technology Product Release", "Quantum Computing", "Artificial Intelligence Applications"]. The system iterates through and counts the occurrences of each tag in the overall list, recording the frequency of each unique tag. For example: "Technology Product Release" appears 3 times (hit by keyword, similarity, and splitting recall); "Quantum Computing" appears 3 times (same as above); "Artificial Intelligence Applications" appears 3 times (same as above); "Chip Technology" appears 1 time (hit only by keyword recall); "Technology News" appears 1 time (hit only by similarity recall). Deduplication: Remove duplicate tags from the overall list, retaining only a unique instance of each tag (e.g., after deduplicating the above overall list, we get: ["Technology Product Release", "Quantum Computing", "Chip Technology", "Artificial Intelligence Applications", "Technology News"]). Sort by frequency: Sort the deduplicated tags by the frequency of their appearance from highest to lowest (higher frequency, higher priority). For example: 3 times: "Technology Product Release", "Quantum Computing", "Artificial Intelligence Applications"; 1 time: "Chip Technology", "Technology News". Screening size control: Screen tags according to preset rules (e.g., retain tags with ≥1 frequency, or select the Top-N tags by frequency to ensure the candidate set size is suitable for LLM processing, which can be 20-50 tags). For example, taking the Top-5 tags, the final candidate tag set is: ["Technology Product Release", "Quantum Computing", "Artificial Intelligence Application", "Chip Technology", "Technology News"].

[0115] In the second embodiment, step S30 includes:

[0116] Step S301: Construct tag filtering prompts based on the target to be labeled, the candidate tag set, and the preset prompt word template, wherein the candidate tag set includes candidate tags and the tag definitions corresponding to the candidate tags.

[0117] It should be understood that, in order for the pre-defined large language model to accurately distinguish semantically similar tags, in this embodiment, when constructing tag filtering prompts, the candidate tag set includes not only candidate tags but also the tag definitions corresponding to the candidate tags. The tag filtering prompts can refer to structured instruction text generated by integrating the target to be labeled, candidate tags, and corresponding tag definitions according to a pre-defined prompt template. Its core function is to guide the pre-defined large language model (LLM) to clarify the task objective, understand the content to be labeled, and the accurate meaning of the candidate tags, thereby selecting the most matching tags from the candidate tag set. The pre-defined prompt template can refer to a pre-set fixed text framework that includes a task description, input information placeholders (such as the location of the target to be labeled, the location of candidate tags, and the definition list), and output format requirements. This framework is used to standardize the structure of the tag filtering prompts, ensuring that the LLM can clearly understand the task boundaries and output standards. The label definition corresponding to the candidate label can be the official detailed explanation of each candidate label, which clarifies the connotation, applicable scenarios and boundaries of the label (such as "technology product launch: refers to the event of a company or institution publicly launching a new technology product"), to help LLM accurately understand the specific meaning of the label, especially labels that are easily confused or rare in the professional field.

[0118] It is understood that in this embodiment, the preset annotation model can refer to an annotation model built based on the "divide and conquer" and "expert decision-making" approaches. It includes a multi-path recall module and a preset large language model, used to complete the entire automated annotation process from candidate tag recall to underlying tag generation. The "expert decision-making" approach aims to enable the LLM (Label Management Model) to make more accurate judgments, especially in the tagging system of specialized fields. Tags may have specific meanings or be rarely seen in general open domains, which may cause difficulties for the LLM in understanding. This embodiment proposes not only providing the candidate tag names, but more importantly, providing the official "definition" of each candidate tag. This greatly enriches the LLM's decision-making basis, enabling it to understand the precise connotations and boundaries behind the tags, thereby correctly distinguishing between semantically similar tags.

[0119] In the specific implementation, a structured input instruction (Prompt) is dynamically constructed based on a preset Prompt template. This Prompt can contain the following parts: 1. Task Description: Clearly tells the LLM the task to be completed (e.g., "Please select all appropriate tags for the given text from the following candidate tags"). 2. Text to be Annotated: The original input text content. 3. List of Candidate Tags with Attached Definitions: Lists the coarsely filtered candidate tags and their corresponding official definitions in a formatted manner. For example: "[Tag1: {Definition of Tag1}], [Tag2: {Definition of Tag2}], ...

[0120] Step S302: Input the tag filtering prompts into the preset large language model to obtain the target tag set output by the preset large language model.

[0121] In its implementation, the constructed prompt is input into a pre-defined large language model (such as the GPT series, Qwen series, etc.). The LLM can comprehensively understand the entire text content and the precise definition of each candidate tag, reasoning to determine which tags are appropriate, and outputting the final selected target tag set according to a specified format. The LLM processing logic is as follows: 1. Parse the task description: Clearly define the tags that need to be selected from the candidate tags to match the target to be labeled, and make judgments based on the tag definitions; 2. Understand the target to be labeled: Deeply analyze the text content and extract core information (such as events, subjects, attributes, etc.); 3. Match tag definitions: Compare the core information of the target to be labeled with the definition of each candidate tag to determine whether the tag conforms to the text's connotation; 4. Output the results: Output the filtered target tag set (excluding unmatched tags) according to the format required by the prompt word template.

[0122] For ease of understanding, please refer to Figure 3 This explanation is provided, but does not limit the scope of this application. Figure 3 This is a schematic diagram of a preset annotation model according to an embodiment of the label annotation method of this application. Figure 3 As an example, assuming the target to be labeled is text, the labeling method includes the following steps:

[0123] 1. Input reception: Receive the text to be annotated.

[0124] 2. Multi-channel Recall Module: The tag library stores all available tags and their detailed text definitions. This module retrieves candidate tags in parallel from a pre-built "large-scale tag and definition library" using multiple methods. Recall methods may include:

[0125] Keyword Recall: Based on keyword matching (such as the BM25 algorithm), find candidate tags in the text that match the tag or its synonyms.

[0126] Semantic similarity recall: Convert the document to be labeled and all tags / definitions into vector representations (e.g., using models such as Sentence-BERT), calculate semantic similarity, and recall the top-K most similar tags.

[0127] Other recall strategies (scalable): More recall strategies can be integrated as needed, such as splitting the tag system into smaller batches and then recalling based on LLM semantic understanding.

[0128] Recall merging: The results of each recall are merged and the number of repeated occurrences is calculated to form a preliminary and broader set of candidate tags.

[0129] 3. Preliminary screening module: Performs preliminary screening on the merged candidate set. For example, a filtering threshold can be set based on voting count, recall score, confidence level, etc., to remove labels with low relevance in order to control the size of the candidate set entering the next stage.

[0130] 4. Dynamic Prompt Generation: This is a crucial part connecting recall and LLM generation. It dynamically constructs a structured input prompt based on a preset prompt template. This prompt can include the following parts:

[0131] Task Description: Clearly inform the LLM of the task to be completed (e.g., “Please select all appropriate labels for the given text from the following candidate labels”).

[0132] Text to be annotated: The original text content entered.

[0133] The accompanying list of candidate tags includes their official definitions. This formatted list displays the shortlisted candidate tags and their corresponding official definitions. For example: "[tag1: {definition of tag1}], [tag2: {definition of tag2}], ...".

[0134] 5. Preset Large Language Model: Input the constructed Prompt into the preset large language model (such as GPT series, Qwen series, etc.). The LLM will comprehensively understand the entire text content and the accurate definition of each candidate tag, reason to determine which tags are appropriate, and output the final list of selected tags in the specified format.

[0135] 6. Output Results: Parse the LLM output, associate the finalized one or more tags with the original text, and complete the annotation process.

[0136] In this embodiment, when constructing tag filtering prompts, the candidate tag set includes not only candidate tags but also the tag definitions corresponding to the candidate tags. This allows the definitions of candidate tags to be incorporated into the prompts, providing a basis for judgment for the preset large language model, enabling the preset large language model to accurately distinguish tags with similar semantics.

[0137] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the labeling method of this application. Based on the above embodiments, a third embodiment of the labeling method of this application is proposed.

[0138] In the third embodiment, before step S10, the method further includes:

[0139] Step S01: Analyze the key hyperparameters that affect the performance of the labeled model and define the value range of the key hyperparameters.

[0140] It should be understood that the entire annotation process involves multiple modules and parameters (such as how many candidate labels to recall, what the filtering threshold is set to, and even the model retrieval and the selection of the label generation model), and the combination of these parameters directly affects the final result. To avoid tedious and inefficient manual parameter tuning, this embodiment introduces an automated hyperparameter optimization mechanism, which finds the optimal parameter configuration for the entire annotation model through systematic searching and evaluation.

[0141] Understandably, labeling model performance refers to the effectiveness and efficiency of the labeling model in completing multi-label text classification tasks, including accuracy (the correctness of labeled tags), recall (the ability to cover all relevant tags), inference speed (labeling time), and cost (computational resource consumption), such as whether the model misses key tags or mislabels irrelevant tags when labeling science and technology news. Key hyperparameters refer to adjustable parameters that significantly affect the performance of the labeling model and need to be optimized to determine their optimal values. These include parameters of the multi-path recall module (such as the number of recalls), parameters of the preliminary screening module (such as the filtering threshold), and parameters of the large language model (such as the generation temperature), etc. (e.g., the "Top-K value of keyword recall" directly affects the coverage of candidate tags). The value range refers to the possible numerical range of key hyperparameters, which are pre-set according to the business scenario and model characteristics (e.g., the range of the "Top-K value" can be set to 10-50 to ensure that the number of recalls is neither too low, leading to missed detections, nor too high, increasing the computational burden).

[0142] In the specific implementation, parameters that significantly impact performance are identified throughout the entire annotation model process. Their reasonable value ranges are determined by combining business requirements and experience, providing clear boundaries for subsequent searches. The specific steps are as follows:

[0143] 1. Identify key hyperparameters:

[0144] Multi-path recall module: The "Top-K value" of each recall strategy (e.g., keyword recall returns the top 20 tags, semantic similarity recall returns the top 20 tags) affects the coverage and scale of candidate tags;

[0145] The initial screening module includes "frequency threshold" (e.g., retaining tags that appear ≥3 times) and "score threshold" (e.g., average recall score ≥0.8), which affect the relevance and size of the tag set after screening.

[0146] The LLM generation module includes "temperature parameters" (which control the randomness of the output, such as 0.1-0.5) and "maximum number of generated tags" (such as returning a maximum of 5 tags), which affect the accuracy and stability of the target tag set.

[0147] 2. Define the range of values:

[0148] Keyword recall Top-K: 10-50 (step size 10), semantic recall Top-K: 10-50 (step size 10), to ensure sufficient tag coverage and controllable scale;

[0149] Frequency threshold: 2-4 (step 1), score threshold: 0.7-0.9 (step 0.1), to balance relevance and recall;

[0150] LLM temperature: 0.1-0.5 (step size 0.1) to avoid incorrect labeling caused by excessive randomness.

[0151] Step S02: Within the range of values, search for the target parameter combination using a preset hyperparameter search algorithm.

[0152] Understandably, the preset hyperparameter search algorithm can refer to an algorithm used to automatically find the optimal combination of parameters within a range of values, including grid search, random search, Bayesian optimization, etc. (For example, Bayesian optimization efficiently locates the optimal combination by learning the relationship between parameters and performance). The target parameter combination can refer to the key hyperparameter combination that optimizes the performance of the labeled model, found by the hyperparameter search algorithm (e.g., "Keyword recall Top-K=20, screening threshold=3 times, LLM temperature=0.3").

[0153] In practical implementation, if there are few parameters (e.g., 2-3), a grid search is used: traverse all parameter combinations (e.g., 6 combinations of Top-K=10, 20, 30 and threshold=2, 3), evaluate the performance of each combination on the test set (e.g., F1 score), and select the combination with the highest score. If there are many parameters, Bayesian optimization is used: build a probabilistic model based on historical evaluation results, predict the next most likely optimal parameter combination (e.g., test 3 groups first, and focus on the high-performing region based on the results), reduce the number of evaluations, and efficiently find the optimal solution.

[0154] Furthermore, in order to transform the abstract parameter combination into a runnable model instance, provide entity objects for performance evaluation, and make the parameter effect directly verifiable, step S02 includes: searching for candidate parameter combinations within the value range using a preset hyperparameter search algorithm; configuring the parameters of the initial annotation model based on the candidate parameter combinations to obtain candidate annotation models; and determining the target parameter combination based on the candidate annotation models.

[0155] It should be understood that candidate parameter combinations can refer to parameter combinations to be verified generated within a range of values ​​through a hyperparameter search algorithm (e.g., "keyword Top-K=20, screening threshold=3 times, LLM temperature=0.3"), which are intermediate results in the search process. Candidate annotation models can refer to specific model instances obtained by configuring candidate parameter combinations into the initial annotation model. Each candidate parameter combination corresponds to a candidate annotation model, used to verify performance on the test set.

[0156] Furthermore, in order to form a closed loop of "generation-testing-feedback" and gradually approach the optimal solution through iterative search to improve the efficiency and reliability of hyperparameter optimization, the step of determining the target parameter combination based on the candidate annotation model includes: inputting the test set into the candidate annotation model to obtain test annotation results; comparing the test annotation results with the standard annotation results of the test set to obtain annotation performance indicators; if the annotation performance indicators meet preset performance conditions, or the computational resources consumed by the search have exceeded a preset resource threshold, then the candidate parameter combination is taken as the target parameter combination.

[0157] After comparing the test annotation results with the standard annotation results of the test set to obtain the annotation performance index, the method further includes: if the annotation performance index does not meet the preset performance conditions and the computational resources consumed in the search do not exceed the preset resource threshold, then return to the step of searching for candidate parameter combinations within the range of the values ​​using a preset hyperparameter search algorithm, until the annotation performance index meets the preset performance conditions or the computational resources consumed in the search exceed the preset resource threshold.

[0158] Understandably, the test set can refer to a collection of labeled samples containing known correct labels, used to verify the performance of the candidate labeling model. Each sample corresponds to a unique "standard labeling result" (e.g., 100 science and technology news articles and their manually labeled correct labels, including "science and technology product release" and "quantum computing"). The candidate labeling model can refer to a specific model instance obtained after configuring the initial labeling model through combinations of candidate parameters (e.g., the model corresponding to "keyword Top-K=20, screening threshold=3 times"), used to verify performance on the test set. The test labeling result can refer to the set of labels output by the candidate labeling model after labeling the test set samples (e.g., the labels of the science and technology news articles labeled by the candidate model), directly reflecting the model's performance. The standard labeling result can refer to the set of correctly labeled labels pre-labeled manually in the test set (e.g., the standard labels for science and technology news articles are "science and technology product release," "quantum computing," and "artificial intelligence applications"), serving as a benchmark for evaluating the accuracy of the test labeling result. Labeling performance metrics can refer to indicators used to quantify the degree of matching between test labeling results and standard labeling results, including precision (the percentage of correctly labeled labels), recall (the proportion of standard labels covered), and F1 score (the harmonic mean of precision and recall) (mentioned in the document as "recall and label accuracy evaluation"). Preset performance conditions can refer to pre-set thresholds for measuring labeling performance (e.g., "F1 score ≥ 0.9"), which are the basis for judging whether candidate parameter combinations are optimal. Search computational resources can refer to the computational resources consumed during the hyperparameter search process (e.g., computing power, time, memory). Preset resource thresholds can refer to the pre-set maximum computational resources that the search process can consume (e.g., "maximum search time 24 hours," "maximum number of iterations 50"), used to prevent excessive resource consumption. Candidate parameter combinations can refer to the parameter combinations to be verified generated by the hyperparameter search algorithm (e.g., "keyword Top-K = 20, screening threshold = 3 times"), corresponding to candidate labeling models. Target parameter combinations can refer to the candidate parameter combinations that meet the preset performance conditions or have the best performance within the resource thresholds, representing the final result of hyperparameter optimization.

[0159] In the specific implementation, the test set samples are labeled using a candidate labeling model to generate a set of labels predicted by the model (test labeling results), which serve as input for performance evaluation. The specific steps are as follows: 1. Input test set samples: Input each sample in the test set (such as science and technology news text) into the candidate labeling model one by one; 2. Model labeling process: The candidate labeling model performs multi-way recall, preliminary screening, LLM generation, and other processes according to its parameter configuration (such as "keyword Top-K=20, screening threshold=3 times"), and outputs a set of labels for each sample; 3. Generate test labeling results: Collect the labeling results of all samples to form test labeling results (such as the model's labeling results for science and technology news being "science and technology product release, quantum computing, artificial intelligence application").

[0160] By quantitatively comparing the differences between test annotation results and standard annotation results, performance metrics (such as the F1 score) are calculated to objectively evaluate the merits of candidate parameter combinations. The specific steps are: 1. Sample-by-sample comparison: For each sample in the test set, its test annotation result is compared with the standard annotation result (e.g., if the test annotation result in the example is "technology product release, quantum computing, artificial intelligence application," and the standard result is the same, then the sample is correctly labeled); 2. Calculation of performance metrics: Based on the overall comparison results, the accuracy (the percentage of correctly labeled tags among all labeled tags), recall (the percentage of correctly labeled tags among all standard tags), and F1 score (the harmonic mean of accuracy and recall) are calculated. For example, in a test set of 100 samples, the model's F1 score is 0.92.

[0161] Based on the labeled performance metrics and resource consumption, a decision is made on whether to terminate the search and determine the target parameter combination, or to continue generating new candidate parameter combinations, ensuring a balance between efficiency and performance. The specific steps are as follows:

[0162] If the marked performance index (e.g., F1 value = 0.92) is greater than or equal to the preset performance condition (e.g., F1 ≥ 0.9), then the current candidate parameter combination meets the performance standard and can be used as the target parameter combination.

[0163] If the performance indicators are not met (e.g., F1=0.85<0.9), but the consumed computing resources (e.g., search time of 25 hours) exceed the preset resource threshold (24 hours), then the search is terminated, and the current optimal candidate parameter combination is selected as the target parameter combination.

[0164] If neither of the above two conditions is met (e.g., F1=0.85<0.9, and search time 10 hours<24 hours), then return to the "Search candidate parameter combinations" step, generate new candidate parameter combinations, and repeat the above process.

[0165] Step S03: Configure the parameters of the initial annotation model based on the target parameter combination to obtain the preset annotation model.

[0166] It should be understood that the initial annotation model can refer to a basic model framework that has not undergone hyperparameter optimization, including modules such as multi-path recall, preliminary screening, and LLM generation, but with default parameters (e.g., default Top-K=10, screening threshold=2 times), and its performance is not optimal. Parameter configuration can refer to the process of assigning target parameter combinations to the initial annotation model, that is, adjusting the parameters of each module of the model to optimal values ​​so that the model runs according to the optimal strategy. The preset annotation model can refer to an optimized model obtained after configuring the target parameter combinations, which has better accuracy, efficiency, or cost control capabilities and can be directly used for actual annotation tasks (such as a model that can accurately annotate science and technology news tags).

[0167] In the specific implementation, the target parameter combination is assigned to the corresponding module of the initial annotation model, so that the model runs with optimal parameters, thereby maximizing performance (such as improving annotation accuracy and reducing inference cost). The specific steps are as follows: 1. Initial model parameter reset: The parameters of the initial annotation model are default values ​​(such as keyword Top-K=10, threshold=2), and the performance is not optimized (such as F1 score=0.8); 2. Apply target parameter combination: The target parameters obtained from the search (such as Top-K=20, threshold=3) are configured to the multi-path recall module (controlling the number of recalls), the preliminary screening module (controlling the screening strictness), and the LLM generation module (controlling the output stability); 3. Generate preset annotation model: After configuration, the model's performance on the test set is significantly improved (such as F1 score=0.92), and it can be directly used for actual annotation tasks (such as automated tag annotation of science and technology news).

[0168] For ease of understanding, please refer to Figure 5 This explanation is provided, but does not limit the scope of this application. Figure 5 This is a schematic diagram illustrating hyperparameter optimization in one embodiment of the labeling method of this application. Figure 5 In this process, the hyperparameter optimization steps include:

[0169] 1. Define the search space: Identify the key hyperparameters affecting performance in the labeling model and define their value ranges. For example: Top-K values ​​of each recall module; filtering thresholds of the preliminary screening module; confidence requirements for LLM-generated labels; and optional recall models or label generation LLMs.

[0170] 2. Select a search strategy: Employ an efficient hyperparameter search algorithm to explore the search space. Examples of possible strategies are as follows:

[0171] Grid search: Iterates through all parameter combinations, suitable for situations with a small number of parameters.

[0172] Random search: Randomly sampling and combining parameters in the parameter space is more efficient.

[0173] Bayesian optimization algorithm: A smarter search method that predicts the next most promising parameter point based on the performance of the evaluated parameter points, thus finding the optimal solution with fewer evaluations.

[0174] 3. Iterative evaluation and optimization loop: Enter an automated loop.

[0175] (1) Parameter selection: The search strategy module proposes a new set of hyperparameter combinations.

[0176] (2) Evaluation of the annotation model: Use this set of hyperparameters to configure the complete annotation model and run it on an independent test set to annotate a batch of sample texts.

[0177] (3) Performance evaluation: Compare the annotation results of the annotation model with the human standard answers of the test set, and calculate key performance indicators (such as accuracy, recall, F1 score, etc.).

[0178] (4) Feedback and Decision: Determine whether the current performance "meets the preset threshold" or "the number of evaluations has reached the limit". If yes, the optimization process ends, and the best parameter combination found is the final configuration. If no, the search strategy continues to propose the next set of parameter combinations based on the new feedback information, and the process is repeated.

[0179] This embodiment introduces an automated hyperparameter optimization mechanism. Through systematic search and evaluation, it finds the optimal parameter configuration for the entire annotation model, thereby avoiding time-consuming and inaccurate manual trial and error, and thus ensuring the robustness and optimal performance of the system under different datasets and label systems.

[0180] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the labeling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0181] This application also provides a label marking device, please refer to... Figure 6 The labeling device includes:

[0182] The model invocation module 10 is used to invoke a preset annotation model in response to the input target to be labeled, wherein the preset annotation model is provided with a multi-way recall module and a preset large language model;

[0183] The first-level filtering module 20 is used to perform multi-way recall in a preset tag library based on the target to be labeled through the multi-way recall module to obtain the candidate tag set corresponding to the target to be labeled.

[0184] The secondary filtering module 30 is used to input the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model;

[0185] The label annotation module 40 is used to annotate the target to be annotated based on the target label set to obtain the annotated target.

[0186] The label labeling device provided in this application, employing the label labeling method described in the above embodiments, addresses the technical problems of low labeling efficiency and inaccurate labeling in existing label labeling methods when dealing with large-scale labels. Compared with the prior art, the beneficial effects of the label labeling device provided in this application are the same as those of the label labeling method provided in the above embodiments, and other technical features in the label labeling device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0187] This application provides a label labeling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the label labeling method in Embodiment 1 above.

[0188] The following is for reference. Figure 7 The diagram illustrates a structural schematic of a labeling device suitable for implementing embodiments of this application. The labeling device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The labeling device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0189] like Figure 7As shown, the tag labeling device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the tag labeling device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the tag labeling device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show tag labeling devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0190] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0191] The label labeling device provided in this application, employing the label labeling method described in the above embodiments, addresses the technical problems of low labeling efficiency and inaccurate labeling in existing label labeling methods when dealing with large-scale labels. Compared with the prior art, the beneficial effects of the label labeling device provided in this application are the same as those of the label labeling method provided in the above embodiments, and other technical features of this label labeling device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0192] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0193] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0194] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the labeling method in the above embodiments.

[0195] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), or flash memory, optical fiber, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0196] The aforementioned computer-readable storage medium may be included in the label labeling device; or it may exist independently and not be assembled into the label labeling device.

[0197] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the label labeling device, cause the label labeling device to perform the aforementioned label labeling method.

[0198] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0200] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0201] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described labeling method. This addresses the technical problems of low labeling efficiency and inaccurate labeling in existing labeling methods when dealing with large-scale labels. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the labeling method provided in the above embodiments, and will not be repeated here.

[0202] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the labeling method described above.

[0203] The computer program product provided in this application addresses the technical problems of low labeling efficiency and inaccurate labeling in existing labeling methods when dealing with large-scale labels. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the labeling method provided in the above embodiments, and will not be repeated here.

[0204] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A labeling method, characterized in that, The labeling method includes: In response to the input target to be labeled, a preset labeling model is invoked, wherein the preset labeling model is equipped with a multi-path recall module and a preset large language model; Based on the target to be labeled, the multi-way recall module performs multi-way recall in the preset tag library to obtain the candidate tag set corresponding to the target to be labeled; Input the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model; The target to be labeled is labeled based on the target label set to obtain the labeled target; The step of performing multi-channel recall in a preset tag library based on the target to be labeled through the multi-channel recall module includes: The preset tag library is split into multiple sub-tag sets according to preset rules, and the tags corresponding to the target to be labeled are recalled in the multiple sub-tag sets based on a preset large language model. The preset rules are to split the tags according to their domain, level, and application scenario. After splitting, each sub-tag set contains tags of a single category or dimension. For each sub-tag set, a prompt word containing the target to be labeled, the tags in the sub-tag set, and their corresponding definitions is constructed. The prompt word is input into the preset large language model and the relevant tags are recalled. The recalled tags of all sub-tag sets are summarized to form the split recall result. The preset annotation model also includes a preliminary screening module, which is used to perform preliminary screening on the candidate label set. Before calling the preset annotation model in response to the input target to be labeled, the model further includes: Analyze the key hyperparameters affecting the performance of the annotation model and define the value range of the key hyperparameters. The key hyperparameters include the recall number of the multi-path recall module, the filtering threshold of the preliminary screening module, and the generation temperature of the preset large language model. Within the value range, search for target parameter combinations using a preset hyperparameter search algorithm. Based on the target parameter combinations, configure the parameters of the initial annotation model to obtain the preset annotation model.

2. The labeling method as described in claim 1, characterized in that, The step of inputting the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model includes: Based on the target to be labeled, the candidate tag set, and the preset prompt word template, a tag filtering prompt word is constructed, wherein the candidate tag set includes candidate tags and the tag definitions corresponding to the candidate tags; Input the tag filtering prompts into the preset large language model to obtain the target tag set output by the preset large language model.

3. The labeling method as described in claim 1, characterized in that, The step of obtaining a candidate tag set corresponding to the target to be labeled by performing multi-way recall in a preset tag library based on the target to be labeled through the multi-way recall module includes: Based on the target to be labeled, the multi-channel recall module performs multi-channel recall in the preset tag library to obtain multi-channel recall results; The multi-path recall results are merged to obtain the candidate tag set corresponding to the target to be labeled.

4. The labeling method as described in claim 3, characterized in that, The multi-path recall results include: keyword recall results, similarity recall results, and split recall results; the process of performing multi-path recall in a preset tag library based on the target to be labeled, and obtaining multi-path recall results, includes: The multi-path recall module extracts key information from the target to be labeled and matches the key information with a preset tag library to obtain keyword recall results. The target to be labeled and the preset tag library are transformed into vectors to obtain a first vector and a second vector, and the similarity index between the first vector and the second vector is calculated to obtain the similarity recall result. The preset tag library is split into multiple sub-tag sets according to preset rules, and the tags corresponding to the target to be labeled are recalled in the multiple sub-tag sets based on the preset large language model to obtain the splitting and recall results.

5. The labeling method as described in claim 3, characterized in that, The step of merging the multi-path recall results to obtain the candidate tag set corresponding to the target to be labeled includes: Calculate the number of times each tag appears repeatedly in the multi-path recall results; Based on the number of recalls, the multi-path recall results are merged to obtain the candidate tag set corresponding to the target to be labeled.

6. The labeling method as described in claim 1, characterized in that, The preset annotation model also includes a preliminary screening module. Before inputting the target to be annotated and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model, the method further includes: The candidate tag set is initially filtered through the preliminary filtering module to obtain the filtered tag set; Accordingly, the step of inputting the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model includes: The target to be labeled and the filtered label set are input into the preset large language model to obtain the target label set output by the preset large language model.

7. A label marking device, characterized in that, The labeling device includes: The model invocation module is used to invoke a preset annotation model in response to the input target to be labeled. The preset annotation model is equipped with a multi-way recall module and a preset large language model. The first-level filtering module is used to perform multi-way recall in a preset tag library based on the target to be labeled through the multi-way recall module to obtain the candidate tag set corresponding to the target to be labeled. The secondary filtering module is used to input the target to be labeled and the candidate label set into the preset large language model to obtain the target label set output by the preset large language model; The label annotation module is used to annotate the target to be annotated based on the target label set, and obtain the annotated target; The first-level filtering module is further configured to split the preset tag library into multiple sub-tag sets according to preset rules, and recall the tags corresponding to the target to be labeled in the multiple sub-tag sets based on a preset large language model; the preset rules are to split according to the domain, level, and application scenario of the tags, and each sub-tag set contains tags of a single category or dimension after splitting; for each sub-tag set, a prompt word containing the target to be labeled and the tags in the sub-tag set and their corresponding definitions is constructed, input into the preset large language model and recall the relevant tags, and the recalled tags of all sub-tag sets are summarized to form the splitting and recall results; The preset annotation model also includes a preliminary screening module, which is used to perform preliminary screening on the candidate label set. Before calling the preset annotation model in response to the input target to be labeled, the following steps are also included: Analyze the key hyperparameters affecting the performance of the annotation model and define the value range of the key hyperparameters. The key hyperparameters include the recall number of the multi-path recall module, the filtering threshold of the preliminary screening module, and the generation temperature of the preset large language model. Within the value range, search for target parameter combinations using a preset hyperparameter search algorithm. Based on the target parameter combinations, configure the parameters of the initial annotation model to obtain the preset annotation model.

8. A label marking device, characterized in that, The labeling device includes: a memory, a processor, and a labeling program stored in the memory and executable on the processor, wherein the labeling program, when executed by the processor, implements the labeling method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a labeling program, which, when executed by a processor, implements the labeling method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a labeling program, which, when executed by a processor, implements the labeling method as described in any one of claims 1 to 6.

Images