A large language model-based open domain label system construction method and device

Abstract

The application provides a large language model-based open domain label system construction method and device, and relates to the technical field of data processing. The method constructs a label association pool according to different to-be-labeled contents in a first time period and initial labels output by labeling each to-be-labeled content using a first large language model; determines an initial label system and a label clustering cluster of any initial label based on a label occurrence frequency of each initial label; determines a normalized label of each initial label using a zero-shot capability of a second large language model based on a preset semantic understanding task description; constructs a label directed graph set; and constructs a target label system if it is detected that a first label in a first label directed graph is a normalized label of a second label in the directed graph, and the normalized label of the first label in a second label directed graph is a third label. The method shortens the response time from the emergence of a new topic to the label system, and improves the semantic accuracy of the label system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a large language model-based open domain label system construction method and device. BACKGROUND

[0002] Text classification, as a core basic task of natural language processing, is essentially to realize accurate mapping from high-dimensional semantic space to discrete label space. Under the background of the increasing complexity of Internet content ecology, traditional text classification has evolved from simple category attribution judgment to multi-dimensional and fine-grained content semantic understanding task. Modern content platforms need to build a precise label system to realize the deep semantic representation of content. This system not only carries the basic function of content understanding, but also becomes a key technology base for driving personalized recommendation, precise search, user portrait construction and other core businesses.

[0003] However, the existing label system has the following defects in the operation and maintenance mechanism:

[0004] Label evolution lags behind: the traditional manually driven label construction and maintenance mode cannot match the exponential growth speed of Internet content and the rapid iteration characteristics of semantic concepts. Emerging topics often have a significant time delay from their appearance to the response of the label system. This time sequence mismatch causes the label system to always be in a "catching up" state, and it cannot realize the forward-looking coverage of changes in the content ecosystem. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a large language model-based open domain label system construction method and device, which does not require human intervention from label generation, system initialization to normalization optimization, greatly shortens the time period from the appearance of emerging topics to the response of the label system, and enables the label system to synchronously perceive changes in the enterprise content ecosystem, no longer in a "catching up" state, and realizes timely response to the iteration of content semantics.

[0006] In a first aspect, a method for constructing an open-domain label system based on a large language model is provided. The method can include: constructing a label association pool based on different content to be labeled of a target enterprise in a first time period and initial labels output by labeling the different content to be labeled using a trained first large language model, the label association pool including an association relationship between the different content to be labeled and corresponding initial labels; the first large language model is trained based on a constructed supervised data set; determining an initial label system based on a label occurrence frequency of each initial label in the content to be labeled; determining a label clustering cluster of any initial label in the initial label system based on a similarity between the initial label and any initial label other than the initial label; determining a normalized label of each initial label by processing each label in the label clustering cluster of the different initial labels using a zero-shot capability of a second large language model based on a pre-set semantic understanding task description; constructing a label directed graph set with each initial label in the initial label system as a node and a mapping relationship between the initial label and the corresponding normalized label as an edge, the label directed graph set including a label directed graph constructed by different initial labels and corresponding normalized labels; if it is detected that a first label in a first label directed graph is a normalized label of a second label in a corresponding label directed graph, and the normalized label of the first label in the second label directed graph is a third label, then constructing a target label system, the target label system including a mapping relationship between the first label and the third label.

[0007] In one possible implementation, before constructing the label association pool, the method further includes: configuring Prompt template information for a third large language model having Few-shot learning capability, the Prompt template information including Prompt templates of different content types and pre-set label definition rules; generating a Few-shot learning example set based on different content of the target enterprise in a second time period and artificial labels of corresponding content satisfying the pre-set label definition rules; and labeling the different content based on the learned third large language model to generate a supervised data set; the supervised data set including different content and corresponding original labels; using each content in the supervised data set as an input training sample and using the corresponding original label as a real label to iteratively train the pre-trained first large language model to obtain the trained first large language model.

[0008] In one possible implementation, the label definition rules include a label broadness determination rule, a label fineness determination rule, and a label format normalization requirement.

[0009] In a possible implementation, the initial label system is determined based on the label occurrence frequency of each initial label in the to-be-labeled content, including: obtaining the label occurrence frequency of each label in the to-be-labeled content in a first time period, the content coverage of each initial label in the corresponding to-be-labeled content, and the number of labels in the label association pool; determining a distribution curve between the number of labels and the content coverage under different label occurrence frequencies based on the number of labels, the content coverage of each initial label, and the label occurrence frequency of each initial label; according to the distribution curve, the label occurrence frequency corresponding to the minimum content coverage in the content coverage greater than a preset content coverage threshold is determined as a first frequency threshold; and a set of initial labels corresponding to the label occurrence frequency greater than the first frequency threshold is determined as the initial label system.

[0010] In a possible implementation, the label clustering cluster of any initial label in the initial label system is determined based on the similarity between the initial label and any initial label other than the initial label, including: calculating the similarity between any initial label in the initial label system and any initial label other than the initial label based on the initial label; and selecting an initial label with a similarity higher than a preset similarity threshold to the initial label to form a label clustering cluster of the initial label.

[0011] In a possible implementation, the normalized label of each initial label is determined by processing each label in the label clustering cluster of each initial label using the zero-shot capability of the second large language model based on the preset semantic understanding task description, including: processing each label in the label clustering cluster of each initial label using the zero-shot capability of the second large language model based on the preset semantic understanding task description to obtain a center label of each label in the corresponding label clustering cluster; if any initial label in the initial label system corresponds to one label clustering cluster, the center label of the label clustering cluster is determined as the normalized label of the initial label; if any initial label in the initial label system corresponds to multiple label clustering clusters, and there is one center label with the highest label occurrence frequency among the center labels of the label clustering clusters, the center label with the highest label occurrence frequency is determined as the normalized label of the initial label; if any initial label in the initial label system corresponds to multiple label clustering clusters, and there are multiple center labels with the highest label occurrence frequency among the center labels of the label clustering clusters, a configured comprehensive quality score algorithm is used to process the multiple center labels with the highest label occurrence frequency respectively to obtain a comprehensive quality score of the corresponding center label, and the center label with the highest comprehensive score is determined as the normalized label of the initial label.

[0012] In a possible implementation, after the target label system is constructed, the method further includes: obtaining a plurality of to-be-added labels, and continuously counting label occurrence frequencies of the to-be-added labels every day, taking a to-be-added label with a label occurrence frequency greater than a second frequency threshold as a candidate label, and constructing a candidate set; calculating semantic similarities between any candidate label in the candidate set and each initial label in the initial label system; determining an initial label in the initial label system corresponding to the highest semantic similarity as a matching label that best matches the candidate label; if the highest semantic similarity is not less than a preset semantic similarity threshold, adding the candidate label as a synonym label of the matching label to the initial label system, and establishing a mapping relationship between the candidate label and the matching label corresponding to the normalized label in the target label system; if the highest semantic similarity is less than the preset semantic similarity threshold, adding the candidate label to the initial label system and the target label system.

[0013] In a second aspect, an open domain label system construction device based on a large language model is provided, which can include:

[0014] The construction unit is configured to construct a label association pool according to different to-be-labeled contents of the target enterprise in a first time period and initial labels output by labeling the different to-be-labeled contents by using a trained first large language model, the label association pool including an association relationship between the different to-be-labeled contents and corresponding initial labels; and the first large language model is trained based on a constructed supervised data set.

[0015] The determination unit is configured to determine an initial label system based on label occurrence frequencies of the output initial labels in the to-be-labeled contents, and determine a label clustering cluster of an initial label based on a similarity between the initial label and any initial label other than the initial label; and determine a normalized label of the initial label by processing each label in the label clustering cluster of the different initial labels based on a preset semantic understanding task description and a zero-shot capability of a second large language model.

[0016] The construction unit is further configured to construct a label directed graph set with each initial label in the initial label system as a node and a mapping relationship between the initial label and a corresponding normalized label as an edge, the label directed graph set including different initial labels and corresponding normalized labels constructing a label directed graph; and construct a target label system if a first label in a first label directed graph is detected to be a normalized label of a second label in a corresponding label directed graph, and a normalized label of the first label in a second label directed graph is a third label, the target label system including a mapping relationship between the first label and the third label.

[0017] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus;

[0018] The memory is configured to store a computer program.

[0019] The processor is configured to execute the program stored in the memory to implement the method steps of any one of the first aspect.

[0020] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method steps of any one of the first aspect.

[0021] The application provides a large language model-based open domain label system construction method and device. The method constructs a label association pool according to different to-be-labeled contents of a target enterprise in a first time period and initial labels output by labeling different to-be-labeled contents by using a trained first large language model. The label association pool includes the association relationship between different to-be-labeled contents and corresponding initial labels. The first large language model is obtained by training a constructed supervised data set. An initial label system is determined based on the label occurrence frequency of each initial label output in the to-be-labeled content. A label clustering cluster of any initial label in the initial label system is determined based on the similarity between the initial label and any initial label other than the initial label. The normalized labels of different initial labels are determined by processing each label in the label clustering cluster of each initial label based on the preset semantic understanding task description and using the zero-shot capability of a second large language model. A label directed graph set is constructed with each initial label in the initial label system as a node and the mapping relationship between each initial label and the corresponding normalized label as an edge. The label directed graph set includes different initial labels and corresponding normalized labels to construct a label directed graph. If it is detected that a first label in the first label directed graph is a normalized label of a second label in the corresponding label directed graph, and the normalized label of the first label in the second label directed graph is a third label, a target label system is constructed, and the target label system includes the mapping relationship between the first label and the third label. This method does not require human intervention from label generation, system initialization to normalization optimization, greatly shortens the response time from the emergence of a new topic to the label system, enables the label system to synchronously perceive changes in the enterprise content ecosystem, and is no longer in a "catching up" state, realizes timely response to content semantic iteration, improves the semantic precision of the label system through semantic similarity clustering, zero-shot semantic understanding and multi-hop mapping optimization, reduces redundancy and ambiguity, provides precise and efficient label support for intelligent scenarios such as downstream content recommendation, accurate retrieval and customer profiling, and can adapt to enterprise individualized business needs through customized model training and task description, and balance generalization and customization value. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a method for constructing an open-domain tagging system based on a large language model, provided in an embodiment of this application;

[0024] Figure 2 A schematic diagram of a statistical chart provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram of a tag-directed graph provided in an embodiment of this application;

[0026] Figure 4 A schematic diagram illustrating the construction process of a target labeling system provided in an embodiment of this application;

[0027] Figure 5 A flowchart illustrating a daily automatic update mechanism provided in an embodiment of this application;

[0028] Figure 6 A schematic diagram of the structure of an open domain tagging system construction device based on a large language model provided in this application embodiment;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application. Unless otherwise defined, the technical terms or scientific terms used in the present application should be understood as the general meaning understood by those of ordinary skill in the art to which the present application belongs. The terms "first", "second", and similar terms used in the present application do not represent any order, number, or importance, but are only used to distinguish different components. The terms "include" or "contain" and similar terms mean that the elements or objects before the terms cover the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect", "couple", or "connect" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0031] The existing open domain tag system construction follows the traditional paradigm of static predefinition and manual maintenance, which has a fundamental timing adaptation defect. The core problem of this mechanism is the structural mismatch between the evolution speed of semantic concepts and the response ability of the tag system, causing timing delay: the traditional manually driven tag construction and maintenance process includes multiple serial links such as "concept recognition → semantic analysis → tag design → review and release", and the overall response period is often measured in weeks or months. The emerging semantic concepts in the Internet content ecosystem grow exponentially, and the time window from the appearance of a hot topic to its peak is usually only hours or days. This order of magnitude difference in timing scale causes the tag system to always be in a "catching up" state, and it cannot achieve synchronous perception and forward-looking coverage of semantic changes in content. At the same time, the current mainstream large language model application paradigm has the dual technical defects of capability constraints and paradigm limitations: (1) Existing solutions force large language models with open language generation capabilities to be constrained within a pre-defined discrete tag space, essentially reducing the unbounded generation problem of continuous semantic space to a bounded decision problem of discrete selection space. This results in a loss of several orders of magnitude in expression capability. (2) The zero-shot and few-shot capabilities of large language models obtained through large-scale pre-training cannot be effectively utilized under the closed set paradigm. The model is forced to compress and map the rich semantic knowledge learned in the pre-training phase to a limited pre-defined concept space, resulting in approximate matching rather than precise generation when facing emerging semantic concepts, which severely restricts the sensitivity and adaptability of the tag system to semantic changes.

[0032] To solve the above problems, the application provides an open domain label system construction method and device based on a large language model. The method discards the limitations of traditional artificial pre-defined closed label set and breaks through the traditional practice of allowing the large language model to select classification within a fixed label range. The large language model is used as a driving engine to construct an intelligent label system with open domain characteristics, fine-grained recognition ability, and self-adaptive updating mechanism. Compared with the traditional pre-defined fixed label set, the label system constructed by the application has openness and stronger semantic expression ability. That is, the application fundamentally breaks through the paradigm constraints of traditional closed set classification. Through core technical innovations such as generative semantic modeling, dynamic label evolution, and multi-granularity adaptive normalization, an intelligent label system that can autonomously perceive semantic changes, respond to content evolution in real time, and dynamically adjust label granularity is constructed. The core innovation of the application is that the large language model is transformed from a "selection judge" to a "semantic creator", realizing a fundamental technological leap from static pre-definition to dynamic self-generation, from fixed granularity to adaptive granularity, and from artificial maintenance to machine autonomy, and providing a new technical path for solving the label bottleneck problem in large-scale content understanding.

[0033] The preferred embodiments of the application are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the application, and are not used to limit the application, and the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0034] Figure 1 A flowchart of a large language model-based open domain label system construction method provided by an embodiment of the application is shown in FIG. 1. As shown in FIG. 1, the method can include the following steps. Figure 1

[0035] In step S110, a label association pool is constructed according to different content to be labeled of the target enterprise in the first time period and initial labels output by labeling the different content to be labeled using the trained first large language model.

[0036] Before performing this step, to ensure that the first large language model can stably output labels in a specified format, the first large language model can be trained in the following manner, including:

[0037] ​Step A, configuring Prompt template information for the third largest language model with Few-shot learning capability, the Prompt template information can include Prompt templates of different content types (including event class content that tells a story around an event and object class content that describes or introduces a related topic around an object) and preset label definition rules. Among them, the preset label definition rules can include label broad determination rules, label fine determination rules and label format standardization requirements.

[0038] (1) Label broad determination rule: the label cannot accurately understand the original content theme after the label is separated from the original content. For example, if the original content is an article introducing the TV series Journey to the West, the output label is "TV series", and when looking at this label alone, it cannot know that the content theme is Journey to the West. The correct label should be a label of the "TV series Journey to the West" type.

[0039] (2) Label fine determination rule: the label is too fine and long. For example, the content is a technical sharing article that mainly introduces the blogger's experience of learning python in 30 days. If the output label is "how to quickly learn python in 30 days", this label is too fine. Although the label can accurately reflect the content theme when it is separated from the original content, when considering the use of downstream tasks, the content that can be corresponded to by this label will be very small. Therefore, the correct label can be "python learning". At the same time, in order to prevent the generation of such fine labels, it can be specified in the Prompt template for generating labels that the number of characters in the label should be within 15 characters to a certain extent to avoid such problems.

[0040] (3) Label format standardization requirement: in order to facilitate the automatic processing and analysis of downstream systems, all generated labels must follow a unified format specification for standardized output, such as all labels being structured information output with "," as the separator, and all letters being in lowercase.

[0041] Step B, based on the different content (including high-quality Q&A content and original articles) obtained by the target enterprise in the second time period (such as the last 3 years) and the corresponding content that meets the preset label definition rule, a Few-shot learning example set is generated; Specifically, 3-5 typical samples are selected from different content, and the original label is manually labeled as Few-shot learning example, thereby forming a Few-shot learning example including "input content-output corresponding artificial label" mapping pair, so as to learn the generated Few-shot learning example set and the configured Prompt template information by the third large language model, and learn the label generation rule that meets the preset label definition rule. Then, based on the learned third large language model, the different content in the second time period is labeled to generate a supervised data set. The supervised data set can include different content and corresponding original labels.

[0042] Further, the labels automatically generated by the third large language model can be comprehensively manually reviewed, and the labeling items that are semantically irrelevant, low quality or do not conform to the format are strictly excluded, thereby obtaining a supervised data set.

[0043] Step C, taking each content in the supervised data set as a training sample and the corresponding original label as a true label, the pre-trained first large language model is iteratively trained to obtain a trained first large language model. Specifically, from the supervised data set, about a preset number of labeled data, such as 12,000, is selected by random sampling method to constitute a training data set required for fine-tuning of the pre-trained first large language model. The data set is carefully designed to ensure that it covers a variety of label types and content scenarios, providing sufficient semantic coverage for model training. Based on the pre-trained first large language model (such as MiniCPM-2B model), full-parameter supervised fine-tuning is implemented on the constructed training data set, focusing on optimizing the following two core objectives: enhancing the model's ability to recognize and understand fine-grained semantic features, improving the semantic accuracy and content relevance of generated labels, and strengthening the model's strict adherence to predefined label format specifications through repeated training, ensuring that the output results meet uniform format requirements such as comma separation, label quantity limitation, etc.

[0044] Further, the first large language model that has been supervised and fine-tuned is deployed to the content automatic labeling task to generate batch labels for the actual Q&A and article content involved in the task. Representative content samples are randomly selected from the labeling results to construct an independent test evaluation data set. The automatically generated labels in the test set are professionally evaluated from multiple dimensions such as label semantic accuracy, content relevance, and format specification, and the final quantitative evaluation results of the model label accuracy are obtained, and the specific label accuracy evaluation results are shown in Table 1:

[0045] Table 1

[0046]

[0047] Optionally, the supervised data set can be constructed in the above manner, or can be constructed manually in advance.

[0048] Returning to step S110, in a specific implementation, the target enterprise's to-be-labeled content in the first time period is obtained. The first time period is a recent time period ending at the current time, such as a time period of the last 2 years. The to-be-labeled content can be high-quality question and answer content and original articles. "High quality" represents that the question and answer content and the original articles can accurately solve user needs, and have outstanding performance in content value, credibility, and expression logic. It can be understood that the high-quality question and answer content and the original articles can be selected by humans, or the computer can use a recommendation algorithm to recommend content according to user reading behavior, interest preferences, etc., or obtain article resources on the Internet through a web crawler, and then use information filtering technology (such as setting keyword filtering, content duplication detection, etc.) to screen out original articles that meet the quality requirements. The present application does not limit this. The first time period and the second time period can be the same, or the second time period can be greater than the first time period.

[0049] Then, the to-be-labeled content is input into the trained first large language model to label the input to-be-labeled content using the trained first large language model to obtain initial labels output by the first large language model; and based on the association relationship between different to-be-labeled content and corresponding initial labels, a label association pool is constructed, that is, the label association pool can include the association relationship between different to-be-labeled content and corresponding initial labels. The first large language model is trained based on the constructed supervised data set, and the supervised data set is determined based on different content in the second time period and a preset label definition rule.

[0050] Step S120, determining an initial label system based on the label occurrence frequency of each initial label output in the to-be-labeled content.

[0051] In a specific implementation, if the first frequency threshold is a preconfigured frequency threshold, the label occurrence frequency of each initial label output in different to-be-labeled content in the first time period can be obtained, and based on each initial label with a label occurrence frequency greater than the first frequency threshold, an initial label system is generated.

[0052] If the first frequency threshold is not a pre-configured frequency threshold, then the frequency of tag occurrences (or "tag occurrence counts") of each initial tag in different content to be labeled within the first time period, the content coverage rate (or "coverage content ratio") of each initial tag relative to the corresponding content to be labeled, and the number of tags in the tag association pool can be obtained. Based on the number of tags, the content coverage rate of each initial tag, and the frequency of tag occurrences of each initial tag, a distribution curve between the number of tags and the content coverage rate under different tag occurrence frequencies can be determined. Then, according to the distribution curve, the first frequency threshold is determined based on the tag occurrence frequency and the content coverage rate; and the set of tags corresponding to the tag occurrence frequencies greater than the first frequency threshold is determined as the initial tag system.

[0053] In one example, the relationship between the frequency of each initial tag's occurrence, the percentage of content it covers, and the number of tags can be referenced as follows: Figure 2 The statistical chart shown exhibits a typical long-tail distribution between the initial tags and content coverage. Specifically, the head characteristic is that a few high-frequency "head" tags cover the vast majority of content, demonstrating strong representativeness and universality; the tail characteristic is that a large number of low-frequency "tail" tags cover only a very small amount of specific content, with relatively limited application value. Based on the above statistical analysis results, and considering the balance between the tag set size (i.e., the number of tags) and content coverage, this application has formulated the following tag selection strategy: 1) Selecting high-frequency tags (or "head tag set") that appear more than 15 times (i.e., the first frequency threshold) as the core tag set. This threshold effectively balances the manageability of the number of tags with the sufficiency of content coverage; 2) The selected head tag set covers more than 94% of the content on the site, totaling approximately 3.6 million valid tags, constituting the initial tag system of this application, or the "Top360w tag set." In other words, under the premise of a coverage rate of not less than 94%, the frequency with the fewest tags is selected as the threshold. As can be seen from the figure, when the frequency of a tag is greater than or equal to 15 times, the content coverage is close to 94%, and the number of tags is relatively small, achieving a balance between the number of tags and the content coverage. Therefore, 15 times is determined as the first frequency threshold.

[0054] Step S130: Based on the similarity between any initial label in the initial label system and any other initial label, determine the label cluster of the initial label.

[0055] In practice, based on any initial label in the initial label system, the similarity between the initial label and any other initial label is calculated; initial labels with similarity higher than a preset similarity threshold are selected to form the label cluster of the initial label, thereby obtaining the label cluster of each initial label in the initial label system.

[0056] Specifically, the initial labels in the initial label system are vectorized and encoded using the Embedding model to generate high-dimensional semantic vector representations, i.e., label vectors of the initial labels; based on cosine similarity measurement, the most similar label vectors to each label vector of the initial labels in the vector space of the initial label system are retrieved, which can be set to the top 50 labels greater than a certain threshold, to construct a label clustering cluster corresponding to each initial label in the Top360w label set.

[0057] In step S140, based on the preset semantic understanding task description, the zero-shot capability of the second large language model is used to process each label in the label clustering cluster of different initial labels to determine the normalized label of each label.

[0058] Based on the preset semantic understanding task description, the zero-shot capability of the second large language model is used to process each label in the label clustering cluster of different initial labels to determine the normalized label of each label.

[0059] Specifically, the determined labels in the label clustering cluster of different initial labels are input into a large language model (such as GPT-4) with zero-shot capability in the format of python list, and the large language model determines the semantic representativeness and universality of each label in the label clustering cluster according to the preset semantic understanding task description through the zero-shot capability, and outputs the most representative center label of the label clustering cluster. The output format is python dict nested in python list. Further, the center label of the label clustering cluster can be regarded as the key of the label clustering cluster, and each label in the label clustering cluster can be regarded as the value corresponding to the key.

[0060] If any initial label in the initial label system corresponds to a label clustering cluster, the center label of the label clustering cluster of the initial label is determined as the normalized label of the initial label.

[0061] If any initial label in the initial label system corresponds to multiple label clustering clusters, and there is a center label with the highest label frequency among the center labels of the label clustering clusters, the center label with the highest label frequency is determined as the normalized label of the initial label. At this time, a certain target initial label in the initial label system corresponds to multiple label clustering clusters, i.e., a certain target initial label corresponds to multiple keys, and the key with the highest label frequency is determined as the normalized label of the target initial label in the initial label system.

[0062] If any initial label in the initial label system corresponds to multiple label clustering clusters, and there are multiple center labels with the highest label occurrence frequency in the center labels of each label clustering cluster, then the configured comprehensive quality score algorithm is used to process the multiple center labels with the highest label occurrence frequency, to obtain the comprehensive quality score of the corresponding center label, and the center label with the highest comprehensive score is determined as the normalized label of the initial label. At this time, a target initial label in the initial label system corresponds to multiple label clustering clusters, that is, a target label corresponds to multiple keys, and there are at least two keys with the highest and equal label occurrence frequency in the multiple keys. At this time, the comprehensive quality score of each key is calculated, and the key with the highest comprehensive score is determined as the normalized label of the target initial label.

[0063] The above embodiment can effectively eliminate the multiple selection ambiguity problem.

[0064] The configured comprehensive quality score algorithm is a linear weighted scoring function, which combines "literal similarity" (number of repeated characters) and "commonality" (label occurrence frequency) two dimensions and gives different weights to solve the case of multiple center labels with the same frequency in the semantic aggregation process.

[0065] In one example, a target initial label in the initial label system corresponds to multiple keys, and there are at least two target keys with the highest and equal label occurrence frequency in the multiple keys. At this time, the comprehensive quality score algorithm is used to calculate the comprehensive quality score of each key, and the key with the highest comprehensive score is determined as the normalized label of the target initial label.

[0066] In one example, the comprehensive quality score algorithm can be expressed as:

[0067] ;

[0068] In the formula, i is the i-th target key in the target key with the highest and equal label occurrence frequency, is the comprehensive quality score of the i-th target key, is the number of repeated characters of the i-th target key and the values of the label clustering cluster corresponding to the i-th key, is the label occurrence frequency of the i-th target key, is the character length of the i-th target key, is the control weight of the number of repeated characters, is the label occurrence frequency weight, and the values of the two weights are both [0, 1], and .

[0069] It can be understood that the number of repeated characters refers to the number of characters shared by the ith target key and each value of the corresponding label cluster, and the positions of the characters do not have to be continuous.

[0070] In step S150, any initial label in the initial label system is taken as a node, and the mapping relationship between the initial label and the corresponding normalized label is taken as an edge, and a label directed graph set is constructed, wherein the label directed graph set includes different initial labels and corresponding normalized labels constructing a label directed graph.

[0071] Specifically, each initial label in the initial label system (i.e., the Top360w label set) is taken as a node, the mapping relationship between the corresponding initial label and the corresponding normalized label is taken as an edge, and the mapping relationship is arranged in order from low to high according to the label occurrence frequency, so as to ensure that there is no loop structure in the mapping relationship, and a standard directed acyclic graph topology is formed, i.e., a label directed graph is constructed, and the constructed label directed graph can avoid the problem of circular dependency.

[0072] In an example, the label "Master's postgraduate enrollment matters needing attention" with a label occurrence frequency of 11, the label "postgraduate enrollment" with a label occurrence frequency of 120, and the label "postgraduate enrollment" with a label occurrence frequency of 560 are taken as examples, wherein the normalized label of the label "Master's postgraduate enrollment matters needing attention" is the label "postgraduate enrollment", and the normalized label of the label "postgraduate enrollment" is the label "postgraduate enrollment", therefore, in order from low to high according to the label occurrence frequency, the connection arrows from the label "Master's postgraduate enrollment matters needing attention" to the label "postgraduate enrollment" and the connection arrows from the label "postgraduate enrollment" to the label "postgraduate enrollment" can be obtained, and thus the label directed graph set constructed by the label directed graph between each label and the corresponding normalized label is obtained. Figure 3

[0073] In step S160, a target label system is constructed based on the detection result of the mapping relationship between the labels involved in different label directed graphs in the label directed graph set.

[0074] The mapping relationship between the labels involved in the first label directed graph and the second label directed graph of the first label directed graph and the second label directed graph in the label directed graph set is detected.

[0075] ​If it is detected that the first label in the first label directed graph is the normalized label of the second label in the corresponding label directed graph, and the normalized label of the first label in the second label directed graph is the third label, a mapping relationship between the first label and the third label is established, and a target label system is constructed, which includes the mapping relationship between the first label and the third label. That is, the "original label → intermediate label → normalized label" existing in the label directed graph set is merged by multiple hops, a direct connection relationship between the source label and the final target label is established, an efficient "multi-hop merging" effect is realized, and a "Top100w label set" is obtained. Through the "multi-hop path compression" strategy, not only the interpretability of the label mapping process can be maintained, but also the overhead during downstream calling can be significantly reduced.

[0076] In the above example, in combination with Figure 3 , the label "Master's postgraduate enrollment matters needing attention" and the label "postgraduate enrollment" can establish a mapping relationship through the label "postgraduate enrollment registration", so the mapping relationship between the label "Master's postgraduate enrollment matters needing attention" and the label "postgraduate enrollment" is connected by a connection arrow, and the mapping relationship between the label "Master's postgraduate enrollment matters needing attention" and the label "postgraduate enrollment" is stored in the constructed target label system, wherein the construction process of the target label system is as shown in Figure 4 .

[0077] Through the above implementation mode, the initial Top360w label set is significantly refined and optimized. 1) On the processing result: 3.6 million labels are efficiently merged and compressed into about 1 million normalized labels to form a refined label system (hereinafter referred to as "Top100w label set"); 2) On the mapping integrity: each original label establishes a complete mapping relationship to the Top100w normalized label set, ensuring full coverage and traceability of the label conversion process; 3) On the optimization effect: under the premise of fully maintaining the semantic characteristics of the label, the content coverage rate of a single label is significantly improved, effectively improving the practicability and application efficiency of the label system.

[0078] In some embodiments, after constructing the target label system, in order to realize the dynamic updating of the T+1 period label system and ensure that the label set can respond to content changes and hot trends in time, the following daily automatic updating mechanism can also be performed, specifically:

[0079] A plurality of labels to be added are obtained, and the frequency of occurrence of each label to be added is continuously counted every day. The label to be added with a frequency of occurrence greater than a second frequency threshold is taken as a candidate label, and a candidate set C is constructed.

[0080] Specifically: A comprehensive statistical analysis is conducted on all tags not included in the Top360w tag set, defining them as "non-Top tags" (i.e. tags to be added), including: newly generated tags on the same day (such as newly added tags daily) and existing tags with low frequency in the past (such as old tags with a cumulative frequency of <15 / historical long-tail tags); a second frequency threshold is set (such as ≥20 times / day), and tags to be added with a frequency greater than the second frequency threshold are used as candidate tags to construct a candidate set C.

[0081] Next, the semantic similarity between any candidate label in the candidate set C and each label in the initial label system is calculated; the label in the initial label system corresponding to the highest semantic similarity is determined as the best matching label with that candidate label.

[0082] If the highest semantic similarity is not less than the preset semantic similarity threshold (e.g., 0.8), the candidate tag will be added to the initial tag system as a synonym of the matching tag, and a mapping relationship between the candidate tag and the normalized tag corresponding to the matching tag will be established in the target tag system.

[0083] If the highest semantic similarity is less than the preset semantic similarity threshold, the candidate label will be added to the initial label system and the target label system.

[0084] like Figure 5 As shown, the daily automatic update mechanism process includes: finding the difference between the tags corresponding to all the content involving the target enterprise and the tags in the Top360w tag set, i.e., finding tags not included in the Top360w tag set and treating them as non-Top tags; and then treating non-Top tags whose frequency of occurrence is greater than the second frequency threshold as Top tags to be added (or "tags to be added").

[0085] Obtain the embedding index of the Top360w tag set, that is, obtain the tag vector of each tag in the Top360w tag set, calculate the similarity (or "semantic similarity") between the tag vector of each tag and the tag vector of the Top tag to be added, and obtain the T1 tag with the highest similarity, that is, obtain the T1 tag with the highest similarity to the Top tag to be added (the T1 tag is a tag in the Top360w tag set).

[0086] Determine whether the highest similarity score is greater than the similarity threshold Y;

[0087] If so, the Top tag to be added will be added as a synonym mapping to the Top360w tag set and merged into the same Top100w tag set as T1 (that is, establish a mapping relationship between the Top tag to be added and the normalized tag corresponding to T1).

[0088] If not, the Top tag to be added is added to the Top 360w tag set as a new tag and to the Top 100w tag set as a new tag.

[0089] The present application realizes full-automatic expansion and maintenance of the label system through the T+1 dynamic updating process described above, the label system can maintain a high degree of real-time sensing ability of content changes, timely capture emerging topics and hot trends; and can automatically identify, filter and integrate new labels according to content evolution, ensuring the completeness and timeliness of label coverage; the whole updating process does not need manual operation intervention and maintenance, significantly reducing the labor cost and operation complexity of label system management, effectively solving the core pain point that the traditional static label system is difficult to adapt to the rapid change of content.

[0090] In some embodiments, whether the label is too wide can be determined by whether the information entropy loss rate of the output label L after being separated from the content to be labeled C is greater than τ1. If the label L cannot uniquely reconstruct the core theme of C after being separated from the content C (i.e., the information entropy loss rate ΔH>τ1), it is considered too wide.

[0091] Whether the label is too fine can be determined by whether the description length of the output label L exceeds 15 characters and whether the expected content coverage P(C|L) of the output label L is less than τ2 (based on historical statistical prior estimation). If the description length of the output label L exceeds 15 characters and the expected content coverage P(C|L)<τ2, it is considered too fine.

[0092] Whether the label meets the format standardization requirement can be determined by whether the output format is in lowercase and uses a comma-separated string sequence (such as tag1,tag2).

[0093] In some embodiments, in order to meet the demand for semantic relevance and generality, avoid the dominance of a single factor, and balance the influence of a single factor while strengthening the contribution of both, the product term can be used to enhance the contribution of both, while the basic linear term can be used to balance the influence of a single factor. The specific calculation logic can include:

[0094] First, the ratio of the number of repeated characters to the character length is denoted as A, which is normalized to the interval [0,1] to eliminate the influence of label length difference;

[0095] Second, the ratio of the label occurrence frequency to the highest label occurrence frequency in the label clustering cluster corresponding to the i-th target key is denoted as B, which is normalized to the interval [0,1] to avoid excessive monopoly of high-frequency labels;

[0096] Finally, the comprehensive quality score algorithm can be expressed as: ;

[0097] wherein, and are the balance coefficients (both sum to 1) of the synergistic term (product term) and the basic term (sum term), the product term can highlight the labels that are both semantically relevant and high-frequency, while the sum term avoids the misselection of labels with extremely high single factor but extremely low another factor (for example, when A = 0.9, B = 0.1, the product term = 0.09, and the sum term = 0.5, the comprehensive score = 0.6 x 0.09 + 0.4 x 0.5 = 0.254, which is not too high). = 0.6, = 0.4, the comprehensive score = 0.6 x 0.09 + 0.4 x 0.5 = 0.254, which is not too high).

[0098] In some embodiments, the weight and ratio can be inversely deduced by the degree of semantic difference within the label clustering cluster, and the core logic is: the greater the semantic difference, the more it needs to rely on semantic relevance, and the more concentrated the semantics, the more it needs to rely on frequency universality. The specific operation is as follows:

[0099] First, calculate the semantic dispersion of each label clustering cluster, specifically: calculate the standard deviation (or average cosine distance) through the Embedding vectors of all labels within the cluster, the higher the dispersion, the greater the semantic difference of the labels within the cluster (for example, when the label clustering cluster with "Amazon after-sales" as the key mixes in the "Apple repair" label, the dispersion significantly increases);

[0100] Subsequently, normalize the semantic dispersion to the [0, 1] interval, and directly take the normalized dispersion value as , then take 1- . For example, the semantic dispersion of a certain cluster is normalized to 0.7, indicating that the semantic difference within the cluster is large, and it is necessary to focus on screening semantically matched labels through repeated character counts, so = 0.7, = 0.3; if another cluster has a normalized dispersion of 0.2, the semantics are highly concentrated, so frequency can better reflect universality, so = 0.2, = 0.8.

[0101] Corresponding to the above method, the embodiments of the present application also provide an open domain label system construction device based on a large language model, as shown in Figure 6 , the device comprises:

[0102] The construction unit 610 is configured to construct a label association pool according to different to-be-labeled content of the target enterprise in a first time period and initial labels output by labeling the different to-be-labeled content by using a trained first large language model, the label association pool including an association relationship between the different to-be-labeled content and the corresponding initial labels; and the first large language model is trained based on a constructed supervised data set.

[0103] The determination unit 620 is configured to determine an initial label system based on a label occurrence frequency of each initial label in the to-be-labeled content, and determine a label clustering cluster of the initial label based on a similarity between any initial label in the initial label system and any initial label other than the initial label; and determine a normalized label of each initial label by processing each label in the label clustering cluster of the different initial labels based on a preset semantic understanding task description and a zero-shot capability of a second large language model.

[0104] The construction unit 610 is further configured to construct a label directed graph set with each initial label in the initial label system as a node and a mapping relationship between the initial label and the corresponding normalized label as an edge, the label directed graph set including a label directed graph constructed by different initial labels and corresponding normalized labels; and construct a target label system if it is detected that a first label in a first label directed graph is a normalized label of a second label in a corresponding label directed graph and a normalized label of the first label in the second label directed graph is a third label, the target label system including a mapping relationship between the first label and the third label.

[0105] The functions of each functional unit of the label system construction device based on the large language model provided in the above embodiments can be realized through the above method steps, and therefore, the specific working process and beneficial effects of each unit in the label system construction device based on the large language model provided in the embodiments of the present application will not be repeated here.

[0106] The embodiments of the present application further provide an electronic device, as shown in the figure. Figure 7 The processor 710, the communication interface 720 and the memory 730 can communicate with each other through the communication bus 740.

[0107] The memory 730 is configured to store a computer program.

[0108] The processor 710 is configured to execute the program stored in the memory 730 to implement the following steps:

[0109] According to different contents to be labeled of the target enterprise in a first time period and initial labels output by labeling the different contents to be labeled using a trained first large language model, a label association pool is constructed, the label association pool including an association relationship between the different contents to be labeled and corresponding initial labels; the first large language model is trained based on a constructed supervised data set;

[0110] Based on the label occurrence frequency of each initial label in the content to be labeled, an initial label system is determined:

[0111] Based on the similarity between any initial label in the initial label system and any initial label other than the initial label, a label clustering cluster of the initial label is determined;

[0112] Based on a preset semantic understanding task description, the zero-shot capability of a second large language model is used to process each label in the label clustering cluster of different initial labels, to determine the normalized label of each initial label;

[0113] Taking each initial label in the initial label system as a node and the mapping relationship between the initial label and the corresponding normalized label as an edge, a label directed graph set is constructed, the label directed graph set including a label directed graph constructed by different initial labels and corresponding normalized labels;

[0114] If it is detected that a first label in a first label directed graph is a normalized label of a second label in a corresponding label directed graph, and the normalized label of the first label in the second label directed graph is a third label, a target label system is constructed, the target label system including the mapping relationship between the first label and the third label.

[0115] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0116] The communication interface is used for communication between the electronic device and other devices.

[0117] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0118] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0119] Since the implementation manners and beneficial effects of the electronic device in the above-mentioned embodiments can solve the problems can be achieved by referring to the steps in the embodiments shown in the above-mentioned embodiments, the specific working process and beneficial effects of the electronic device provided by the embodiments of the present application are not repeated here. Figure 1 The specific working process and beneficial effects of the electronic device provided by the embodiments of the present application are not repeated here.

[0120] In another embodiment provided by the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores instructions, when the instructions are run on a computer, the computer executes the method for constructing an open domain label system based on a large language model in any of the above-mentioned embodiments.

[0121] In another embodiment provided by the present application, a computer program product containing instructions is also provided, when the instructions are run on a computer, the computer executes the method for constructing an open domain label system based on a large language model in any of the above-mentioned embodiments.

[0122] Those skilled in the art should understand that the embodiments in the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the embodiments in the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments in the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0123] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0124] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0125] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks ​ means for functionally implementing the steps listed in the flowchart block or blocks.

[0126] While preferred embodiments of the application have been described, modifications and variations can be apparent to those skilled in the art once aware of the general underlying concepts. Therefore, it is intended that the scope of the appended claims should include all such modifications and variations.

[0127] Obviously, numerous modifications and variations of the present embodiments are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for constructing an open domain label system based on a large language model, applied to text classification, characterized in that, The method comprises: According to the different content to be labeled of the target enterprise in the first time period and the initial labels output by labeling the different content to be labeled using the trained first large language model, a label association pool is constructed, the label association pool comprising the association relationship between the different content to be labeled and the corresponding initial labels; the first large language model is trained based on the constructed supervised data set; Based on the label occurrence frequency of each initial label in the content to be labeled, an initial label system is determined: Based on the similarity between any initial label in the initial label system and any initial label other than the initial label, a label clustering cluster of the initial label is determined; Based on the preset semantic understanding task description, the zero-shot ability of the second large language model is used to process each label in the label clustering cluster of different initial labels respectively, and the normalized label of each initial label is determined; Taking each initial label in the initial label system as a node and the mapping relationship between the initial label and the corresponding normalized label as an edge, a label directed graph set is constructed, the label directed graph set comprising the label directed graph constructed by different initial labels and corresponding normalized labels; If it is detected that the first label in the first label directed graph is the normalized label of the second label in the corresponding label directed graph, and the normalized label of the first label in the second label directed graph is the third label, a target label system is constructed, the target label system comprising the mapping relationship between the second label and the third label; Wherein, based on the label occurrence frequency of each initial label in the content to be labeled, the initial label system is determined, comprising: Obtain the label occurrence frequency of each label in the different content to be labeled in the first time period, the content coverage rate of each initial label in the corresponding content to be labeled, and the number of labels in the label association pool; Based on the number of labels, the content coverage rate of each initial label, and the label occurrence frequency of each initial label, a distribution curve between the number of labels and the content coverage rate under different label occurrence frequencies is determined; According to the distribution curve, the first frequency threshold is determined according to the label occurrence frequency and the content coverage rate; The set of initial labels corresponding to the label occurrence frequency greater than the first frequency threshold is determined as the initial label system.

2. The method of claim 1, wherein, Before constructing the label association pool, the method further comprises: Configure the Prompt template information for the third large language model with Few-shot learning ability, the Prompt template information comprising Prompt templates of different content types and preset label definition rules; Based on the different content of the target enterprise in the second time period and the artificial labels of the corresponding content satisfying the preset label definition rules, a Few-shot learning example set is generated; and based on the Few-shot learning example set, the third large language model after learning is obtained, and the different content is labeled using the third large language model after learning to generate a supervised data set; the supervised data set comprises different content and corresponding original labels; The pre-trained first large language model is iteratively trained by taking each content in the supervised data set as an input training sample and taking a corresponding original label as a true label, to obtain a trained first large language model.

3. The method of claim 2, wherein, The label definition rules include a label broadness judgment rule, a label fineness judgment rule, and a label format normalization requirement.

4. The method of claim 1, wherein, Based on the similarity between any initial label in the initial label system and any initial label other than the initial label, a label clustering cluster of the initial label is determined, including: Based on any initial label in the initial label system, the similarity between the initial label and any initial label other than the initial label is calculated. Selecting initial labels with a similarity higher than a preset similarity threshold to the initial label to form a label clustering cluster of the initial label.

5. The method of claim 1, wherein, Based on a preset semantic understanding task description, the zero-shot capability of the second large language model is used to process each label in the label clustering cluster of different initial labels to determine the normalized label of each initial label, including: Based on a preset semantic understanding task description, the zero-shot capability of the second large language model is used to process each label in the label clustering cluster of different initial labels to determine the normalized label of each initial label, including: If any initial label in the initial label system corresponds to a label clustering cluster, the center label of the label clustering cluster is determined as the normalized label of the initial label. If any initial label in the initial label system corresponds to multiple label clustering clusters, and there is a center label with the highest label occurrence frequency among the center labels of the label clustering clusters, the center label with the highest label occurrence frequency is determined as the normalized label of the initial label. If any initial label in the initial label system corresponds to multiple label clustering clusters, and there are multiple center labels with the highest label occurrence frequency among the center labels of the label clustering clusters, a configured comprehensive quality score algorithm is used to process the multiple center labels with the highest label occurrence frequency to obtain a comprehensive quality score of the corresponding center label, and the center label with the highest comprehensive score is determined as the normalized label of the initial label.

6. The method of claim 1, wherein, After the target label system is constructed, the method further includes: Obtaining a plurality of to-be-added labels, and continuously counting the label occurrence frequency of each to-be-added label every day, taking the to-be-added label with a label occurrence frequency greater than a second frequency threshold as a candidate label to construct a candidate set; Calculating the semantic similarity between any candidate label in the candidate set and each initial label in the initial label system; The initial label in the initial label system corresponding to the highest semantic similarity is determined as the matching label that best matches the candidate label. If the highest semantic similarity is not less than a preset semantic similarity threshold, the candidate label is added to the initial label system as a synonymous label of the matching label, and a mapping relationship between the candidate label and the normalized label corresponding to the matching label is established in the target label system. If the highest semantic similarity is less than a preset semantic similarity threshold, the candidate label is added to the initial label system and the target label system.

7. A large language model-based open domain label system construction apparatus applied to text classification, characterized in that, The device comprises: The construction unit is configured to construct a label association pool according to different to-be-labeled contents of the target enterprise in a first time period and initial labels output by labeling the different to-be-labeled contents by using a trained first large language model, the label association pool comprising an association relationship between the different to-be-labeled contents and corresponding initial labels, and the first large language model being trained based on a constructed supervised data set; The determination unit is configured to determine an initial label system based on label occurrence frequencies of each initial label in the to-be-labeled content, and determine a label clustering cluster of the initial label based on a similarity between any initial label in the initial label system and any initial label other than the initial label, process each label in the label clustering cluster of different initial labels based on a preset semantic understanding task description and a zero-shot capability of a second large language model, and determine a normalized label of each initial label. The construction unit is further configured to construct a label directed graph set with each initial label in the initial label system as a node and a mapping relationship between the initial label and a corresponding normalized label as an edge, the label directed graph set comprising a label directed graph constructed by different initial labels and corresponding normalized labels, and construct a target label system if a first label in a first label directed graph is detected to be a normalized label of a second label in a corresponding label directed graph and a normalized label of the first label in a second label directed graph is a third label, the target label system comprising a mapping relationship between the second label and the third label. The determination unit is specifically configured to acquire label occurrence frequencies of each label in different to-be-labeled contents in a first time period, content coverage rates of each initial label in corresponding to-be-labeled contents, and a number of labels in the label association pool, determine a distribution curve between a number of labels and a content coverage rate under different label occurrence frequencies based on the number of labels, the content coverage rates of each initial label, and the label occurrence frequencies of each initial label, determine a first frequency threshold according to the label occurrence frequencies and the content coverage rates according to the distribution curve, and determine a set of initial labels corresponding to label occurrence frequencies greater than the first frequency threshold as the initial label system.

8. An electronic device, comprising: The electronic device comprises a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is configured to store a computer program. The processor is configured to execute the program stored on the memory to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-6. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-6.

Images