Label classification method and device based on large model, equipment and medium

Abstract

The invention discloses a label classification method and device based on a large model, equipment and a medium, and relates to the technical field of natural language processing, model fine tuning, pre-training and model optimization are performed on an initial semantic retrieval model and an initial artificial intelligence large model on a preset data set, and a semantic retrieval model and an artificial intelligence large model are obtained; inputting the to-be-classified text into a semantic retrieval model, so that the semantic retrieval model screens out an initial tag corresponding to the to-be-classified text from a preset tag library; and inputting the to-be-classified text and the initial tag into the artificial intelligence large model, so that the artificial intelligence large model screens out the target tag corresponding to the to-be-classified text from the initial tag to complete tag classification of the to-be-classified text, thereby shortening the model training period, improving the development efficiency and the business iteration speed, and improving the user experience. The classification capability of the model on the middle tail labels is enhanced, the accuracy and comprehensiveness of the model during label classification are improved, and the flexibility and efficiency of model training are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly to a label classification method, device, equipment and medium based on a large model. Background Technique

[0002] Currently, text label classification is a key technology and is widely used in many scenarios, such as news classification, product annotation, content recommendation, etc. Currently, the mainstream text multi-label classification methods usually adopt pre-trained language models and perform parameter fine-tuning on specific datasets. This method performs well in scenarios with a small number of labels and a fixed label system, and has the advantages of relatively simple modeling methods, controllable effects and costs.

[0003] However, with the explosive growth of Internet content, in many practical applications, it is necessary to characterize content in multiple granularities and dimensions, and the label magnitude often reaches the order of hundreds of thousands or even millions. In the face of a massive label system, traditional methods have exposed many problems:

[0004] 1. Long training cycle: Due to the large number of labels, to ensure the learning effect of the model for each label, a large number of samples need to be labeled for each label, which makes the training sample size increase sharply. In the process of fitting a massive number of samples to a massive number of labels, the model often needs to be trained for at least dozens of rounds to converge, resulting in a significant extension of the sample annotation and model training cycle, seriously affecting the development efficiency and business iteration speed. For example, in a large content platform, for multi-label classification of an article, if it involves tens of thousands of labels, training according to the traditional method may take weeks or even months from sample annotation to model training completion;

[0005] 2. Poor generalization performance of the model: For head labels (i.e., labels with a large number of annotated samples), the model has a good fitting effect and high accuracy; but for middle and tail labels (i.e., labels with fewer or no annotated samples), the model accuracy is low and it is difficult to improve. This is because the annotated samples of middle and tail labels are scarce, and the model is difficult to learn enough feature information, resulting in poor classification effects on these labels in actual applications, unable to accurately identify relevant content, and affecting the comprehensiveness and accuracy of content classification;

[0006] 3. Long iteration cycle of the label system: With the continuous development and change of the business, the label category system also needs to be updated accordingly. Under the traditional method, every time a new batch of labels is added, a large number of samples need to be re-annotated and the model needs to be retrained. Due to the overall large magnitude of the annotated samples, the newly added samples are difficult to ensure good performance of the model on the new labels, resulting in slow iteration of the label system, unable to adapt to business changes in a timely manner, and reducing the flexibility and efficiency of content operation;

[0007] 4. In the traditional method, tags are encoded in the one - hot form. This encoding method simply indicates the presence or absence of tags, without fully exploring and utilizing the semantic information of tags, resulting in a waste of information resources, restricting the model's understanding and grasp of the semantic relationships of tags, and thus affecting the accuracy and effectiveness of classification.

[0008] As can be seen from the above, how to shorten the model training cycle, improve the development efficiency and business iteration speed, enhance the model's classification ability for medium and tail tags, improve the accuracy and comprehensiveness of the model in tag classification, and improve the flexibility and efficiency of model training are problems to be solved in this field. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to provide a tag classification method, device, equipment and medium based on a large model, which can shorten the model training cycle, improve the development efficiency and business iteration speed, enhance the model's classification ability for medium and tail tags, improve the accuracy and comprehensiveness of the model in tag classification, and improve the flexibility and efficiency of model training. The specific solutions are as follows:

[0010] In the first aspect, the present application discloses a tag classification method based on a large model, including:

[0011] Construct an initial semantic retrieval model and an initial artificial intelligence large model;

[0012] Perform model fine - tuning, pre - training and model optimization on the initial semantic retrieval model and the initial artificial intelligence large model on a preset data set to obtain a semantic retrieval model and an artificial intelligence large model;

[0013] Obtain the text to be classified, and input the text to be classified into the semantic retrieval model, so that the semantic retrieval model screens out the initial tags corresponding to the text to be classified from a preset tag library;

[0014] Input the text to be classified and the initial tags into the artificial intelligence large model, so that the artificial intelligence large model screens out the target tags corresponding to the text to be classified from the initial tags to complete the tag classification of the text to be classified.

[0015] Optionally, performing model fine - tuning and pre - training on the initial semantic retrieval model on a preset data set includes:

[0016] Perform model fine - tuning on the initial semantic retrieval model on a preset data set using the hard sample mining method to obtain the fine - tuned initial semantic retrieval model;

[0017] Use the dense retrieval pre - training strategy and perform masking and pre - training on the fine - tuned initial semantic retrieval model based on a full - scale encoder and a single - layer decoder.

[0018] Optionally, performing model fine-tuning and pre-training on the initial artificial intelligence large model on a preset data set, including:

[0019] Performing model fine-tuning on the initial artificial intelligence large model on a preset data set to obtain the fine-tuned initial artificial intelligence large model;

[0020] Inputting the historical data set in the form of a prompt into the fine-tuned initial artificial intelligence large model and performing pre-training on the fine-tuned initial artificial intelligence large model.

[0021] Optionally, performing model optimization on the initial semantic retrieval model and the initial artificial intelligence large model after fine-tuning and pre-training, including:

[0022] Constructing a loss function based on the historical data set, optimizing the loss function to obtain the optimized loss function;

[0023] Adding corresponding explanatory information to each label in the initial label database to obtain a label database; the explanatory information includes relevant entity information, English name, founder, and main products corresponding to the label;

[0024] Using the optimized loss function and the label database to perform model optimization on the initial semantic retrieval model and the initial artificial intelligence large model after fine-tuning and pre-training.

[0025] Optionally, inputting the text to be classified into the semantic retrieval model so that the semantic retrieval model filters out the initial labels corresponding to the text to be classified from a preset label library, including:

[0026] Determining the number of initial label filters;

[0027] Inputting the text to be classified into the semantic retrieval model so that the semantic retrieval model calculates the similarity between the text to be classified and each label in the preset label library, and based on the number of initial label filters, filters out the corresponding number of initial labels from each label in the order of similarity from large to small.

[0028] Optionally, inputting the text to be classified and the initial labels into the artificial intelligence large model so that the artificial intelligence large model filters out the target labels corresponding to the text to be classified from the initial labels, including:

[0029] Input the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model calculates the probabilities of the initial labels based on the text to be classified, obtains the predicted probabilities, and filters out the target labels corresponding to the text to be classified from the initial labels according to the predicted probabilities.

[0030] Optionally, the step of inputting the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model calculates the probabilities of the initial labels based on the text to be classified and obtains the predicted probabilities, includes:

[0031] Input the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model calculates the probabilities of each character in the initial labels based on the text to be classified, obtains the character probabilities, calculates the average value of the character probabilities, and uses the average value as the predicted probability of the initial label.

[0032] In a second aspect, the present application discloses a label classification device based on a large model, including:

[0033] A model construction module, configured to construct an initial semantic retrieval model and an initial large artificial intelligence model;

[0034] A model fine-tuning, pre-training, and optimization module, configured to perform model fine-tuning, pre-training, and model optimization on the initial semantic retrieval model and the initial large artificial intelligence model on a preset data set to obtain a semantic retrieval model and a large artificial intelligence model;

[0035] An initial label screening module, configured to obtain the text to be classified, input the text to be classified into the semantic retrieval model, so that the semantic retrieval model screens out the initial labels corresponding to the text to be classified from a preset label library;

[0036] A label classification module, configured to input the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model screens out the target labels corresponding to the text to be classified from the initial labels to complete the label classification of the text to be classified.

[0037] In a third aspect, the present application discloses an electronic device, including:

[0038] A memory, configured to store a computer program;

[0039] A processor, configured to execute the computer program to implement the foregoing label classification method based on a large model.

[0040] Fourthly, the present application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the foregoing disclosed label classification method based on a large model are implemented.

[0041] It can be seen that the present application provides a label classification method based on a large model, including constructing an initial semantic retrieval model and an initial artificial intelligence large model; performing model fine-tuning, pre-training, and model optimization on the initial semantic retrieval model and the initial artificial intelligence large model on a preset data set to obtain a semantic retrieval model and an artificial intelligence large model; obtaining a text to be classified, and inputting the text to be classified into the semantic retrieval model so that the semantic retrieval model screens out initial labels corresponding to the text to be classified from a preset label library; inputting the text to be classified and the initial labels into the artificial intelligence large model so that the artificial intelligence large model screens out target labels corresponding to the text to be classified from the initial labels to complete the label classification of the text to be classified. The present application constructs an initial semantic retrieval model and an initial artificial intelligence large model, performs model fine-tuning, pre-training, and model optimization on the initial semantic retrieval model and the initial artificial intelligence large model on a preset data set. Through an innovative model architecture and training method, it is possible to reduce the sample annotation amount and the number of model training rounds, enable the model to complete training in a short time, quickly adapt to business requirements, improve development efficiency and business iteration speed, enhance the model's classification ability for medium and tail labels, enable the model to have a high accuracy rate when facing various labels, ensure the comprehensiveness and accuracy of content classification, more precisely depict the text content, input the text to be classified into the semantic retrieval model so that the semantic retrieval model screens out initial labels corresponding to the text to be classified from a preset label library, input the text to be classified and the initial labels into the artificial intelligence large model so that the artificial intelligence large model screens out target labels corresponding to the text to be classified from the initial labels. The present application can quickly adapt to new labels, shorten the iteration cycle of the label system, improve the flexibility and efficiency of content operation, improve the accuracy and comprehensiveness of the model during label classification, and improve the flexibility and efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0043] Figure 1 It is a flowchart of a label classification method based on a large model disclosed in the present application;

[0044] Figure 2 Flowchart of a RetroMAE training framework disclosed in this application;

[0045] Figure 3 Flowchart of a mainstream label classification method disclosed in this application;

[0046] Figure 4 Flowchart of multi-label classification of a semantic retrieval model and a large artificial intelligence model disclosed in this application;

[0047] Figure 5 Specific flowchart of a method for label classification based on a large model disclosed in this application;

[0048] Figure 6 Schematic diagram of the structure of a label classification device based on a large model disclosed in this application;

[0049] Figure 7 Structural diagram of an electronic device provided by this application. Detailed implementation manners

[0050] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0051] Currently, text label classification is a key technology and is widely used in many scenarios, such as news classification, product annotation, content recommendation, etc. Currently, the mainstream text multi-label classification methods usually adopt pre-trained language models and perform parameter fine-tuning on specific data sets. This method performs well in scenarios with a small number of labels and a fixed label system, and has the advantages of relatively simple modeling methods, controllable effects and costs.

[0052] However, with the explosive growth of Internet content, in many practical applications, it is necessary to characterize content in multiple granularities and dimensions, and the number of tags often reaches the order of hundreds of thousands or even millions. When facing a massive tag system, traditional methods expose many problems: 1. Long training cycle: Due to the large number of tags, to ensure the learning effect of the model for each tag, a large number of samples need to be labeled for each tag, which makes the training sample size increase sharply. In the process of fitting massive samples to massive tags, the model often needs to be trained for at least dozens of rounds to converge, resulting in a significant extension of the sample annotation and model training cycle, seriously affecting the development efficiency and business iteration speed. For example, in a large content platform, for multi-label classification of an article, if it involves tens of thousands of tags, training in the traditional way may take weeks or even months from sample annotation to model training completion; 2. Poor generalization performance of the model: For head tags (i.e., tags with a large number of labeled samples), the model has a good fitting effect and high accuracy; but for middle and tail tags (i.e., tags with fewer labeled samples or no labeled samples), the model accuracy is low and it is difficult to improve. This is because the labeled samples of middle and tail tags are scarce, and the model is difficult to learn enough feature information, resulting in poor classification effects for these tags in actual applications, unable to accurately identify relevant content, and affecting the comprehensiveness and accuracy of content classification; 3. Long iteration cycle of the tag system: With the continuous development and change of the business, the tag category system also needs to be updated accordingly. In the traditional method, for each new batch of tags, a large number of samples need to be re-labeled and the model needs to be re-trained. Due to the overall large scale of the labeled samples, the newly added samples are difficult to ensure good performance of the model on the new tags, resulting in slow iteration of the tag system, unable to adapt to business changes in a timely manner, and reducing the flexibility and efficiency of content operation; 4. The traditional method encodes tags in the form of one-hot. This encoding method only simply represents the presence or absence of tags, without fully exploring and utilizing the semantic information of tags, resulting in a waste of information resources, restricting the model's understanding and grasp of tag semantic relationships, and thus affecting the accuracy and effect of classification. As can be seen from the above, how to shorten the model training cycle, improve the development efficiency and business iteration speed, enhance the model's classification ability for middle and tail tags, improve the accuracy and comprehensiveness of the model in tag classification, and improve the flexibility and efficiency of model training are problems to be solved in this field.

[0053] See Figure 1 As shown, an embodiment of the present invention discloses a tag classification method based on a large model, which may specifically include:

[0054] Step S11: Construct an initial semantic retrieval model and an initial artificial intelligence large model.

[0055] Step S12: Perform model fine-tuning, pre-training, and model optimization on the initial semantic retrieval model and the initial large artificial intelligence model on a preset dataset to obtain a semantic retrieval model and a large artificial intelligence model.

[0056] In this embodiment, the process of model fine-tuning and pre-training on the initial semantic retrieval model is as follows: Fine-tune the initial semantic retrieval model on a preset dataset using the hard sample mining method to obtain the fine-tuned initial semantic retrieval model; Use the dense retrieval pre-training strategy and perform masking and pre-training on the fine-tuned initial semantic retrieval model based on a full-scale encoder and a single-layer decoder.

[0057] The semantic retrieval model in this application includes, but is not limited to, BGE (BAAI General Embedding, an open-source text vector model), and is fine-tuned on a preset dataset, which includes, but is not limited to, the Zhihu dataset. The BGE model performs excellently in semantic vector representation in the field of natural language processing, and RetroMAE (dense retrieval pre-training strategy) adopted in its pre-training stage is a key factor in improving performance.

[0058] In the design of RetroMAE, a unique asymmetric model structure and masking ratio are introduced. In the Encoder stage, a full-scale encoder is used, which can fully extract rich features of the input text and comprehensively and deeply understand the text semantics. While in the Decoder stage, a single-layer decoder is adopted, and this asymmetric structure breaks the pattern of traditional symmetric architectures. At the same time, moderate masking is performed in the Encoder stage, and the masking rate is controlled at about 15%-30%. On the basis of retaining the key information of the text, the model learns how to recover the complete semantics from partial information; radical masking is performed in the Decoder stage, and the masking rate is as high as 50%-70%, which makes the sentence reconstruction process extremely challenging. Under this design, the semantic vector output by the encoding side can more accurately represent the sentence itself, because when the model deals with high-difficulty reconstruction tasks, it will pay more attention to extracting key semantic features, thus significantly improving the performance of the pre-trained model in dense retrieval tasks.

[0059] In this embodiment, the RetroMAE training framework process is as Figure 2 shown. The training framework of RetroMAE in the pre-training stage of the BGE model includes an asymmetric model structure (a full-scale encoder in the Encoder stage and a single-layer decoder in the Decoder stage) and an asymmetric masking ratio (moderate masking in the Encoder stage and radical masking in the Decoder stage).

[0060] In the fine-tuning stage, the BGE model adopts the hard-mining method. In the traditional in-batch negative sampling method, only different samples within the current batch are selected as negative samples. The negative samples mined in this way are relatively simple, and it is difficult for the model to learn complex semantic discrimination information from them. The hard-mining method, on the other hand, searches for samples that are literally similar but semantically different from the training samples as negative samples through semantic screening. For example, for a sample describing "apple (fruit)", traditional negative sampling might select a completely unrelated sample, such as "car", as a negative sample; while hard-mining might find a sample like "apple (tech company)" that is literally similar but semantically different as a negative sample. In this way, the model can be exposed to more challenging negative samples, learn finer semantic differences, and thus more effectively distinguish semantics.

[0061] In this embodiment, the process of model fine-tuning and pre-training for the initial large artificial intelligence model is as follows: The initial large artificial intelligence model is fine-tuned on a preset data set to obtain the fine-tuned initial large artificial intelligence model; the historical data set is input into the fine-tuned initial large artificial intelligence model in the form of a prompt, and the fine-tuned initial large artificial intelligence model is pre-trained.

[0062] Taking the Zhihu self-developed Zhihai Map AI (Artificial Intelligence) large model as an example of the initial large artificial intelligence model, the historical data set is input into the fine-tuned initial large artificial intelligence model in the form of a prompt. The design of the prompt is the key to guiding the large model to make accurate selections. It is like a detailed instruction given to the large model, telling the large model which aspects to focus on and how to make the correct choice from the candidate labels.

[0063] After receiving the input, the large model will, based on its powerful semantic understanding and generation capabilities, select the label that matches the input text from the candidate labels. During the output process of the large model, it will calculate the prediction probability of each token (character) in the label. For example, for the label "game streamer", the large model will calculate the prediction probabilities of the two tokens "game" and "streamer" respectively. Then, the average value of the probabilities of all tokens for each label is taken as the probability of that label to determine the final predicted label.

[0064] This data construction method exhibits great advantages, especially when dealing with changes in the label system. In actual business scenarios, the label system is not static, and new labels are often added or existing labels are adjusted. When there are minor changes in the label system, such as adding a new sub-category "Game Streamer - League of Legends Streamer" label, based on this data construction method, there is no need to retrain the model, and zero-shot generation can be performed on the newly added label. This means that the model can directly identify and process new labels based on existing knowledge and the input prompt, greatly improving the flexibility and efficiency of label system updates.

[0065] In this embodiment, the process of optimizing the initial semantic retrieval model and the initial artificial intelligence large model is as follows: constructing a loss function based on the historical data set, optimizing the loss function to obtain the optimized loss function; adding corresponding explanatory information to each label in the initial label database to obtain a label database; the explanatory information includes relevant entity information, English name, founder, and main products corresponding to the label; using the optimized loss function and the label database to optimize the fine-tuned and pre-trained initial semantic retrieval model and the initial artificial intelligence large model.

[0066] This application uses the Zhihu high-quality corpus to fine-tune it. During the fine-tuning process, the method of negative sampling within a batch is adopted to optimize the loss between the input text and the label. This method selects different samples as negative samples within each batch, enabling the model to learn how to distinguish positive samples (matching text-label pairs) from negative samples (non-matching text-label pairs).

[0067] However, in actual operation, when the data is not fully shuffled or the number of model labels is not large enough, the situation of false negative examples within a batch will occur. Suppose there are 10 samples in a batch, and both sample i and sample j should be labeled with the label "Tech News". Due to data arrangement or other reasons, when calculating the loss, the model will take sample j as the negative sample of sample i, and vice versa. This causes confusion in the learning process of the model because it mistakenly treats two correctly matched samples as non-matching samples for processing.

[0068] To address this issue, the present invention optimizes the loss function by masking the Loss Function. The specific implementation process is as follows: When calculating the loss, first construct a mask matrix with the same shape as the sample matrix. This mask matrix is like a "filter" used to adjust the loss calculation process. Set the corresponding positions of sample i and sample j (i.e., the i, j position and the j, i position) to 0, which means that when calculating the loss between these two samples, the situation of regarding them as negative samples for each other is ignored. Then, multiply the loss matrix by the mask matrix to obtain the final masked loss matrix for calculating the loss.

[0069] To further optimize the accuracy of the initial label screening of the semantic retrieval model, the present invention adds explanatory information to the labels. In actual text classification tasks, many labels have semantic ambiguities. For example, the label "apple" can refer to both the fruit and Apple Inc. Such semantic ambiguities can lead to inaccurate situations when the model conducts initial label screening.

[0070] The present invention solves this problem by adding rich explanatory information to the labels. Taking the "apple" label as an example, enrich its semantic information to (Apple Inc., English name: Apple Inc., founder Steve Jobs, and the most famous hardware products include iPhone smartphones, iPad tablets, Mac personal computers, Apple Watch smartwatches, etc.). When the model conducts retrieval, these detailed explanatory information can help the model better understand the meaning of the label, thereby more accurately completing the alignment of the query and the label.

[0071] This method not only solves problems such as inaccurate screening caused by semantic ambiguities but also improves the model's ability to understand and utilize label semantic information. The model can learn more context knowledge about the label from these rich explanatory information, thereby more accurately finding the label that matches the input text in the massive label library.

[0072] The process of the mainstream label classification method in this application is as Figure 3 shown. Usually, a pre-trained language model is adopted, and then parameter fine-tuning is performed on a specific dataset. The model outputs a K-dimensional vector, and each element represents the probability that the sample belongs to the label. The difference between the predicted label and the true label in the historical dataset is calculated through a multi-classification loss function. After being processed by the fully connected layer and the sigmoid activation function, the threshold is set to select the label to complete multi-label classification.

[0073] Step S13: Obtain the text to be classified, and input the text to be classified into the semantic retrieval model so that the semantic retrieval model can screen out the initial label corresponding to the text to be classified from the preset label library.

[0074] In this embodiment, the text to be classified is obtained, and the initial label screening quantity is determined; the text to be classified is input into the semantic retrieval model, so that the semantic retrieval model calculates the similarity between the text to be classified and each label in the preset label library, and based on the initial label screening quantity, corresponding quantities of initial labels are screened out from each label in the descending order of similarity.

[0075] In addition, after fine-tuning, the label will pass through the label encoder to obtain the embedding corresponding to the label at the cls (central login service) position. This process is like generating a unique "digital fingerprint" for each label to accurately represent its semantic features. Then, the embeddings of all labels are calculated and indexed offline. During inference, the query encoder encodes the input query to obtain the embedding of the query. By calculating the similarity between the query and each label, the K labels most similar to the query can be screened. In practical applications, the selection of the K value is crucial. If the K value is too small, it may result in insufficient screened labels and miss some potential correct labels; if the K value is too large, although the recall rate will increase, it will bring greater pressure to the subsequent large model screening, resulting in a decrease in the accuracy of the correct labels generated by the large model. After a large number of experimental verifications, when the K value is selected to be around 10, it can not only ensure a high label screening rate to meet the business expectations, but also enable the large model to have a good accuracy in generating correct labels.

[0076] Step S14: Input the text to be classified and the initial labels into the artificial intelligence large model, so that the artificial intelligence large model screens out the target labels corresponding to the text to be classified from the initial labels to complete the label classification of the text to be classified.

[0077] In this embodiment, the text to be classified and the initial labels are input into the artificial intelligence large model, so that the artificial intelligence large model calculates the probability of each character in the initial labels based on the text to be classified to obtain the character probability, calculates the average value of the character probability, and uses the average value as the prediction probability of the initial labels, and screens out the target labels corresponding to the text to be classified from the initial labels according to the prediction probability.

[0078] This application forms an efficient, massively labeled, multi-classification system, with a tightly coordinated workflow across the entire system. First, the text to be classified is input into a carefully fine-tuned semantic retrieval model. This process is like "inputting a key" into the model. With its powerful retrieval capabilities, the model quickly finds the "key" with a high degree of matching in the vast "key library" of a massive library of preset tags, thereby filtering out the initial set of tags related to the text to be classified.

[0079] The text to be classified and the initial set of labels are then used as input for the AI model. These serve as "clues" and "options" for the AI model, allowing it to be fine-tuned through Prompt Engineering. Prompt Engineering plays a key guiding role here, designing the most appropriate prompts based on specific business needs and text characteristics, enabling the model to better understand the task and input information. Based on these inputs and fine-tuning, the model selects target labels from the candidate label set, completing the multi-label classification task for the massive amount of text to be classified. This seamless process efficiently and accurately achieves the goal of transitioning from text to multi-label classification.

[0080] In this embodiment, the semantic retrieval model and the multi-label classification process of the artificial intelligence large model are as follows: Figure 4 As shown, the text to be classified is first input into the semantic retrieval model to filter out the initial labels, and then the text to be classified and the initial labels are input into the artificial intelligence big model, and the target labels are output after being filtered by the artificial intelligence big model.

[0081] The specific process of implementing the label classification method based on the large model in this application is as follows Figure 5 As shown, the specific process is as follows:

[0082] The first step is to transform the classification problem into a retrieval problem. Using a pre-trained semantic retrieval model, fine-tune it on a labeled dataset. This fine-tuning process is a meticulous optimization process, like "customizing" the model. By learning from labeled data, the model can better understand the semantic relationship between text and labels, achieve alignment between text and labels, and lay a solid foundation for subsequent screening of initial labels.

[0083] The second step is to use the semantic retrieval model to filter out initial labels from the text to be classified. Initial labels corresponding to the text to be classified are filtered out from the massive number of labels to form an initial label set. In this process, the model analyzes the input text based on the semantic relationships learned previously, and then quickly searches the massive label library to filter out labels with a high degree of semantic match with the text. These labels together constitute the initial labels.

[0084] Step 3: Adopt Prompt technology and use the text to be classified as the input to the large artificial intelligence model. The design of Prompt needs to be carefully adjusted according to specific business scenarios and text characteristics to guide the large model to better understand the task. Based on these inputs and fine-tuning, the large model screens out the correct target labels from the initial labels to complete the multi-classification task of massive labels for the text. In this process, the large model gives full play to its powerful semantic understanding and generation capabilities, deeply analyzes and screens the initial labels, and finally outputs accurate multi-label classification results.

[0085] Compared with the prior art, the present application has the following remarkable beneficial effects:

[0086] The training cycle is significantly shortened: By adopting the architecture combining a semantic retrieval model and a large artificial intelligence model, as well as techniques such as difficult sample mining and loss function optimization, the dependence on a large number of samples is reduced, and the number of model training rounds is decreased. Experimental data shows that when processing the same amount of massive labels, compared with traditional methods, the training time of the present invention is shortened, greatly improving the development efficiency and business iteration speed.

[0087] The model generalization performance is significantly improved: The optimization of the semantic retrieval model and the powerful semantic understanding ability of the large artificial intelligence model significantly improve the classification accuracy of the model for medium and tail labels. In practical applications, the classification accuracy for medium and tail labels is significantly improved compared with traditional methods, effectively improving the comprehensiveness and accuracy of content classification, and being able to more accurately depict the text content.

[0088] The label system iteration is efficient and flexible: The model structure and data construction method of the present invention have good compatibility with changes in the label system. When there are minor changes in the label system, there is no need to retrain the model. Only the embedding of the new labels needs to be calculated and updated to the index library; when there is a major version update, only a small amount of labeled data is required to make the model adapt to the new labels, greatly shortening the label system iteration cycle. According to statistics, the label system iteration cycle is significantly shortened compared with traditional methods, improving the flexibility and efficiency of content operation.

[0089] Making full use of the semantic information of tags, the classification accuracy is significantly improved: By adding explanatory information to the tags and improving the model structure, the semantic information of the tags is deeply mined and utilized. The experimental results on the Zhihu self-annotated dataset show that for the short text (Zhihu questions) dataset of the present invention, the accuracy rate is increased by more than 15% (~70% -> 85%+), and the coverage rate is increased by about 15% (~80% -> 95%+); for the long text (Zhihu answers and articles), the accuracy rate is increased by more than 20% (~65% -> 85%+), and the coverage rate is increased by about 20% (~75% -> 95%+). The model can more accurately understand the semantic relationship between the text and the tags, realize a more accurate and comprehensive characterization of the content, and provide strong support for the accurate distribution of the content.

[0090] The annotation cost is significantly reduced: Since the amount of sample annotation is reduced, especially the demand for new tag annotation data is greatly reduced when the tag system is updated, the annotation cost is significantly reduced. After accounting, the annotation cost is significantly reduced compared with the traditional method, effectively saving labor and time costs and improving economic benefits.

[0091] In this embodiment, an initial semantic retrieval model and an initial large artificial intelligence model are constructed; the initial semantic retrieval model and the initial large artificial intelligence model are subjected to model fine-tuning, pre-training, and model optimization on a preset data set to obtain a semantic retrieval model and a large artificial intelligence model; a text to be classified is obtained, and the text to be classified is input into the semantic retrieval model so that the semantic retrieval model screens out initial labels corresponding to the text to be classified from a preset label library; the text to be classified and the initial labels are input into the large artificial intelligence model so that the large artificial intelligence model screens out target labels corresponding to the text to be classified from the initial labels to complete the label classification of the text to be classified. This application constructs an initial semantic retrieval model and an initial large artificial intelligence model, and performs model fine-tuning, pre-training, and model optimization on the initial semantic retrieval model and the initial large artificial intelligence model on a preset data set. Through an innovative model architecture and training method, it is possible to reduce the sample annotation amount and the number of model training rounds, enable the model to complete training in a shorter time, quickly adapt to business requirements, improve development efficiency and business iteration speed, enhance the model's classification ability for medium and tail labels, enable the model to have a high accuracy rate when facing various labels, ensure the comprehensiveness and accuracy of content classification, more accurately depict the text content, input the text to be classified into the semantic retrieval model so that the semantic retrieval model screens out initial labels corresponding to the text to be classified from a preset label library, input the text to be classified and the initial labels into the large artificial intelligence model so that the large artificial intelligence model screens out target labels corresponding to the text to be classified from the initial labels. This application can quickly adapt to new labels, shorten the iteration cycle of the label system, improve the flexibility and efficiency of content operation, improve the accuracy and comprehensiveness of the model when performing label classification, and improve the flexibility and efficiency of model training.

[0092] See Figure 6 As shown, an embodiment of the present invention discloses a label classification device based on a large model, which may specifically include:

[0093] A model construction module 11, configured to construct an initial semantic retrieval model and an initial large artificial intelligence model;

[0094] A model fine-tuning, pre-training, and optimization module 12, configured to perform model fine-tuning, pre-training, and model optimization on the initial semantic retrieval model and the initial large artificial intelligence model on a preset data set to obtain a semantic retrieval model and a large artificial intelligence model;

[0095] An initial label screening module 13, configured to obtain a text to be classified, and input the text to be classified into the semantic retrieval model so that the semantic retrieval model screens out initial labels corresponding to the text to be classified from a preset label library;

[0096] The label classification module 14 is used to input the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model can screen out the target labels corresponding to the text to be classified from the initial labels, so as to complete the label classification of the text to be classified.

[0097] In this embodiment, an initial semantic retrieval model and an initial large artificial intelligence model are constructed; the initial semantic retrieval model and the initial large artificial intelligence model are subjected to model fine-tuning, pre-training, and model optimization on a preset data set to obtain a semantic retrieval model and a large artificial intelligence model; the text to be classified is obtained, and the text to be classified is input into the semantic retrieval model, so that the semantic retrieval model can screen out the initial labels corresponding to the text to be classified from a preset label library; the text to be classified and the initial labels are input into the large artificial intelligence model, so that the large artificial intelligence model can screen out the target labels corresponding to the text to be classified from the initial labels, so as to complete the label classification of the text to be classified. In this application, an initial semantic retrieval model and an initial large artificial intelligence model are constructed, and the initial semantic retrieval model and the initial large artificial intelligence model are subjected to model fine-tuning, pre-training, and model optimization on a preset data set. Through an innovative model architecture and training method, the amount of sample annotation and the number of model training rounds can be reduced, so that the model can complete training in a short time, quickly adapt to business requirements, improve development efficiency and business iteration speed, enhance the classification ability of the model for medium and tail labels, enable the model to have a high accuracy rate when facing various labels, ensure the comprehensiveness and accuracy of content classification, more accurately depict the text content, input the text to be classified into the semantic retrieval model, so that the semantic retrieval model can screen out the initial labels corresponding to the text to be classified from a preset label library, input the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model can screen out the target labels corresponding to the text to be classified from the initial labels. This application can quickly adapt to new labels, shorten the iteration cycle of the label system, improve the flexibility and efficiency of content operation, improve the accuracy and comprehensiveness of the model when performing label classification, and improve the flexibility and efficiency of model training.

[0098] In some specific embodiments, the model fine-tuning, pre-training, and optimization module 12 may specifically include:

[0099] The initial semantic retrieval model fine-tuning module is used to fine-tune the initial semantic retrieval model on a preset data set by using the hard sample mining method to obtain the fine-tuned initial semantic retrieval model;

[0100] The initial semantic retrieval model masking and pre-training module is used to perform masking and pre-training on the fine-tuned initial semantic retrieval model by using the dense retrieval pre-training strategy and based on a full-scale encoder and a single-layer decoder.

[0101] In some specific embodiments, the model fine-tuning, pre-training and optimization module 12 may specifically include:

[0102] An initial artificial intelligence large model fine-tuning module, configured to fine-tune the initial artificial intelligence large model on a preset data set to obtain the fine-tuned initial artificial intelligence large model;

[0103] An initial artificial intelligence large model pre-training module, configured to input a historical data set into the fine-tuned initial artificial intelligence large model in the form of a prompt and pre-train the fine-tuned initial artificial intelligence large model.

[0104] In some specific embodiments, the model fine-tuning, pre-training and optimization module 12 may specifically include:

[0105] A loss function construction and optimization module, configured to construct a loss function based on a historical data set, optimize the loss function, and obtain the optimized loss function;

[0106] An explanation information adding module, configured to add corresponding explanation information to each label in the initial label database to obtain a label database; the explanation information includes relevant entity information, English name, founder, and main product corresponding to the label;

[0107] A model optimization module, configured to optimize the fine-tuned and pre-trained initial semantic retrieval model and the initial artificial intelligence large model by using the optimized loss function and the label database.

[0108] In some specific embodiments, the initial label screening module 13 may specifically include:

[0109] A quantity determination module, configured to determine the initial label screening quantity;

[0110] An initial label selection module, configured to input the text to be classified into the semantic retrieval model, so that the semantic retrieval model calculates the similarity between the text to be classified and each label in the preset label library, and based on the initial label screening quantity, selects a corresponding number of initial labels from each label in the order of similarity from large to small.

[0111] In some specific embodiments, the label classification module 14 may specifically include:

[0112] A target label screening module, configured to input the text to be classified and the initial labels into the artificial intelligence large model, so that the artificial intelligence large model calculates the probabilities of the initial labels based on the text to be classified, obtains predicted probabilities, and screens out target labels corresponding to the text to be classified from the initial labels according to the predicted probabilities.

[0113] In some specific embodiments, the label classification module 14 may specifically include:

[0114] A probability calculation module, configured to input the text to be classified and the initial labels into the artificial intelligence large model, so that the artificial intelligence large model calculates the probabilities of each character in the initial labels based on the text to be classified, obtains character probabilities, calculates the average value of the character probabilities, and uses the average value as the predicted probability of the initial labels.

[0115] Figure 7 The figure is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the large model-based label classification method executed by the electronic device disclosed in any of the foregoing embodiments.

[0116] In this embodiment, the power supply 23 is used to provide operating voltages for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is made thereto here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application requirements, and no specific limitation is made here.

[0117] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223, etc., and the storage method may be short-term storage or permanent storage.

[0118] Among them, the operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the data 223 in the memory 22 by the processor 21. It can be Windows, Unix, Linux, etc. In addition to the computer program that can be used to complete the large model-based label classification method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program that can be used to complete other specific tasks. In addition to the data that can include the data transmitted by the external device received by the large model-based label classification device, the data 223 may also include the data collected by its own input / output interface 25, etc.

[0119] The steps of the methods or algorithms described in combination with the embodiments disclosed in this article can be implemented directly by hardware, software modules executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well known in the technical field.

[0120] Furthermore, the embodiments of the present application also disclose a computer-readable storage medium. When the computer program stored in the storage medium is loaded and executed by a processor, the steps of the large model-based label classification method disclosed in any of the foregoing embodiments are implemented.

[0121] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0122] The above has introduced in detail a method, apparatus, device, and storage medium for label classification based on a large model. In this article, specific examples are used to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A label classification method based on a large model, characterized in that Including: Construct an initial semantic retrieval model and an initial large artificial intelligence model; Perform model fine-tuning, pre-training, and model optimization on the initial semantic retrieval model and the initial large artificial intelligence model on a preset data set to obtain a semantic retrieval model and a large artificial intelligence model; Obtain the text to be classified, and input the text to be classified into the semantic retrieval model, so that the semantic retrieval model filters out the initial labels corresponding to the text to be classified from a preset label library; Input the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model filters out the target labels corresponding to the text to be classified from the initial labels to complete the label classification of the text to be classified.

2. The method for label classification based on a large model according to claim 1, wherein Performing model fine-tuning and pre-training on the initial semantic retrieval model on a preset data set includes: Performing model fine-tuning on the initial semantic retrieval model on a preset data set by using a hard sample mining method to obtain the fine-tuned initial semantic retrieval model; Using a dense retrieval pre-training strategy and based on a full-scale encoder and a single-layer decoder, performing masking and pre-training on the fine-tuned initial semantic retrieval model.

3. The method for label classification based on a large model according to claim 1, wherein, Performing model fine-tuning and pre-training on the initial large artificial intelligence model on a preset data set includes: Performing model fine-tuning on the initial large artificial intelligence model on a preset data set to obtain the fine-tuned initial large artificial intelligence model; Inputting the historical data set into the fine-tuned initial large artificial intelligence model in the form of a prompt, and performing pre-training on the fine-tuned initial large artificial intelligence model.

4. The label classification method based on a large model according to claim 1, wherein Performing model optimization on the fine-tuned and pre-trained initial semantic retrieval model and the initial large artificial intelligence model includes: Constructing a loss function based on the historical data set, optimizing the loss function to obtain the optimized loss function; Adding corresponding explanatory information to each label in the initial label database to obtain a label database; the explanatory information includes relevant entity information, English name, founder, and main products corresponding to the label; Using the optimized loss function and the label database to perform model optimization on the fine-tuned and pre-trained initial semantic retrieval model and the initial large artificial intelligence model.

5. The label classification method based on a large model according to claim 1, characterized in that The step of inputting the text to be classified into the semantic retrieval model so that the semantic retrieval model filters out the initial labels corresponding to the text to be classified from a preset label library includes: Determining the number of initial label filters; Inputting the text to be classified into the semantic retrieval model, so that the semantic retrieval model calculates the similarity between the text to be classified and each label in the preset label library, and based on the number of initial label filters, filters out the corresponding number of initial labels from each label in the order of similarity from large to small.

6. The large model-based label classification method according to any one of claims 1 to 5, characterized in that The step of inputting the text to be classified and the initial labels into the large artificial intelligence model so that the large artificial intelligence model filters out the target labels corresponding to the text to be classified from the initial labels includes: Input the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model calculates the probabilities of the initial labels based on the text to be classified, obtains the predicted probabilities, and filters out the target labels corresponding to the text to be classified from the initial labels according to the predicted probabilities.

7. The method for label classification based on a large model according to claim 6, wherein, The step of inputting the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model calculates the probabilities of the initial labels based on the text to be classified, obtains the predicted probabilities, includes: Input the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model calculates the probabilities of each character in the initial labels based on the text to be classified, obtains the character probabilities, calculates the average value of the character probabilities, and uses the average value as the predicted probability of the initial label.

8. A label classification device based on a large model, characterized in that, It includes: A model construction module for constructing an initial semantic retrieval model and an initial large artificial intelligence model; A model fine-tuning, pre-training and optimization module for performing model fine-tuning, pre-training and model optimization on the initial semantic retrieval model and the initial large artificial intelligence model on a preset data set to obtain a semantic retrieval model and a large artificial intelligence model; An initial label screening module for obtaining the text to be classified and inputting the text to be classified into the semantic retrieval model, so that the semantic retrieval model screens out the initial labels corresponding to the text to be classified from a preset label library; A label classification module for inputting the text to be classified and the initial labels into the large artificial intelligence model, so that the large artificial intelligence model screens out the target labels corresponding to the text to be classified from the initial labels to complete the label classification of the text to be classified.

9. An electronic device, characterized in that, It includes: A memory for storing computer programs; A processor for executing the computer programs to implement the large model-based label classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For storing computer programs; wherein, when the computer programs are executed by the processor, the large model-based label classification method according to any one of claims 1 to 7 is implemented.

Images