A Domain-Specific Text Classification and Labeling Method and System Based on Large Language Model

Abstract

The application relates to a specific field text classification labeling method and system based on a large language model, and is applied to the technical field of the large language model, which comprises the following steps: designing a calling interface for a large language model based on data labeling requirements to be classified; setting an automatic labeling process of the large language model based on the calling interface; searching a preset initial prompt based on a genetic algorithm to obtain an extended prompt; and controlling the large language model to automatically classify and label texts based on the extended prompt and the automatic labeling process. The application designs a genetic algorithm suitable for the large language model, generates more extended prompts through the initial prompt, searches for better prompts, improves the accuracy of text labeling of the large language model, and reduces the text quantity requirement of a special data set compared with a traditional labeling method, greatly reduces the required labeling text quantity of experts, facilitates the acquisition of text data, and improves the convenience of special field text labeling.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a domain-specific text classification and annotation method and system based on large language models. Background Technology

[0002] Text classification is a fundamental and widely applied task in the field of natural language processing. It involves dividing text data into specific categories based on the characteristics of different texts. For example, sentiment classification requires analyzing the implicit emotional information within the text to be tested and identifying the most likely emotion, such as joy, sadness, or anger.

[0003] For text classification tasks, training data is crucial. The quality and quantity of training data directly determine the model's accuracy and generalization ability. Since the semantic information and hidden features contained in text are not directly reflected in the words, a large amount of labeled data is needed for the text classification model to learn the high-level features contained in the text. However, obtaining labeled data is difficult, and the total amount of high-quality labeled data is limited. This is especially true for domain-specific text classification tasks, where no open-source labeled datasets are available. Labeling the entire dataset from scratch is required, and the categories to be classified can be extremely complex and numerous. Furthermore, the number of labelers is also limited; specific domains contain many proper nouns and complex texts, making it difficult for ordinary people to accurately classify such texts. Only a very small number of domain experts can label the data with high accuracy.

[0004] Due to the difficulties in labeling text classification tasks within the aforementioned specific domains, only a very small number of labeled samples can be obtained. Using these samples to train a text classification model results in a very poor performance when automatically labeling the remaining data. Data augmentation is a quick and easy solution to address the problem of insufficient training samples. Current mainstream text data augmentation methods modify the original text at the word and sentence levels to obtain semantically similar new text, assigning it the same classification label as the original text. Word-level methods include synonym replacement and word embedding space nearest neighbor replacement. Sentence-level methods include back-translation and word order rearrangement. However, these methods can only produce text that is semantically identical or similar to the labeled text. In the context of feature-domain text classification, experts can only label a small portion of the data. After data augmentation, these augmented texts are semantically identical to the labeled data, while the unlabeled data contains a large amount of text with a completely different semantic distribution, belonging to out-of-domain data. Even with data augmentation, the model cannot learn how to correctly classify texts from these out-of-domain data because it has never learned the features and classification methods of these texts.

[0005] Traditional data augmentation methods struggle to train highly accurate text classification and automatic annotation models. However, with the development of large language model technology, automatic text annotation tasks can be accomplished using large language models such as ChatGPT. These large language models possess rich knowledge reserves and natural language understanding capabilities, enabling them to directly annotate many text classification tasks with high accuracy.

[0006] Regarding the aforementioned technologies, it is argued that directly using large language models to annotate text in a specific domain results in low accuracy because there are no publicly available datasets for such text data, and complex texts and specialized vocabulary are not found in the training set of large language models. Summary of the Invention

[0007] To address the issue of low accuracy in labeling domain-specific texts when directly using large language models, where there are no publicly available datasets and complex texts and proper nouns are not present in the training sets of large language models, this application provides a domain-specific text classification and labeling method and system based on large language models.

[0008] Firstly, this application provides a domain-specific text classification and annotation method based on a large language model, employing the following technical solution: including:

[0009] Based on the requirements for labeling the data to be classified, a calling interface for text labeling was designed for a large language model;

[0010] The automatic annotation process for large language models is set up based on the aforementioned API call interface;

[0011] The genetic algorithm is used to search for the preset initial prompt and obtain the expanded prompt.

[0012] Based on the extended prompt and the automatic annotation process, the large language model is used for automatic text classification and annotation.

[0013] Optionally, the interface for text annotation designed for a large language model based on the requirements for labeling the data to be classified includes:

[0014] The calling interface is designed for the large language model through interactive dialogue, so that the large language model can autonomously select the calling interface needed to complete the text classification and labeling task based on the output text content. The result returned by the calling interface is also returned to the large language model in the form of dialogue.

[0015] The calling interface includes:

[0016] The precise query interface is used to precisely query the meaning of domain-specific terms.

[0017] Fuzzy query interface, used for fuzzy querying the meaning of domain-specific terms;

[0018] The vocabulary addition interface is used to add words and their corresponding meanings to the proprietary vocabulary list;

[0019] The same-table query interface is used to query other data and corresponding annotation results within the same table in the expert annotation dataset.

[0020] The field query interface is used to query the annotation results of text data with similar field names in the expert annotation dataset;

[0021] Unknown words interface: If the large language model cannot determine the meaning of a word, the word is added to the unknown meaning vocabulary list.

[0022] The annotation output interface provides the classification annotation results.

[0023] Optionally, the automatic annotation process includes:

[0024] Initialize the proprietary vocabulary list by adding all proprietary terms in the field to the list;

[0025] Try to provide the text classification and annotation results;

[0026] Determine if the text contains unfamiliar words;

[0027] Call the precise query interface; if no match is found, then call the fuzzy query interface.

[0028] Query other data in the same table and attempt to provide classification and labeling results;

[0029] Query similar data in the query fields and attempt to provide classification and labeling results;

[0030] Determine whether the meaning of an unfamiliar word can be determined, and add the word to the proprietary vocabulary or the unknown vocabulary based on the classification and labeling results.

[0031] Optionally, after setting up the automatic annotation process for the large language model based on the calling interface, the method further includes:

[0032] Upon receiving the unknown vocabulary list annotated by experts, the annotated unknown words are re-added to the proprietary vocabulary;

[0033] The data that failed to be classified are re-added to the data to be classified.

[0034] Optionally, the step of searching the preset initial prompt using a genetic algorithm to obtain the expanded prompt includes:

[0035] The expert-annotated dataset is divided into a search set and a validation set according to a preset ratio;

[0036] The search is performed on the lookup set using a large language model, and the data in the validation set is labeled.

[0037] The initial prompts without performance metrics are evaluated on the validation set to obtain the performance metrics, which are: the classification labeling accuracy on the validation set after the automatic labeling process is completed.

[0038] The initial prompt is inherited based on the genetic algorithm to generate several new prompts, and the effectiveness of the new prompts is evaluated to obtain the corresponding effectiveness index.

[0039] Repeat the above effect evaluation and genetic process until the preset number of iterations is reached;

[0040] The prompt with the highest performance index is selected as the prompt used in the formal automatic annotation process.

[0041] Optionally, the step of generating several new prompts by genetically processing the initial prompt using the genetic algorithm includes:

[0042] In the initial prompt As weights, the k prompts with the highest weights are selected and combined in pairs for inheritance to generate several new prompts, and the new prompts include the initial prompts;

[0043] Where m represents how many times the prompt is modified in the genetic algorithm, p is the performance index of the prompt, and α is a preset coefficient.

[0044] Optionally, α less than 1 means that the more times the prompt is inherited, the lower the probability of being selected.

[0045] Secondly, this application provides a domain-specific text classification and annotation device based on a large language model, which adopts the following technical solution, including:

[0046] The interface design module is used to design calling interfaces for text annotation for large language models based on the annotation requirements of the data to be classified.

[0047] The process design module is used to set up an automatic annotation process for large language models based on the calling interface;

[0048] The genetic search module is used to search for a preset initial prompt based on a genetic algorithm to obtain an expanded prompt.

[0049] The automatic annotation module is used to automatically classify and annotate text based on the extended prompt and the automatic annotation process control large language model.

[0050] Thirdly, this application also provides a control device, the device comprising:

[0051] It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the domain-specific text classification and annotation method based on the large language model described above.

[0052] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above for a domain-specific text classification and annotation method based on a large language model.

[0053] In summary, the system in this application sets up a call interface based on the data to be classified, and sets up the automatic annotation process of the large language model through the call interface. This facilitates the large language model to automatically annotate the data to be classified according to the call interface and the automatic standard process. At the same time, the system uses a genetic algorithm to genetically optimize the prompt of the large language model, generating more extended prompts based on the initial prompt. A genetic algorithm suitable for the large language model is designed to facilitate the search for better prompts, thereby improving the accuracy of text annotation of the large language model. In addition, compared with traditional annotation methods, this application reduces the text quantity requirement for special datasets, greatly reduces the amount of annotated text required by experts, facilitates the acquisition of text data, and improves the convenience of text annotation in special domains. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a domain-specific text classification and annotation method based on a large language model.

[0055] Figure 2 This is a structural block diagram of a domain-specific text classification and annotation device based on a large language model.

[0056] Figure labeling: 210, Interface design module; 220, Process design module; 230, Genetic search module; 240, Automatic annotation module. Detailed Implementation

[0057] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0058] Unlike typical text classification tasks, domain-specific text classification tasks are significantly more challenging. For example, the task of classifying data generated during the operation of a healthcare institution. This task lacks any publicly labeled datasets, the classification categories are complex, and classifying individual texts is difficult. It often contains specialized vocabulary used only within specific healthcare institutions, requiring individuals with both medical backgrounds and familiarity with hospital operational processes to accurately label the data. Furthermore, the scale of the data to be classified is enormous, making extensive manual labeling impossible.

[0059] Prompt is a technical term in Natural Language Processing (NLP), referring to the input format or template that enables a NLP model to perform a specific task. It is typically a piece of natural language describing the task to be completed, including the required input and the desired outcome. For example, a prompt for a text classification task like determining whether a movie review is positive might look like this:

[0060] Please indicate whether the movie review is positive or negative. If positive, output P; otherwise, output N. The movie review content is: {movie review text}

[0061] When the same natural language processing model performs the same task, different prompts often yield different results. Therefore, for each specific task, it is essential to find a better prompt to improve the model's performance. This is especially true for large language models, where the prompt has a greater impact. The clarity of the problem description, the sufficiency of contextual information, and the conciseness of the language all significantly affect the quality of the large language model's output. Some research has even found that adding expressions of praise or identifying information such as "You are a senior medical expert" to the prompt can significantly improve the performance of large language models on many tasks. Designing prompts for large language models has become an essential process for their efficient use.

[0062] Based on this, this application discloses a domain-specific text classification and annotation method based on a large language model. The execution entity is a control system that sets up a call interface based on the data to be classified. Through the call interface, the automatic annotation process of the large language model is set up, facilitating automatic annotation of the data to be classified by the large language model according to the call interface and the automatic standard process. Simultaneously, the system uses a genetic algorithm to genetically optimize the prompt of the large language model, generating more extended prompts based on the initial prompt. A genetic algorithm suitable for the large language model is designed to facilitate the search for better prompts, thereby improving the accuracy of text annotation by the large language model. Furthermore, compared to traditional annotation methods, this application reduces the text quantity requirement for special datasets, significantly reducing the amount of annotated text needed by experts, facilitating text data acquisition, and improving the convenience of domain-specific text annotation.

[0063] Reference Figure 1 The embodiments of this application include at least steps S10 to S60.

[0064] S10 is designed as a call interface for large language models based on the requirements for labeling the data to be classified.

[0065] S20 is an automatic annotation process for large language models based on API calls.

[0066] S30: Based on the genetic algorithm, search the preset initial prompt to obtain the expanded prompt.

[0067] S40 performs automatic text classification and annotation based on the extended prompt and automatic annotation process control of the large language model.

[0068] Specifically, the system sets up a call interface based on the data to be classified, and sets up the automatic annotation process of the large language model through the call interface. This facilitates the large language model to automatically annotate the data to be classified according to the call interface and the automatic standard process. At the same time, the system uses a genetic algorithm to optimize the prompt of the large language model, generating more extended prompts based on the initial prompt. A genetic algorithm suitable for the large language model is designed to facilitate the search for better prompts, thereby improving the accuracy of text annotation of the large language model. In addition, compared with traditional annotation methods, this application reduces the text quantity requirement for special datasets, greatly reduces the amount of annotated text required by experts, facilitates the acquisition of text data, and improves the convenience of text annotation in special domains.

[0069] In some embodiments, step S10 includes at least the following steps: designing a calling interface for text annotation for a large language model through an interactive dialogue, so that the large language model can autonomously select the calling interface needed to complete the text classification and annotation task based on the output text content, and the result returned by the calling interface is also returned to the large language model in the form of a dialogue.

[0070] The API calls include: a precise query interface for precisely querying the meanings of domain-specific terms; a fuzzy query interface for fuzzy querying the meanings of domain-specific terms, through which the large language model returns the top ten fuzzy matches and their corresponding meanings; a vocabulary addition interface for adding words and their corresponding meanings to the domain-specific vocabulary table; and a same-table query interface for querying other data and corresponding annotation results within the same table of the expert annotation dataset. For database-type data, the same-table query interface allows the LLM to obtain information from other fields in the entire table, providing a deeper understanding of the overall semantics of the table. The system provides a solution for more accurately classifying individual fields in a table; a field query interface for querying annotation results of text data with similar field names in the expert annotation dataset; within the same database, the field query interface allows the LLM to obtain annotation results of data with similar field names. Generally, similar field annotation results are consistent, allowing the LLM to determine the true meaning of content in field names that was originally incomprehensible, such as some abbreviations of Chinese pinyin; an unknown vocabulary interface for adding words whose meanings the large language model cannot determine to an unknown meaning vocabulary list; and an annotation output interface that provides classification annotation results or a conclusion that the data cannot be classified.

[0071] Specifically, for text in a specific domain, it is necessary to design a callable interface for the large language model, taking into account the characteristics of that domain and the characteristics of the text data to be classified. The entire interaction process is conducted in a dialogue manner. The large language model can autonomously select the functions (calling the interface) needed to complete the text classification and annotation task through the input text content. The results returned by the calling interface are also returned to the large language model in the form of a dialogue, thereby making full use of the powerful natural language understanding and memory capabilities of the large language model to complete the automatic text classification and annotation.

[0072] In some embodiments, an automatic annotation process is set up for a large language model based on the API call. The automatic annotation process specifically includes: initializing a proprietary vocabulary and adding all proprietary terms in the domain to the table; attempting to provide text classification annotation results; determining whether the text contains unfamiliar words; calling the precise query interface, and if no match is found, calling the fuzzy query interface; querying other data in the same table and attempting to provide classification annotation results; querying data with similar fields and attempting to provide classification annotation results; determining whether the meaning of unfamiliar words can be determined, and adding the words to the proprietary vocabulary or the unknown vocabulary based on the classification annotation results.

[0073] Specifically, by designing an automatic annotation process for large language models, it is possible to correctly use the calling interfaces designed for them, and at the same time, by automatically expanding the proprietary vocabulary to accumulate knowledge, the ability to automatically classify and annotate text becomes stronger and stronger.

[0074] Furthermore, considering the problem that large language models cannot classify data, the following corresponding processing is performed: when receiving an unknown vocabulary list from experts, the annotated unknown words are added back to the proprietary vocabulary; the data that failed to be classified are re-added to the data to be classified.

[0075] In practice, after a certain number of annotations are completed, the large language model checks the unknown word meaning table, where experts interpret the unknown words and add them to the proprietary vocabulary. Simultaneously, all data that the model previously failed to classify is re-added to the data to be classified by the model. This process is repeated iteratively, thereby significantly improving the accuracy of automatic annotation by the large language model.

[0076] In some embodiments, step S30 specifically includes the following steps: the expert annotation dataset is divided into a search set and a validation set according to a preset ratio; a large language model is used to search the search set and annotate the data in the validation set; the prompts without performance metrics in the initial prompts are evaluated on the validation set to obtain performance metrics, which are: the classification and annotation accuracy on the validation set after the automatic annotation process is completed; the initial prompts are genetically modified using a genetic algorithm to generate several new prompts, and the performance of the new prompts is evaluated to obtain the corresponding performance metrics; the above performance evaluation and genetic process is repeated until a preset number of iterations is reached; the prompt with the highest performance metric is selected as the prompt used in the formal automatic annotation process.

[0077] Furthermore, in the initial prompt... As weights, the k prompts with the highest weights are selected and paired together for inheritance to generate several new prompts, and the new prompts include the initial prompts;

[0078] Where m represents how many times the prompt is modified in the genetic algorithm, p is the performance index of the prompt, and α is a preset coefficient, and α is less than 1, which means that the more times the prompt is genetically modified, the lower the probability of it being selected.

[0079] Specifically, according to the automatic annotation process, a certain number of natural language description prompts are manually designed. These initial prompts are grouped into a set, denoted as S. Each prompt has a fusion count parameter m and an effectiveness metric p, where m represents the number of times the prompt has been modified in the genetic algorithm, and p is the effectiveness metric evaluating the prompt's quality. The expert-annotated dataset is then divided into a search set and a validation set in a 1:1 ratio. During the prompt search process, the large language model automatically annotates the data in the validation set. Finally, steps A and B are repeatedly executed to perform the genetic algorithm-based search until the specified number of iterations is reached.

[0080] A. For all prompts in set S that do not have an effectiveness metric, evaluate their effectiveness on the validation set. Use the classification accuracy on the validation set after the automatic annotation process is completed as the effectiveness metric p.

[0081] B. Prompt inheritance process:

[0082] In set S, with As weights, the k prompts with the highest weights are selected and paired together for inheritance to generate new prompts. Unlike traditional genetic algorithms, all new prompts and the initial prompts are placed in the dataset S to continue the inheritance process.

[0083] Because the prompts in large language models exhibit mutational characteristics—small changes can lead to significant variations in results—a penalty for the number of genetic iterations is designed to mitigate the problem of prompts getting stuck in local minima by consistently focusing on a few specific prompts during the search process. The more iterations, the lower the probability of a prompt being selected. This ensures that all prompts undergo the genetic process. α represents the coefficient of this penalty. In the actual implementation, k=20 and α=0.05.

[0084] For example, the following mutations were designed for a set of natural language prompts:

[0085] i. Add statements from the same process in another prompt within the group to each paragraph.

[0086] ii. Each paragraph is randomly swapped with the corresponding paragraph from the same process in another prompt within the combination.

[0087] iii. Restate the paragraph using a large language model.

[0088] iv. Regenerate the paragraph using back-translation.

[0089] Each mutation randomly selects a method to generate a new prompt.

[0090] After the iteration terminates, the prompt with the highest performance metric p is selected as the prompt used in the formal automatic annotation process. If the annotation accuracy of the best prompt does not meet the target, the automatic annotation process is redesigned.

[0091] The implementation principle of a domain-specific text classification and annotation method based on a large language model in this application is as follows: The system sets up a call interface based on the data to be classified, and sets up the automatic annotation process of the large language model through the call interface. This facilitates the large language model to automatically annotate the data to be classified according to the call interface and the automatic standard process. At the same time, the system performs genetic optimization of the large language model's prompt through a genetic algorithm, generating more extended prompts based on the initial prompt. A genetic algorithm suitable for the large language model is designed to facilitate the search for better prompts, thereby improving the accuracy of text annotation of the large language model. Compared with traditional annotation methods, this application reduces the text quantity requirement for special datasets, greatly reduces the amount of annotated text required by experts, facilitates the acquisition of text data, and improves the convenience of domain-specific text annotation.

[0092] Figure 1 This is a flowchart illustrating a domain-specific text classification and annotation method based on a large language model in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0093] Based on the same technical concept, referring to Figure 2 This application also provides a domain-specific text classification and annotation device based on a large language model, which adopts the following technical solution: The device includes:

[0094] Interface design module 210 is used to design a call interface for text annotation for a large language model based on the annotation requirements of the data to be classified.

[0095] Process design module 220 is used to set up an automatic annotation process for large language models based on the API call interface;

[0096] Genetic search module 230 is used to search for a preset initial prompt based on a genetic algorithm to obtain an expanded prompt;

[0097] The automatic annotation module 240 is used to automatically classify and annotate text based on the extended prompt and the automatic annotation process control of the large language model.

[0098] In some embodiments, the video frame splitting module 210 is specifically used to split the source video file into frames to obtain all the image files;

[0099] The image files are divided into a first image file and a second image file according to the time sequence of the source video files.

[0100] In some embodiments, the interface design module 210 is specifically used to design a calling interface for text annotation for the large language model through interactive dialogue, so that the large language model can autonomously select the calling interface needed to complete the text classification and annotation task based on the output text content, and the result returned by the calling interface is also returned to the large language model in the form of dialogue.

[0101] The API calls include:

[0102] The precise query interface is used to precisely query the meaning of domain-specific terms.

[0103] Fuzzy query interface, used for fuzzy querying the meaning of domain-specific terms;

[0104] The vocabulary addition interface is used to add words and their corresponding meanings to the proprietary vocabulary list;

[0105] The same-table query interface is used to query other data and corresponding annotation results within the same table in the expert annotation dataset.

[0106] The field query interface is used to query the annotation results of text data with similar field names in the expert annotation dataset;

[0107] Unknown words interface: If the large language model cannot determine the meaning of a word, the word is added to the unknown meaning vocabulary list.

[0108] The annotation output interface provides the classification annotation results.

[0109] In some embodiments, the automatic annotation process includes:

[0110] Initialize the proprietary vocabulary list by adding all proprietary terms in the field to the list;

[0111] Try to provide the text classification and annotation results;

[0112] Determine if the text contains unfamiliar words;

[0113] Call the exact query interface; if no match is found, then call the fuzzy query interface.

[0114] Query other data in the same table and attempt to provide classification and labeling results;

[0115] Query similar data in the query fields and attempt to provide classification and labeling results;

[0116] Determine whether the meaning of an unfamiliar word can be determined, and add the word to the proprietary vocabulary or the unknown vocabulary based on the classification and labeling results.

[0117] In some embodiments, the process design module 220 is further configured to add the annotated unknown words back to the proprietary vocabulary when receiving an unknown vocabulary list annotated by experts.

[0118] The data that failed to be classified were re-added to the data to be classified.

[0119] In some embodiments, the genetic search module 230 is specifically used to divide the expert-annotated dataset into a search set and a validation set according to a preset ratio;

[0120] The search is performed on the lookup set using a large language model, and the data in the validation set is labeled.

[0121] The initial prompts without performance metrics are evaluated on the validation set to obtain performance metrics. Performance metrics: classification labeling accuracy on the validation set after the automatic labeling process is completed.

[0122] The initial prompt is inherited using a genetic algorithm to generate several new prompts, and the effectiveness of the new prompts is evaluated to obtain the corresponding performance indicators.

[0123] Repeat the above effect evaluation and genetic process until the preset number of iterations is reached;

[0124] The prompt with the highest performance index is selected as the prompt used in the formal automatic annotation process.

[0125] In some embodiments, the genetic search module 230 is further configured to, in the initial prompt, use As weights, the k prompts with the highest weights are selected and paired together for inheritance to generate several new prompts, and the new prompts include the initial prompts;

[0126] Where m represents how many times the prompt is modified in the genetic algorithm, p is the performance index of the prompt, and α is a preset coefficient.

[0127] In some embodiments, α is less than 1, which means that the more times the prompt is inherited, the lower the probability of being selected.

[0128] This application also discloses a control device.

[0129] Specifically, the control device includes a memory and a processor, with the memory storing a computer program that can be loaded by the processor and executed to perform the aforementioned domain-specific text classification and annotation method based on a large language model.

[0130] This application also discloses a computer-readable storage medium.

[0131] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the domain-specific text classification and annotation method based on the large language model described above. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A large language model-based specific field text classification labeling method, characterized in that, The application comprises: Designing a calling interface for text labeling for a large language model based on the data labeling requirements of the data to be classified, wherein the calling interface is an interactive dialogue interface, so that the large language model autonomously selects the calling interface required when completing the text classification labeling task through the output text content, and the result returned by the calling interface is also returned to the large language model in the form of dialogue; Setting an automatic labeling process for the large language model based on the calling interface, comprising: Initializing a special vocabulary table, and adding all special terms in the field to the table; Attempting to give a text classification labeling result; Judging whether the text contains words that are not understood; Calling the precise query interface, and if there is no match, calling the fuzzy query interface; Querying other data in the same table to attempt to give a classification labeling result; Querying similar data in the field to attempt to give a classification labeling result; Judging whether the words that are not understood can be determined to have a meaning, and adding the words to the special vocabulary table or the unknown vocabulary table according to the classification labeling result; Searching for a preset initial prompt based on a genetic algorithm to obtain an extended prompt, comprising: An expert annotated data set is divided into a search set and a verification set according to a preset ratio; Searching for the search set by the large language model, and labeling the data in the verification set; Effectively evaluating the prompts in the initial prompt on the verification set to obtain the effect index, and the effect index is the classification labeling accuracy on the verification set after the automatic labeling process ends; Genetically modifying the initial prompt based on the genetic algorithm to generate a plurality of new prompts, and effectively evaluating the new prompts to obtain the corresponding effect index; Repeating the above effect evaluation and genetic process until a preset iteration number is reached; Selecting the prompt with the highest effect index as the prompt used in the formal automatic labeling process; Controlling the large language model to perform automatic text classification labeling based on the extended prompt and the automatic labeling process.

2. The method according to claim 1, wherein, The calling interface comprises: A precise query interface for precisely querying the meanings of special vocabulary in the field; A fuzzy query interface for fuzzy querying the meanings of special vocabulary in the field; A vocabulary adding interface for adding words and their corresponding meanings to the special vocabulary table; A same table query interface for querying other data and corresponding labeling results in the same table in the expert annotated data set; A field query interface for querying the labeling results of text data with similar field names in the expert annotated data set; An unknown vocabulary interface, which adds the vocabulary to the unknown meaning vocabulary table when the large language model cannot determine the meaning of the vocabulary; A labeling output interface for giving a classification labeling result.

3. The method according to claim 1, wherein, After setting the automatic labeling process for the large language model based on the calling interface, it further comprises: When receiving the unknown vocabulary table from expert labeling, re-adding the labeled unknown vocabulary to the special vocabulary; Re-adding the data that failed to be classified to the data to be classified.

4. The method according to claim 1, wherein, The genetic modification of the initial prompt based on the genetic algorithm to generate a plurality of new prompts comprises: In the initial prompt As the weight, the k prompts with the highest weights are selected to be combined two by two for inheritance to generate a plurality of new prompts, and the new prompts include the initial prompt; Wherein, m represents how many times the prompt is changed in the genetic algorithm, p is the effect index of the prompt, and a is a preset coefficient.

5. The large language model-based specific domain text classification labeling method according to claim 4, characterized in that, The method further includes that a is greater than 0, and the more times the prompt is genetically, the lower the probability of being selected is. 6.A large language model-based specific domain text classification labeling apparatus, characterized by comprising: The device includes: An interface design module is configured to design a calling interface for text labeling for a large language model based on data labeling requirements to be classified, wherein the calling interface is an interactive dialogue interface, so that the large language model autonomously selects the calling interface required when completing the text classification labeling task through the output text content, and the result returned by the calling interface is also returned to the large language model in the form of dialogue. A process design module is configured to set an automatic labeling process of the large language model based on the calling interface, including: Initializing a special vocabulary table, and adding special words in the field to the table; Attempting to give a text classification labeling result; Judging whether the text contains words that are not understood; Calling an accurate query interface, and calling a fuzzy query interface again if there is no match; Querying other data in the table to attempt to give a classification labeling result; Querying similar data in the field to attempt to give a classification labeling result; Judging whether the words that are not understood can be determined to have a meaning, and adding the words to a special vocabulary table or an unknown vocabulary table according to the classification labeling result; A genetic search module is configured to search a preset initial prompt based on a genetic algorithm to obtain an extended prompt, including: An expert labeling data set is divided into a lookup set and a verification set according to a preset ratio; Searching on the lookup set by the large language model, and labeling data in the verification set; Effectively evaluating prompts without an effect index in the initial prompt on the verification set to obtain the effect index, wherein the effect index is a classification labeling accuracy rate on the verification set after the automatic labeling process ends; Genetically modifying the initial prompt based on the genetic algorithm to generate a plurality of new prompts, and effectively evaluating the new prompts to obtain corresponding effect indexes; Repeating the above evaluation and genetic process until a preset iteration number is reached; Selecting a prompt with the highest effect index as a prompt used in a formal automatic labeling process; An automatic labeling module is configured to control the large language model to perform text automatic classification labeling based on the extended prompt and the automatic labeling process.

7. A control device, characterized by The device includes a memory and a processor, and the memory stores a computer program that can be loaded and executed by the processor to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The memory stores a computer program that can be loaded and executed by the processor to perform the method of any one of claims 1 to 5.

Images