Text labeling method and system and computer equipment

Abstract

The invention relates to a text labeling method and system and computer equipment. Comprising the following steps: determining a cue word associated with a labeling task through an obtained initial text and the labeling task and determining a text fragment obtained by performing multi-size segmentation on the initial text; the text segments are labeled through the large language model and the cue words, and initial labeled data are obtained; and finally, performing feature score filtering on the multiple pieces of initial annotation data through the target model to obtain target annotation data. By adopting the method, the accuracy and the consistency of the annotated data can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text annotation method, system and computer device. Background Art

[0002] Artificial intelligence technology has a wide range of applications in text annotation. By combining natural language processing, machine learning, semi-supervised learning and other technical means, it can provide strong support for efficient and accurate understanding and application.

[0003] The existing large language model processing large-scale domain question labeling methods, although do not require domain experts to label and feature engineering, but its unsupervised method relies heavily on large-scale data, has high training costs, and the model quality and labeling accuracy are directly related to the amount of training data, which easily leads to the problem of low labeling accuracy. In addition, the existing technology lacks dynamic feedback and optimization mechanisms for the quality of labeled data, and cannot be adjusted and optimized in real time according to the quality of the labeling results in actual applications. Summary of the invention

[0004] Based on this, the purpose of this application is to provide a text annotation method, system and computer device that improve the accuracy and consistency of annotation to solve the technical problems mentioned in the above background technology.

[0005] In a first aspect, the present application provides a text annotation method, including:

[0006] Obtain initial text and annotation tasks;

[0007] Determining text segments after the initial text is segmented into multiple sizes;

[0008] Determining prompt words associated with the labeling task, and labeling the text segment using a large language model and the prompt words to obtain initial labeling data;

[0009] The target model is used to perform feature scoring and filtering on multiple pieces of initial labeled data to obtain target labeled data.

[0010] In one embodiment, determining text segments after multi-size segmentation of the initial text includes: determining segmentation requirements of the initial text according to the annotation task; the segmentation requirements include segment length or segment granularity; multi-size segmentation of the initial text according to a segmentation method associated with the segmentation requirements; the segmentation method includes at least fixed-length segmentation, paragraph segmentation, sliding window segmentation and dynamic segmentation.

[0011] In one embodiment, the target model includes a value model and a scoring model; the feature scoring and filtering of multiple initial annotation data through the target model to obtain target annotation data includes: determining the value of the initial annotation data through the value model, and filtering candidate annotation data from the multiple initial annotation data according to the value; determining the final score of the candidate annotation data through the scoring model, and filtering the target annotation data from the multiple candidate annotation data according to the final score.

[0012] In one embodiment, the value of the initial annotated data is determined by a value model, including: extracting features from the initial annotated data; the extracted features include at least annotation consistency, matching degree and context relevance; determining the value corresponding to each piece of initial annotated data by a preset value assessment algorithm; the value assessment algorithm includes at least statistical analysis or a machine learning model.

[0013] In one embodiment, the scoring model is pre-trained with a high-quality annotated data set; the annotated data set includes manually annotated data and target annotated data obtained after different text annotations; the final score of the candidate annotated data is determined by the scoring model, including: determining a scoring mechanism associated with the candidate annotated data, and determining an initial score for each candidate annotated data based on the scoring mechanism; adjusting the initial score to obtain a final score for the candidate annotated data.

[0014] In one embodiment, after obtaining the target annotation data, the method further includes: feeding back the target annotation data to the large language model so that the large language model is retrained to obtain a new large language model; returning to the process of annotating the text segment using the new large language model and the prompt word until the optimized target annotation data is obtained.

[0015] In one embodiment, the large language model and the target model are deep learning models based on an attention mechanism or a generative adversarial network.

[0016] In a second aspect, the present application also provides a text annotation system, including:

[0017] A multi-size segmentation module is used to segment the acquired initial text into multiple sizes to obtain text fragments;

[0018] A prompt word generation module, used to determine prompt words associated with the labeling task;

[0019] An intelligent generation module, used to annotate the text segment using a large language model and the prompt word to obtain initial annotated data;

[0020] The small model distillation module is used to filter the feature scores of multiple initial labeled data through the target model to obtain the target labeled data;

[0021] The manual quality control module is used to perform manual quality control processing on the target annotation data.

[0022] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the relevant steps of the above text annotation method when executing the computer program.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the relevant steps of the above text annotation method are implemented.

[0024] The above-mentioned text annotation method, system, computer device and readable storage medium, by obtaining the initial text and annotation tasks, and determining the text segments after multi-size segmentation of the initial text, and determining the prompt words associated with the annotation tasks, can then annotate the text segments through the large language model and the prompt words to obtain initial annotation data. Finally, the multiple initial annotation data are feature-scored and filtered through the target model to obtain the target annotation data. Therefore, the present application first performs a preliminary annotation of the text using a large language model and prompt words, and then combines the target model to perform a detailed feature scoring and filtering process, which can improve the accuracy and consistency of the annotation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A diagram of an application environment of a text annotation method in an embodiment;

[0026] Figure 2 A flowchart of a text annotation method in one embodiment;

[0027] Figure 3 A working principle diagram of a text annotation system in one embodiment;

[0028] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The text annotation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, and through the interaction between the terminal 102 and the server 104. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 is used to send the acquired initial text and annotation tasks to the server 104. The server 104 is used to determine the text fragments after multi-size segmentation of the initial text; determine the prompt words associated with the annotation task, and annotate the text fragments through the large language model and the prompt words to obtain initial annotation data. The server 104 is also used to perform feature scoring and filtering on multiple initial annotation data through the target model to obtain target annotation data, so that the terminal 102 displays the target annotation data. Among them, the terminal 102 can be, but is not limited to, various personal computers, smart phones, tablet computers, drone equipment, smart vehicle-mounted equipment, portable wearable devices, etc. Server 104 is implemented by an independent server or a server cluster composed of multiple servers; it can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0031] In one embodiment, Figure 2 As shown, a text annotation method is provided, and the method is applied to a computer device as an example for explanation. The computer device includes Figure 1 The terminal and server in the embodiment include the following steps:

[0032] Step 202: Obtain initial text and annotation tasks.

[0033] The initial text is also the text to be annotated, such as a legal document to be annotated.

[0034] Specifically, the computer device will display a terminal to respond to the user's annotation task selection or input operation, and then perform a natural language query on the received annotation task, and perform semantic understanding on the query statement to obtain a semantic understanding result. The computer device obtains the initial text to be annotated from the pre-stored database based on the semantic understanding result. Taking the annotation task in which the user inputs "annotate important clauses in legal documents" as an example, after the computer device performs semantic understanding on the query statement, it recognizes that the user's query intention is "annotate important clauses in legal documents", and at this time it determines that the initial text is a legal document.

[0035] In one embodiment, the computer device may also directly determine the corresponding annotation task according to the text type of the initial text input by the user.

[0036] Step 204, determining text segments after the initial text is segmented into multiple sizes.

[0037] Specifically, the computer device performs multi-size segmentation on the initial text according to the corresponding segmentation method according to different requirements of the annotation task to obtain different text fragments. The different requirements of the annotation task at least include annotation tasks with relatively uniform text, sentence-level annotation tasks, annotation tasks with strong context dependence, and annotation tasks requiring overlapping information. The segmentation methods at least include fixed-length segmentation, sentence segmentation, paragraph segmentation, sliding window segmentation, and dynamic segmentation.

[0038] Therefore, the computer device can be based on fixed-length segmentation, according to a predetermined fixed length, such as segmentation every 100 characters, which is suitable for annotation tasks with relatively uniform text. The computer device can be based on sentence segmentation, segmentation is performed according to the sentence structure of the text, and each segment contains one or more complete sentences, which is suitable for sentence-level annotation tasks. The computer device can be based on paragraph segmentation, and each segment contains one or more complete paragraphs, which is suitable for paragraph-level or context-dependent annotation tasks. The computer device can be based on sliding window segmentation, using sliding window technology, moving a fixed-length window each time and segmenting, and different window lengths and step sizes can be set, which is suitable for annotation tasks that require overlapping information. The computer device can be based on dynamic segmentation, dynamically adjusting the segmentation strategy according to the text content and structure, such as segmentation according to punctuation, keywords or theme changes, which is suitable for complex annotation tasks.

[0039] Step 206 , determining prompt words associated with the labeling task, and labeling the text segment using the large language model and the prompt words to obtain initial labeling data.

[0040] Among them, the Large Language Model (LLM) is a large-scale neural network language model that has acquired powerful language understanding and generation capabilities on massive text data through pre-training. The large model can generate realistic text, understand complex language, answer questions, etc.

[0041] Specifically, the computer device will also obtain various sample data input by the user in advance, and then extract prompt words from the sample data according to the labeling task. For example, if the labeling task description is: Please mark the contract terms. The sample data is the contract period: This contract is valid for one year and will take effect from the date of signing. Rent payment: The rent should be paid at the beginning of each month, and the amount is 5,000 yuan. Liability for breach of contract: If Party B fails to pay the rent on time, it shall pay liquidated damages. Therefore, the prompt words generated are: Please mark the contract terms, including the contract period, rent payment and liability for breach of contract.

[0042] For another example, if the annotation task description is: Please mark the legal basis. The sample data is: According to Article 52 of the Contract Law of the People's Republic of China, the circumstances in which a contract is invalid include: malicious collusion that damages the interests of the state, the collective or a third party. According to Article 36 of the Labor Contract Law, the labor contract can be terminated by mutual agreement. The generated prompt words are: Please mark the legal basis, including legal provisions and regulatory references, such as Article 52 of the Contract Law and Article 36 of the Labor Contract Law. Since the computer equipment can automatically generate annotation prompt words based on the annotation task description and sample data input by the user, it provides a reference for the annotation generation of the large language model, reduces manual intervention, and improves the efficiency and accuracy of subsequent annotation.

[0043] Furthermore, the large language model is integrated into an intelligent generation module of a text annotation system, and the large language model has been pre-trained and fine-tuned in advance, has strong language understanding and generation capabilities, and can generate high-quality annotation results on large-scale data. Therefore, the computer device uses the large language model and the generated prompt words to perform preliminary annotations on the segmented text fragments, such as marking the starting and ending positions of the contract terms, identifying the content of the contract terms, etc.

[0044] Step 208: Perform feature scoring and filtering on the plurality of initial annotated data using the target model to obtain target annotated data.

[0045] Among them, the target model is a small model compared to the large language model, including a value model and a scoring model. The computer device works together through the value model and the scoring model to distill and filter the initial annotation data initially generated by the large language model, that is, to score and screen the initial annotation data, retain high-quality annotation data, and filter out low-quality or non-standard annotation data. Therefore, the quality of the annotation data is further optimized and improved, and the high-value target annotation data is retained.

[0046] In one embodiment, multiple initial annotated data are filtered by feature scoring through a target model to obtain target annotated data, including: determining the value of the initial annotated data through a value model, and filtering candidate annotated data from the multiple initial annotated data according to the value; determining the final score of the candidate annotated data through a scoring model, and filtering the target annotated data from the multiple candidate annotated data according to the final score.

[0047] Specifically, the main workflow of the value model and the scoring model working together includes: Preliminary evaluation: First, the value model performs a preliminary evaluation on the initial annotation data generated by the large language model, calculates the value of each initial annotation data, and performs preliminary filtering of the candidate annotation data. Advanced scoring: The scoring model is then used to perform more advanced scoring and evaluation on the candidate annotation data after preliminary filtering, optimizes or re-annotates the candidate annotation data with lower scores, and confirms the candidate annotation data with higher scores. Comprehensive optimization: Finally, the evaluation results of the two models are combined to comprehensively optimize and filter the initial annotation data and candidate annotation data, retaining the high-quality target annotation data that has undergone double screening and optimization.

[0048] Therefore, through the collaborative work of the value model and the scoring model, the quality and consistency of the labeled data can be effectively improved, ensuring that the final output labeled data meets the expected standards. This multi-level distillation and filtering mechanism significantly improves the performance and reliability of the automatic labeling system.

[0049] In one embodiment, the target annotation data output by the computer device will also be subject to manual quality control to ensure that the final target annotation data meets the expected quality standards. By checking and confirming the target annotation data, such as confirming whether the annotations of contract terms and legal basis are accurate, and then based on the feedback of manual quality control, the annotation results are further optimized to ensure high-quality output.

[0050] In the above-mentioned text annotation method, by obtaining the initial text and the annotation task, determining the text fragments after multi-size segmentation of the initial text, and determining the prompt words associated with the annotation task, the text fragments can be annotated through the large language model and the prompt words to obtain the initial annotation data. Finally, the multiple initial annotation data are feature-scored and filtered through the target model to obtain the target annotation data. Therefore, the present application first performs a preliminary annotation of the text using a large language model and prompt words, and then combines the target model to perform a detailed feature scoring and filtering process, which can improve the accuracy and consistency of the annotation data.

[0051] In one embodiment, determining text segments after multi-size segmentation of the initial text includes: determining segmentation requirements of the initial text according to the annotation task; and segmenting the initial text in multiple sizes according to a segmentation method associated with the segmentation requirements.

[0052] The segmentation requirements include fragment length or fragment granularity.

[0053] Specifically, since different annotation tasks have different requirements for the length or granularity of text segments, the computer equipment needs to perform multi-size segmentation on the initial text according to the segmentation method associated with the segmentation requirements. For example, in text classification, short text classification can use fixed-length segmentation, while long text classification can use paragraph segmentation or sliding window segmentation. For sentiment analysis, it is necessary to capture fine-grained sentiment information, and sentence segmentation or short paragraph segmentation can be used. For named entity recognition, in order to improve annotation accuracy, sliding window segmentation can be used to capture contextual information before and after the entity. For information extraction, it is necessary to extract specific types of information, and the appropriate segmentation strategy can be selected according to the distribution of the information, such as dynamic segmentation based on keyword or topic changes.

[0054] In this embodiment, determining the segmentation requirements of the initial text according to the annotation task can help determine the key information and paragraph structure of the text, so as to more effectively perform subsequent processing and analysis. At the same time, according to the segmentation method associated with the segmentation requirements, it is ensured that the length or granularity of the text segments can be flexibly adjusted according to the requirements of the annotation task, so that it can be applied to annotation tasks of different types and complexity.

[0055] In one embodiment, the value of the initial annotated data is determined by a value model, including: extracting features from the initial annotated data; and determining the value corresponding to each piece of initial annotated data by a preset value evaluation algorithm.

[0056] Among them, the extracted features include at least annotation consistency, matching degree and context relevance; the value assessment algorithm includes at least statistical analysis or machine learning model.

[0057] Specifically, the main task of the value model is to evaluate the value and importance of each annotated data, that is, a data value calculation model. By calculating the value of each annotated data, it is possible to determine which data is of high quality and which data may contain noise or errors. The main workflow includes: Feature extraction: The computer device extracts a variety of features from the initial annotated data generated by the large language model, such as annotation consistency, matching degree between the annotation and the original text, context relevance, etc. Value assessment algorithm: The computer device calculates the value of each annotated data based on the extracted features using a preset value assessment algorithm. These algorithms can include rule-based methods, or they can be trained regression models, classification models, etc. Threshold filtering: The computer device pre-sets a value threshold and only retains candidate annotated data with a value score higher than the threshold. This process can filter out low-quality or potentially inaccurate annotated data.

[0058] In this embodiment, determining the value of the initial annotated data through a value model can help optimize the use and allocation of the annotated data, thereby improving the quality and efficiency of the annotated data. Feature extraction of data helps to understand the characteristics and key information of the data; determining the value of each piece of initial annotated data through a preset value assessment algorithm can help allocate limited resources to the most valuable data, thereby improving annotation efficiency and data utilization.

[0059] In one embodiment, the final score of the candidate annotation data is determined by a scoring model, including: determining a scoring mechanism associated with the candidate annotation data, and determining an initial score for each candidate annotation data based on the scoring mechanism; and adjusting the initial score to obtain a final score for the candidate annotation data.

[0060] Among them, the scoring model is pre-trained through a high-quality annotated data set, that is, an AI scoring model; the annotated data set includes manually annotated data and target annotated data obtained after annotating different texts.

[0061] Specifically, the role of the scoring model is to score the initial annotation data generated by the large language model or the filtered candidate annotation data to further improve the accuracy and consistency of the annotation data. The main workflow includes: Model training: The computer device pre-trains the scoring model using a high-quality annotation data set so that it can effectively score the annotation data. The training data set may include high-quality data annotated manually, machine annotation data that has been screened and optimized, and target annotation data obtained after actually annotating different texts. Scoring mechanism: The scoring model scores the received initial annotation data or candidate annotation data. The scoring mechanism may include a variety of strategies, such as context-based similarity scoring, task-based accuracy scoring, etc. Score adjustment: The computer device adjusts the score according to the scoring results of the scoring model, for example, re-annotating those data with lower scores, or confirming and optimizing the data with higher scores. Filtering and optimization: Finally, the computer device filters out low-scoring data and retains and optimizes high-scoring target annotation data based on the scoring results and the preset scoring threshold.

[0062] In this embodiment, the process of determining the final score of the candidate annotated data through the scoring model can help to more accurately and objectively evaluate the quality and relevance of the annotated data, and improve the accuracy and efficiency of data processing and analysis. By determining the scoring mechanism and the initial score, it is helpful to establish the evaluation criteria and basis, and to ensure consistency; by adjusting the initial score, the impact of various factors on the data can be more comprehensively considered, and the accuracy and authority of the scoring results can be improved.

[0063] In one embodiment, after obtaining the target annotation data, the method further includes: feeding back the target annotation data to the large language model so that the large language model can be retrained to obtain a new large language model; returning to the process of annotating the text fragments using the new large language model and prompt words until the optimized target annotation data is obtained.

[0064] Specifically, the computer device first feeds back the final filtered target annotation data to the intelligent generation module as new input data for the intelligent generation module. The intelligent generation module uses these high-quality target annotation data to re-execute the annotation task to produce a more accurate annotation result. Assuming that the first round of target annotation data contains some entity recognition errors, such as "Apple" is incorrectly labeled as a common noun instead of a company name, after obtaining the target annotation data, the correct entity annotation will be fed back to the intelligent generation module, and the module will re-identify and correctly label "Apple" as a company name during the regeneration process.

[0065] The computer equipment then adjusts and optimizes the large language model and fine-tunes it based on the feedback data. The fine-tuning process includes updating the model parameters to better adapt it to the new high-quality annotated data and improve the accuracy and consistency of the model. Through the feedback data, if it is found that a certain vertical category term is often mislabeled, the training data for this category of terms will be added when fine-tuning the model, so that the new large language model can more accurately identify and label this category of terms.

[0066] The computer equipment finally performs reinforcement learning and adaptive adjustment, using the reinforcement learning algorithm to continuously improve the model's annotation capabilities through trial and error. The new large language model learns the best annotation strategy in a continuous feedback and optimization cycle, enabling it to dynamically respond to annotation tasks of different types and complexity, and then obtain optimized target annotation data. For example, in legal documents, the large language model gradually learns how to correctly annotate legal terms and entities in different contexts through multiple feedback and reinforcement learning, thereby improving annotation accuracy.

[0067] In this embodiment, feeding the target labeled data back to the large language model for retraining can help the language model better understand and process data in a specific task field, and improve its ability to label and understand data. Through continuous feedback and iterative training, the large language model can gradually optimize learning and perform subsequent re-labeling, improving adaptability and accuracy to new data and tasks. Returning to the process of labeling text fragments with a new large language model and prompt words can ensure the accuracy and consistency of optimized labeled data, improve data processing and analysis effects, and further enhance the performance and performance of the model.

[0068] In one embodiment, the large language model and the target model are deep learning models based on an attention mechanism or a generative adversarial network.

[0069] Specifically, deep learning models such as the Transformer model. The attention mechanism and the Transformer model can process text in parallel and focus on important parts of the text, which is suitable for complex annotation tasks. In the re-annotation process of legal documents, the Transformer model can identify and annotate key legal terms and citations through the attention mechanism to improve the consistency of annotation. Generative adversarial networks (GANs) can be used to generate high-quality annotated data, and improve the quality of data generation through adversarial training of generators and discriminators. In vertical document annotation, generative adversarial networks can generate high-quality vertical term annotation data, and filter low-quality data through discriminators to ensure the accuracy of annotation. Therefore, the use of deep learning models based on attention mechanisms or generative adversarial networks for feedback and retraining of target annotation data improves the quality and diversity of model-generated data.

[0070] In one embodiment, Figure 3 As shown, Figure 3 The figure is a working principle diagram of a text annotation system based on an intelligent generation module and a small model distillation module. For the specific working principle, please refer to the implementation process of the text annotation method specifically described above, which will not be repeated here. The text annotation system includes a multi-size segmentation module, a prompt word generation module, an intelligent generation module, a small model distillation module and a manual quality control module, wherein: the multi-size segmentation module is used to perform multi-size segmentation on the acquired initial text to obtain text fragments; the prompt word generation module is used to determine the prompt words associated with the annotation task; the intelligent generation module is used to annotate the text fragments through the large language model and the prompt words to obtain the initial annotation data; the small model distillation module is used to perform feature scoring and filtering on multiple initial annotation data through the target model to obtain the target annotation data; the manual quality control module is used to perform manual quality control processing on the target annotation data.

[0071] In this embodiment, the text annotation system of the present application provides an efficient and accurate text automatic annotation solution through multi-size text segmentation, automatic generation of prompt words, combination of large and small models, and strict manual quality control, which significantly improves the flexibility, efficiency and quality of annotation tasks. In addition, by establishing human-computer collaborative optimization and continuous online learning mechanisms, the system continuously optimizes model parameters and annotation strategies based on actual annotation effects and user feedback, forming a closed-loop optimization mechanism to ensure that the system continues to learn and improve in actual applications, and improves annotation quality and system performance.

[0072] The text annotation system proposed in this application can be widely used in the following scenarios:

[0073] (1) Data annotation platform: It provides efficient and accurate text annotation services, which are suitable for the data annotation needs of data annotation companies, scientific research institutions and enterprises. It can significantly reduce the workload and time of manual annotation and improve the efficiency and accuracy of annotation.

[0074] (2) Intelligent customer service system: Automatically label and classify user questions to improve the response speed and accuracy of the customer service system. It can quickly understand and classify user questions, provide more accurate and faster answers, and improve user experience.

[0075] (3) Content review system: Automatically annotate and review massive amounts of text content to improve the efficiency and accuracy of content review. Applicable to social media platforms, news websites, content review companies, etc., to help identify and filter inappropriate content, spam, etc.

[0076] (4) Natural Language Processing (NLP) Research: Provide high-quality annotated data sets for NLP research, supporting research tasks such as text classification, sentiment analysis, and information extraction. It can provide researchers with a large amount of high-quality annotated data to promote the development and progress of NLP technology.

[0077] (5) Question and answer system: used to build and optimize the question and answer system, automatically annotate questions and answers, and improve the accuracy and response speed of the question and answer system. It is suitable for scenarios such as intelligent question and answer, online education, and improves the user interaction experience.

[0078] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0079] Based on the same inventive concept, the embodiment of the present application also provides a text annotation device for implementing the above-mentioned text annotation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more text annotation device embodiments provided below can refer to the limitations on the text annotation method above, and will not be repeated here.

[0080] In one embodiment, a text annotation device is provided, comprising: a text segmentation module, an initial annotation module and a scoring and filtering module, wherein:

[0081] The text segmentation module is used to obtain the initial text and the annotation task, and determine the text segments after multi-size segmentation of the initial text.

[0082] The initial annotation module is used to determine the prompt words associated with the annotation task, and annotate the text fragments through the large language model and the prompt words to obtain initial annotation data.

[0083] The scoring and filtering module is used to perform feature scoring and filtering on multiple pieces of initial labeled data through the target model to obtain target labeled data.

[0084] Each module in the above text annotation can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0085] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O), a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a text annotation method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0086] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0087] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0088] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0089] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0090] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0091] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A text annotation method, characterized in that: The method comprises: Obtain initial text and annotation tasks; Determining text segments after the initial text is segmented into multiple sizes; Determining prompt words associated with the labeling task, and labeling the text segment using a large language model and the prompt words to obtain initial labeling data; The target model is used to perform feature scoring and filtering on multiple pieces of initial labeled data to obtain target labeled data.

2. The method according to claim 1, characterized in that The determining of the text segments after the initial text is segmented into multiple sizes includes: Determining segmentation requirements of the initial text according to the annotation task; the segmentation requirements include segment length or segment granularity; The initial text is segmented into multiple sizes according to the segmentation method associated with the segmentation requirement; the segmentation method at least includes fixed-length segmentation, paragraph segmentation, sliding window segmentation and dynamic segmentation.

3. The method according to claim 1, characterized in that The target model includes a value model and a scoring model; the target model is used to perform feature scoring and filtering on multiple pieces of initial annotated data to obtain target annotated data, including: Determining the value of the initial annotated data through a value model, and filtering candidate annotated data from the plurality of initial annotated data according to the value; The final score of the candidate annotation data is determined by a scoring model, and the target annotation data is screened out from the plurality of candidate annotation data according to the final score.

4. The method according to claim 3, characterized in that The determining the value of the initial labeled data by using a value model includes: Extracting features from the initial annotated data; the extracted features at least include annotation consistency, matching degree and context relevance; The value corresponding to each piece of initial annotated data is determined by a preset value assessment algorithm; the value assessment algorithm at least includes statistical analysis or a machine learning model.

5. The method according to claim 3, characterized in that: The scoring model is pre-trained by a high-quality annotated data set; the annotated data set includes manually annotated data and target annotated data obtained after annotating different texts; Determining the final score of the candidate labeled data by using a scoring model includes: Determining a scoring mechanism associated with the candidate annotation data, and determining an initial score for each candidate annotation data based on the scoring mechanism; The initial score is adjusted to obtain a final score for the candidate labeled data.

6. The method according to claim 1, characterized in that After obtaining the target labeling data, the method further includes: Feeding back the target labeled data to the large language model so that the large language model is retrained to obtain a new large language model; The process of returning to labeling the text segment using the new large language model and the prompt word is continued until optimized target labeling data is obtained.

7. The method according to any one of claims 1 to 6, characterized in that: The large language model and the target model are deep learning models based on an attention mechanism or a generative adversarial network.

8. A text annotation system, characterized in that: The system includes a multi-size segmentation module, a prompt word generation module, an intelligent generation module, a small model distillation module and a manual quality control module, wherein: A multi-size segmentation module is used to segment the acquired initial text into multiple sizes to obtain text fragments; A prompt word generation module, used to determine prompt words associated with the labeling task; An intelligent generation module, used to annotate the text segment using a large language model and the prompt word to obtain initial annotated data; The small model distillation module is used to filter the feature scores of multiple initial labeled data through the target model to obtain the target labeled data; The manual quality control module is used to perform manual quality control processing on the target annotation data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images