Data labeling system, method, medium and terminal based on large language model

Abstract

The application provides a data labeling system and method based on a large language model, a medium and a terminal. The system comprises: a data access layer for collecting data to be labeled and preprocessing the data to be labeled to convert it into a unified data format; an intelligent labeling engine layer for retrieving context information in a vertical field for the received data to be labeled in the unified data format, and inputting the retrieved context information and the data to be labeled into a plurality of pre-trained large language models after splicing; the large language model is used to generate a labeling result of the data to be labeled; a quality control layer for monitoring the quality of the labeling results generated by the plurality of large language models, and performing feedback analysis according to the monitored labeling quality problems. The application can realize full-process automation from obtaining data to be labeled to generating a labeling result, integrates vertical field knowledge, strictly controls the labeling result, and continuously optimizes the process through feedback.

Description

Technical Field

[0001] This application relates to the field of data annotation technology, and in particular to a data annotation system, method, medium and terminal based on a large language model. Background Technology

[0002] With the rapid development of artificial intelligence technology, data annotation has become a core infrastructure of the AI ​​industry. Data annotation, through operations such as feature extraction, classification, annotation, and labeling, transforms human knowledge and logical thinking into computer-recognizable language, and is a crucial step in building high-quality AI datasets. Currently, the industry is showing a development trend of "technology-driven, quality-oriented, and vertically integrated," marking a comprehensive transformation from traditional models to intelligent, professional, and large-scale operations.

[0003] However, the traditional model of "manual annotation + Excel management" that the industry has long relied on is not only inefficient in dealing with the massive and highly complex data annotation needs required for large model training, but also brings high labor costs. Furthermore, due to the human's misunderstanding of the annotation rules, the annotation quality is inconsistent.

[0004] To overcome the shortcomings of manual annotation, AI-assisted annotation technology has emerged. However, existing automated annotation technologies still have the following limitations:

[0005] (1) Limited automation: The accuracy of current AI annotation presents a "smile-shaped" curve. The accuracy can reach 99.5% in closed scenes such as face boxes and license plate numbers, but the accuracy drops sharply to below 70% in open world. Conventional automation tools can only handle simple tasks and do not support the needs of complex semantic understanding and cross-modal association annotation.

[0006] (2) Lack of vertical industry targeting: General AI annotation tools are difficult to meet the professional needs of vertical industries. Annotation in fields such as medical care and finance requires both understanding of professional knowledge and algorithm logic. However, existing tools lack the ability to integrate domain knowledge, resulting in insufficient feature extraction accuracy.

[0007] (3) Inadequate quality control mechanism: The existing automated annotation system lacks an effective quality assurance system and cannot achieve intelligent positioning from surface anomalies to deep root causes. When annotation quality problems are detected, it is difficult to quickly locate the root cause of the problem and provide optimization suggestions.

[0008] Therefore, it is necessary to provide a data annotation system, method, medium, and terminal based on a large language model to solve the above-mentioned problems in the existing technology. Summary of the Invention

[0009] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a data annotation system, method, medium and terminal based on a large language model to solve the technical problems of limited automation, lack of vertical industry targeting and imperfect quality control mechanism in the prior art.

[0010] To achieve the above and other related objectives, a first aspect of this application provides a data annotation system based on large language models, comprising: a data access layer for collecting data to be annotated and preprocessing the data to be annotated to convert it into a unified data format; an intelligent annotation engine layer for retrieving contextual information of the vertical domain for the received data to be annotated in the unified data format, and concatenating the retrieved contextual information with the data to be annotated before inputting it into multiple pre-trained large language models; the large language models for generating annotation results for the data to be annotated; and a quality control layer for monitoring the quality of the annotation results generated by the multiple large language models and performing feedback analysis based on the monitored annotation quality problems.

[0011] In some embodiments of the first aspect of this application, the data access layer includes: a multimodal data acquisition unit for acquiring data to be labeled from multiple heterogeneous data sources based on a unified data access interface; a data preprocessing unit for processing the acquired data to be labeled with missing values, outliers, and noise data, and for fusing and aligning the data to be labeled from each modality; and a data standardization unit for converting the preprocessed data to be labeled into a unified data format to construct standardized data.

[0012] In some embodiments of the first aspect of this application, the intelligent annotation engine layer includes: a context information retrieval unit, configured to retrieve context information related to the data to be annotated from a knowledge base in a vertical domain according to configured business rules, and to concatenate the retrieved context information and the data to be annotated; a model input data construction unit, configured to add the concatenated context information and the data to be annotated to a specified position of a preset prompt word for input into multiple large language models; and an annotation generation unit, configured to use the large language models to understand the semantic relationships between multimodal data to be annotated based on the Transformer architecture, and to generate annotation results for the data to be annotated based on the model input data.

[0013] In some embodiments of the first aspect of this application, the intelligent annotation engine layer further includes: an interactive annotation interface for providing various annotation type tools and displaying the annotation results generated by the large language model in real time; an annotation rule engine unit for presetting annotation rules for various scenarios to form an annotation rule template library; and an active learning unit for identifying data samples with high model prediction uncertainty and prioritizing their transfer to manual annotation.

[0014] In some embodiments of the first aspect of this application, the quality control layer includes: a multi-level quality inspection unit, used to perform layer-by-layer quality inspection on the generated annotation results according to the preset quality inspection rules of each layer based on a multi-level hierarchical inspection mechanism; a consistency inspection unit, used to compare the annotation results generated by multiple large language models and identify the differences between the annotation results; an anomaly detection unit, used to detect outliers in the generated annotation results to identify abnormal annotation results; and a quality feedback unit, used to feed back the layer-by-layer quality inspection results output by the multi-level quality inspection unit, the annotation differences output by the consistency inspection unit, and the abnormal annotation results output by the anomaly detection unit to the large language model for fine-tuning the large language model.

[0015] In some embodiments of the first aspect of this application, a system monitoring layer is further included, comprising: a performance monitoring unit for real-time tracking of various operational metrics across the entire chain; a log management unit for recording events occurring in each operation across the entire chain, and storing all recorded events in a hierarchical and categorized manner according to log level; a security audit unit for performing access control based on identity authentication information and setting secondary verification rules for sensitive operations; and an alarm notification unit for issuing alarms based on preset multi-level alarm rules and notifying operation and maintenance personnel of the alarms through multiple channels.

[0016] In some embodiments of the first aspect of this application, a workflow management layer is further included, comprising: a task allocation unit, used to allocate tasks according to task difficulty, data type, and urgency, and to prioritize task allocation based on a task priority queue; a progress tracking unit, used to display the progress of each annotation task in real time, and to visualize the task progress in the form of statistical reports and charts; a resource scheduling unit, used to dynamically allocate computing resources according to the length and priority of the annotation task; and a version control unit, used to create a version number for each piece of data to be annotated and the annotation result, and to record version change data.

[0017] To achieve the above and other related objectives, a second aspect of this application provides a data annotation method based on large language models, comprising: collecting data to be annotated and preprocessing the data to be annotated to convert it into a unified data format; retrieving contextual information of a vertical domain for the received data to be annotated in the unified data format, and concatenating the retrieved contextual information with the data to be annotated and inputting it into multiple pre-trained large language models; the large language models are used to generate annotation results for the data to be annotated; monitoring the quality of the annotation results generated by the multiple large language models, and performing feedback analysis based on the monitored annotation quality issues.

[0018] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0019] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method.

[0020] As described above, the data annotation system, method, medium, and terminal based on a large language model of this application have the following beneficial effects:

[0021] The system acquires data to be labeled through the data access layer, and the intelligent labeling engine layer automatically generates labeling results using multiple pre-trained large language models. This achieves full automation from acquiring data to generating labeling results, greatly improving labeling efficiency. Furthermore, the pre-trained large language models possess cross-modal semantic understanding and context awareness capabilities, enabling them to understand the semantic relationships between multimodal data and generate labeling results by reasoning within the retrieved contextual information of the vertical domain. This improves the labeling accuracy in the open world and integrates vertical domain knowledge. Simultaneously, a quality control layer is designed to monitor the quality of multiple labeling results. When quality issues are detected, the system can pinpoint the problematic node and continuously optimize the process. Attached Figure Description

[0022] Figure 1 The diagram shown is a block diagram of a data annotation system based on a large language model according to an embodiment of this application.

[0023] Figure 2 The diagram shows the workflow of a data annotation system based on a large language model in one embodiment of this application.

[0024] Figure 3 The diagram shown is a flowchart of a data annotation method based on a large language model in one embodiment of this application.

[0025] Figure 4 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0026] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0027] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0028] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0029] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0030] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0031] <1> Large Language Model (LLM): refers to a deep learning model trained on a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics by being trained on massive datasets. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, which to some extent simulates the human language cognition and generation process.

[0032] <2> Transformer architecture: a deep learning model architecture based on self-attention mechanism, widely used in natural language processing (NLP) and other sequence-to-sequence tasks. It consists of two parts: an encoder and a decoder. Each part usually contains multiple identical layers (blocks) so that the model can learn complex feature representations and semantic relationships.

[0033] <3> RESTful API is an architectural style based on the HTTP protocol. Its core lies in manipulating resources through HTTP methods. HTTP methods are an important part of RESTful APIs and are used to define the interaction between clients and servers.

[0034] <4> YAML is a hierarchical and highly readable data format used to represent data structures and configuration information. It is characterized by its simplicity, extensibility, and ease of understanding.

[0035] <5> Hugging Face Evaluate is a library for evaluating machine learning models and datasets. It provides a simple and standardized way to report model performance. The library includes many popular evaluation metrics, covering a variety of tasks from natural language processing to computer vision, and includes metrics for specific datasets.

[0036] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1 This diagram illustrates a block diagram of a data annotation system based on a large language model, as described in an embodiment of the present invention. The data annotation system based on a large language model in this embodiment includes: a data access layer 101, an intelligent annotation engine layer 102, a quality control layer 103, a system monitoring layer 104, and a workflow management layer 105. Through the modular, layered architecture design of the data access layer 101, intelligent annotation engine layer 102, quality control layer 103, system monitoring layer 104, and workflow management layer 105, the entire data annotation process is automated, significantly improving annotation efficiency and reducing labor costs.

[0037] The data access layer 101 is used to collect data to be labeled and to preprocess the data to be labeled so as to convert it into a unified data format.

[0038] In some embodiments of this application, the data access layer includes: a multimodal data acquisition unit, used to acquire data to be labeled from multiple heterogeneous data sources based on a unified data access interface; a data preprocessing unit, used to process the acquired data to be labeled for missing values, outliers and noise data, and to fuse and align the data to be labeled from each modality; and a data standardization unit, used to convert the preprocessed data to be labeled into a unified data format to construct standardized data.

[0039] Specifically, the multimodal data acquisition unit supports the collection of labeled data from various heterogeneous data sources, including public datasets, enterprise internal databases, data obtained from web crawlers, real-time streaming media data, and internal materials provided by industry experts. The multimodal data acquisition unit employs a unified data access interface, such as a RESTful API interface, message subscription, and file transfer interface, to achieve unified access to massive, heterogeneous data sources.

[0040] The data preprocessing unit cleans the collected raw data to be labeled, specifically including: identifying missing values ​​and NULL values ​​in the data, and filling or safely removing them according to strategies (i.e., handling missing values); detecting and processing outlier data points that significantly deviate from the normal range using statistical methods or machine learning algorithms, which can be smoothed, corrected, or removed (i.e., handling outliers); and filtering out meaningless random interference (such as image noise, garbled text, and audio noise) to improve the signal-to-noise ratio (i.e., handling noisy data). Furthermore, the data preprocessing unit fuses and aligns data from the same event but different modalities to form a complete dataset.

[0041] The data standardization unit transforms the cleaned, fused, and aligned data to be labeled into standardized data with a unified structure, making it suitable for input into large language models.

[0042] The intelligent annotation engine layer 102 is used to retrieve contextual information of the vertical domain for the received data to be annotated in a unified data format, and then concatenate the retrieved contextual information with the data to be annotated and input it into multiple pre-trained large language models; the large language models are used to generate annotation results for the data to be annotated.

[0043] In some embodiments of this application, the intelligent annotation engine layer includes: a context information retrieval unit, used to retrieve context information related to the data to be annotated from a knowledge base in a vertical domain according to configured business rules, and to concatenate the retrieved context information and the data to be annotated; a model input data construction unit, used to add the concatenated context information and the data to be annotated to a specified position of a preset prompt word for input into multiple large language models; and an annotation generation unit, used by the large language models to understand the semantic relationships between multimodal data to be annotated based on the Transformer architecture, and to generate annotation results for the data to be annotated based on the model input data.

[0044] Specifically, based on configured business rules, the system retrieves contextual information related to the data to be labeled from a knowledge base within the vertical domain. This abandons the traditional isolated text labeling model, providing rich contextual information for label generation. For example, for a social media comment (data to be labeled), it associates the content of its parent post, sub-forum information, and image links within the post, and then concatenates this retrieved contextual information with the data to be labeled. The concatenated data is then added to a preset prompt word template as input data for a large language model. At this point, the input data contains the data to be labeled, contextual information, and prompt words. Multiple pre-trained large language models are invoked, with the input data fed into each model. For example, the GPT model and the Qwen model are invoked simultaneously. These multiple large language models, based on the Transformer architecture, can understand the semantic relationships between data of different modalities, such as the correspondence between character actions in a video and the emotional content of dialogue in audio. They generate labeling results for the data to be labeled based on the input data, supporting automatic labeling of multimodal data such as text, images, audio, and video.

[0045] In some embodiments of this application, the intelligent annotation engine layer further includes: an interactive annotation interface for providing various annotation type tools and displaying the annotation results generated by the large language model in real time; an annotation rule engine unit for presetting annotation rules for various scenarios to form an annotation rule template library; and an active learning unit for identifying data samples with high model prediction uncertainty and prioritizing their transfer to manual annotation.

[0046] Specifically, the interactive annotation interface provides an intuitive user interface that supports efficient collaboration between annotators and the system. It offers a rich set of annotation tools, including bounding boxes, polygons, key points, semantic segmentation, etc., and displays the annotation results of multiple LLM models in real time. If the annotation results of the LLM models are incorrect, the annotators can quickly correct the errors.

[0047] The annotation rule engine unit supports user-defined annotation rules and templates, presets annotation specifications for different scenarios (such as psychology and pregnancy) to form a built-in annotation rule template library, and supports custom annotation processes through YAML configuration files.

[0048] Active learning units can automatically identify data samples with high model prediction uncertainty and prioritize submitting these samples to manual annotation, thereby obtaining effective annotation results with minimal human cost.

[0049] The quality control layer 103 is used to monitor the quality of the annotation results generated by the multiple large language models and to perform feedback analysis based on the monitored annotation quality problems.

[0050] In some embodiments of this application, the quality control layer includes: a multi-level quality inspection unit, used to perform layer-by-layer quality inspection on the generated annotation results according to the preset quality inspection rules of each layer based on a multi-level hierarchical inspection mechanism; a consistency inspection unit, used to compare the annotation results generated by multiple large language models and identify the differences between the annotation results; an anomaly detection unit, used to detect outliers in the generated annotation results to identify abnormal annotation results; and a quality feedback unit, used to feed back the layer-by-layer quality inspection results output by the multi-level quality inspection unit, the annotation differences output by the consistency inspection unit, and the abnormal annotation results output by the anomaly detection unit to the large language model for fine-tuning the large language model.

[0051] For example, the multi-level quality inspection unit includes three levels of quality inspection, specifically: Level 1 inspection (annotator self-inspection) achieves 100% coverage. After completing the annotation, the annotator immediately reviews it, and the system provides real-time prompts for annotation specifications, while automatically detecting obvious formatting and logical errors; Level 2 inspection (team leader spot check) selectively checks no less than 30% of the data. Experienced annotation team leaders conduct random checks, focusing on complex samples and boundary conditions, and use the Hugging Face Evaluate library to test accuracy, setting clear rejection criteria. For example, if the F1 value is below 0.9, the data must be returned for re-annotation to avoid misunderstandings by the system; Level 3 inspection (quality inspection expert review) covers 5-10% of the samples. Vertical domain experts conduct the final review of key samples. For example, professional content such as medical terminology and legal provisions must be reviewed by experts, which can reduce the error rate by 40%.

[0052] Since multiple large language models are scheduled to generate annotation results, after the annotation results are generated, the consistency check unit compares the annotation results generated by the multiple large language models, automatically identifies the annotation differences between the annotation results and highlights them for the annotators to focus on, so as to ensure the accuracy of the annotation results.

[0053] The anomaly detection unit automatically detects outliers in the labeled results, such as "clouds" appearing in a scene labeled "indoors," enabling intelligent quality inspection of large-scale data and improving inspection efficiency.

[0054] The quality feedback unit collects quality issues from multi-level quality inspection units, consistency check units, and anomaly detection units, analyzes the root causes of these issues, provides targeted feedback and optimization, and feeds the feedback back to the pre-trained large language model to fine-tune the model and continuously improve the annotation generation model.

[0055] The data annotation system based on a large language model in this application automates the entire process from data collection to final delivery through intelligent agent-based automatic annotation, significantly improving data processing efficiency and automatically completing the original 41-step annotation process. Furthermore, the Transformer architecture based on the large language model can understand the deep semantic relationships between different modal data, achieving true cross-modal annotation. At the same time, the design of multi-level quality inspection units and anomaly detection units greatly reduces the amount of manual review and enables intelligent localization from surface anomalies to deep-seated root causes based on quality issues, thereby continuously optimizing the annotation process.

[0056] In some embodiments of this application, the data annotation system based on a large language model further includes a system monitoring layer, which includes: a performance monitoring unit for real-time tracking of various operational metrics across the entire chain; a log management unit for recording events occurring in each operation across the entire chain and storing all recorded events in a hierarchical and categorized manner according to log level; a security audit unit for performing access control based on identity authentication information and setting secondary verification rules for sensitive operations; and an alarm notification unit for issuing alarms based on preset multi-level alarm rules and notifying maintenance personnel of the alarms through multiple channels.

[0057] Specifically, the performance monitoring unit is responsible for tracking various operational metrics across the entire chain in real time, such as monitoring the usage of CPU, memory, GPU, disk, and network to ensure that the annotation task has sufficient computing resources. The real-time data display facilitates real-time monitoring.

[0058] The log management unit records all operations in the system, including logs for user logins, operations, and exceptions. It supports hierarchical management and categorized storage of logs, and provides log query, analysis, and export functions to ensure the auditability of system operations.

[0059] The security audit unit ensures the security of the system and data by: implementing strict identity authentication and access control, performing secondary verification of sensitive operations, regularly scanning and patching security vulnerabilities, supporting encrypted data storage and transmission, and ensuring that labeled data and system resources are not accessed, tampered with, or leaked without authorization.

[0060] The alarm notification unit promptly detects and handles system anomalies. It sets up multi-level alarm rules, including warnings, errors, and critical errors, and supports multiple notification methods such as email, SMS, and instant messaging to notify maintenance personnel. It provides an automatic alarm recovery mechanism and supports batch processing and confirmation of alarms.

[0061] In some embodiments of this application, the data annotation system based on a large language model further includes a workflow management layer, which includes: a task allocation unit, used to allocate tasks according to task difficulty, data type, and urgency, and to prioritize task allocation based on a task priority queue; a progress tracking unit, used to display the progress of each annotation task in real time, and to visualize the task progress in the form of statistical reports and charts; a resource scheduling unit, used to dynamically allocate computing resources according to the length and priority of the annotation tasks; and a version control unit, used to create a version number for each piece of data to be annotated and the annotation result, and to record version change data.

[0062] Specifically, the task allocation unit implements intelligent task allocation, which dynamically allocates tasks based on the annotator's skill level and workload. It supports allocation according to dimensions such as task difficulty, type, and urgency, and adopts a priority queue mechanism to ensure that important tasks are processed first. It also supports team collaboration mode, allowing multiple people to annotate the same dataset at the same time.

[0063] The progress tracking unit provides full-process visual monitoring, displays the completion progress of each task in real time, predicts task completion time and resource requirements, automatically identifies progress bottlenecks and issues warnings, and provides multi-dimensional statistical reports and charts.

[0064] The resource scheduling unit is responsible for optimizing computing resources and dynamically allocating computing resources such as GPU, CPU, and memory based on the length and priority of the task queue.

[0065] The version control unit ensures the traceability of data and annotation results, creates a unique version number for each dataset and annotation result, records all change history, including the operator, time, content, etc., and supports version rollback and comparison functions to ensure data integrity and consistency.

[0066] This application's data annotation system based on a large language model accelerates the training and iteration of AI models by providing high-quality, low-cost, and efficient data annotation services, promoting the application of AI technology in various fields, particularly in key areas such as healthcare, autonomous driving, and finance, where it will have a profound impact. Furthermore, it provides crucial data infrastructure support for the digital transformation of traditional industries. For example, in intelligent manufacturing, automated annotation technology enables intelligent product quality inspection; in smart city construction, it supports applications such as traffic flow analysis and security monitoring. Simultaneously, it can build a new data service platform, providing enterprises with pay-as-you-go data annotation services. This business model will lower the barrier to AI application, allowing more SMEs to enjoy the convenience brought by AI technology. Moreover, through automation and intelligent technologies, it significantly reduces the demand for professional annotation talent, alleviating the current talent shortage in the data annotation industry. At the same time, the system's training function helps more people quickly master data annotation skills. In addition, through efficient data annotation and management, it fully explores the value of data, avoiding the waste of data resources. Especially for massive amounts of historical data, automated annotation technology can achieve the rediscovery and utilization of value.

[0067] Figure 2 This illustration shows a schematic diagram of the workflow of a data annotation system based on a large language model according to one embodiment of this application. For example... Figure 2 As shown, combined with specific application scenarios Figure 2 The workflow of the data annotation system based on a large language model will be further explained below. Scenario description: Annotating the sentiment (positive, negative, and neutral) of a social media comment.

[0068] The system retrieves a social media comment, "This battery life is unbeatable!", as the data to be labeled. This comment is preprocessed and converted into a unified data format suitable for input into a large language model. The system then retrieves the context of this comment from a knowledge base, such as the parent post: "Review of the new X phone: Beautiful appearance, but extremely poor battery life, requires charging three times a day," and the sub-forum information: "Digital Product Complaint Section." The system then concatenates the retrieved parent post content, sub-forum information, and the comment together. The comments are populated into a preset prompt template, for example, the prompt template is "You are an accurate sentiment analysis assistant. Please judge the sentiment tendency of the comments based on the following background and comments." This constitutes the input data. The input data is then fed into the GPT model and the Qwen model. The GPT model outputs labeling result 1 as "negative," and the Qwen model outputs labeling result 2 as "positive." At this point, the difference between labeling result 1 and labeling result 2 is highlighted on the interactive labeling interface. The labeler judges labeling result 1 as correct, clicks the confirmation button, and the labeling of this data is completed. Furthermore, the model is continuously iterated and optimized based on the labeling differences.

[0069] Figure 3 This is a flowchart illustrating the data annotation method based on a large language model provided in an embodiment of this application. Figure 3 As shown, the data annotation method based on a large language model includes the following steps:

[0070] Step S31: Collect the data to be labeled and preprocess the data to be labeled to convert it into a uniform data format;

[0071] Step S32: Retrieve contextual information of the vertical domain for the received data to be labeled in a unified data format, and concatenate the retrieved contextual information with the data to be labeled and input it into multiple pre-trained large language models; the large language models are used to generate the labeling results of the data to be labeled;

[0072] Step S33: Monitor the quality of the annotation results generated by the multiple large language models, and conduct feedback analysis based on the annotation quality problems detected.

[0073] It should be understood that the data annotation method based on a large language model provided in this application embodiment can achieve the functions of the above-mentioned data annotation system based on a large language model. For the sake of brevity, it will not be described in detail here.

[0074] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0075] Figure 4 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 4 As shown, the electronic terminal 400 includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic terminal 400 are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general will label all buses as bus systems.

[0076] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0077] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0078] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic terminal 400. Examples of this data include: any executable program for operation on the electronic terminal 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The methods provided in this embodiment of the invention can be included in the application program 4022.

[0079] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0080] In an exemplary embodiment, the electronic terminal 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0081] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figure 3 The method in the illustrated embodiment.

[0082] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0083] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0084] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0088] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0089] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] In summary, addressing the limitations of existing technologies such as limited automation, lack of industry-specific focus, and inadequate quality control mechanisms, this application provides a data annotation system, method, medium, and terminal based on large language models. The system acquires data to be annotated through a data access layer, and multiple pre-trained large language models in the intelligent annotation engine layer automatically generate annotation results. This achieves full automation from data acquisition to result generation, significantly improving annotation efficiency. Furthermore, the pre-trained large language models possess cross-modal semantic understanding and context awareness capabilities, enabling them to understand semantic relationships between multimodal data and generate annotation results based on retrieved contextual information from the vertical domain, thus improving the annotation accuracy in open-world environments and integrating vertical domain knowledge. Simultaneously, a quality control layer is designed to monitor the quality of multiple annotation results. When quality issues are detected, the problematic nodes can be located for continuous optimization. Therefore, this application effectively overcomes the shortcomings of existing technologies and possesses high industrial applicability.

[0091] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A data annotation system based on a large language model, characterized in that, include: The data access layer is used to collect data to be labeled and to preprocess the data to be labeled in order to convert it into a unified data format. The intelligent annotation engine layer is used to retrieve contextual information of the vertical domain for the received data to be annotated in a unified data format, and then concatenate the retrieved contextual information with the data to be annotated and input it into multiple pre-trained large language models. The large language model is used to generate annotation results for the data to be annotated. The intelligent annotation engine layer includes: The context information retrieval unit is used to retrieve context information related to the data to be labeled from the knowledge base of the vertical domain according to the configured business rules, and to concatenate the retrieved context information with the data to be labeled. The data to be labeled includes social media comments. The process of retrieving context information related to the data to be labeled from the knowledge base of the vertical domain according to the configured business rules and concatenating the retrieved context information with the data to be labeled specifically includes: associating the parent post content, sub-forum information, and image links in the post with the social media comments from the knowledge base of the vertical domain according to the configured business rules, and concatenating the retrieved parent post content, sub-forum information, image links in the post, and social media comments. The model input data construction unit is used to add the concatenated context information and the data to be labeled to the specified positions of the preset prompt words, so that they can be input into multiple large language models; The annotation generation unit is used by large language models to understand the semantic relationships between multimodal data to be labeled based on the Transformer architecture, and to generate annotation results for the data to be labeled based on the model input data. The quality control layer is used to monitor the quality of the annotation results generated by the multiple large language models and to perform feedback analysis based on the annotation quality problems detected. The quality control layer includes: Multi-level quality inspection units are used to perform layer-by-layer quality inspection on the generated annotation results based on the preset quality inspection rules of each layer, according to the multi-level hierarchical inspection mechanism. The consistency check unit is used to compare the annotation results generated by multiple large language models and identify the differences between the annotation results. Anomaly detection unit is used to detect outliers in the generated annotation results in order to identify abnormal annotation results; The quality feedback unit is used to feed back the layer-by-layer quality inspection results output by the multi-level quality inspection unit, the annotation differences output by the consistency check unit, and the abnormal annotation results output by the anomaly detection unit to the large language model, so as to fine-tune the large language model.

2. The data annotation system based on a large language model according to claim 1, characterized in that, The data access layer includes: A multimodal data acquisition unit is used to collect labeled data from multiple heterogeneous data sources based on a unified data access interface; The data preprocessing unit is used to process the collected data to be labeled by removing missing values, outliers and noisy data, and to fuse and align the data to be labeled from different modalities. Data standardization units are used to convert preprocessed data to be labeled into a unified data format in order to construct standardized data.

3. The data annotation system based on a large language model according to claim 1, characterized in that, The intelligent annotation engine layer also includes: An interactive annotation interface is provided to offer various annotation tools and display the annotation results generated by the large language model in real time; The annotation rule engine unit is used to preset annotation rules for various scenarios to form an annotation rule template library; The active learning unit is used to identify data samples with high uncertainty in model predictions and prioritize their transfer to human annotation.

4. The data annotation system based on a large language model according to claim 1, characterized in that, It also includes a system monitoring layer, which includes: The performance monitoring unit is used to track various operational metrics across the entire value chain in real time. The log management unit is used to record events that occur in each operation of the entire chain, and to store all recorded events in a hierarchical and categorized manner according to log level; The security audit unit is used for access control based on identity authentication information and for setting secondary verification rules for sensitive operations. The alarm notification unit is used to generate alarms based on preset multi-level alarm rules and notify operation and maintenance personnel of the alarms through multiple channels.

5. The data annotation system based on a large language model according to claim 1, characterized in that, It also includes a workflow management layer, which includes: The task allocation unit is used to allocate tasks according to their difficulty, data type, and urgency, and to prioritize tasks based on a task priority queue. The progress tracking unit is used to display the progress of each labeled task in real time and to visualize the task progress in the form of statistical reports and charts. The resource scheduling unit is used to dynamically allocate computing resources based on the length and priority of the labeling task; The version control unit is used to create a version number for each piece of data to be labeled and the labeling result, and to record version change data.

6. A data annotation method based on a large language model, wherein the method is applied to the data annotation system based on a large language model as described in any one of claims 1-5, characterized in that, The method includes: Collect the data to be labeled and preprocess the data to be labeled to convert it into a uniform data format; The system retrieves contextual information from the vertical domain for the received data to be labeled in a unified data format, and then concatenates the retrieved contextual information with the data to be labeled before inputting it into multiple pre-trained large language models; the large language models are used to generate the labeling results for the data to be labeled. The quality of the annotation results generated by multiple large language models is monitored, and feedback analysis is performed based on the annotation quality issues detected.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in claim 6.

8. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method as described in claim 6.

Images