A Visual Annotation Method for Text and Image Data

Abstract

This invention proposes a visual annotation method for text and image data. By annotating entity types, entity relationship types, and entity attributes in the text, and adding visual annotations to image data, it improves the accuracy and performance of the information extraction model for knowledge graph construction, while further realizing the annotation of image data, thus solving the problem of not being able to meet the application scenarios of multimodal knowledge graph construction.

Description

Technical Field

[0001] This invention relates to the field of text and image annotation corpus technology in the military field, specifically to a visual annotation method for text and image data, and further to functions such as classification management, visual data annotation, intelligent pre-annotation, and annotation result statistics. It can provide accurate learning samples for tasks such as data labeling, knowledge graph construction, event extraction, and target recognition in the field of artificial intelligence, and can be used for training, testing, and verification evaluation of the performance of deep learning models in the field of text and image data processing. Background Technology

[0002] With the rapid development of artificial intelligence technology, data annotation, as an important data processing method, is gradually being widely applied. Data annotation involves manually or semi-automatically annotating or labeling data to help algorithmic models better understand and process it. Data annotation has always been a crucial step in training supervised learning algorithms because it provides accurate outputs so that machine learning algorithms can gradually adjust and improve their predictive accuracy. The significance and main functions of data annotation are mainly reflected in the following aspects:

[0003] (1) Providing a training dataset. Data labeling is the foundation for building a training dataset. When training an artificial intelligence model, a large amount of labeled data is needed as training samples. Through data labeling, the correct label can be added to each sample in the dataset, thus providing the model with a reference for learning.

[0004] (2) Improve model performance. Data labeling can improve the performance of machine learning models. By adding labels to samples in a dataset, the model can better understand and learn the features of the data during training. Labeled data can help the model identify key information, gradually adjust and improve prediction accuracy and overall model performance.

[0005] (3) Promoting the development of intelligent applications. Data annotation is a crucial step in developing intelligent applications. Whether it's image recognition, speech recognition, or natural language processing, a large amount of labeled data is needed to train the model. Through data annotation, sufficient training data can be provided for intelligent applications, thereby improving the application's recognition and understanding capabilities and meeting user needs.

[0006] Data annotation has significant practical implications in the field of artificial intelligence. Through data annotation, training datasets can be provided, model performance can be improved, algorithm research and development can be promoted, the development of intelligent applications can be supported, and industry development and innovation can be fostered.

[0007] Existing annotation methods mainly have the following two problems:

[0008] (1) Existing annotation methods often focus on covering all data types and annotation tasks, or only provide annotation functions for specific data types in a specific domain. This not only fails to meet the application scenarios of multimodal knowledge graph construction, but also requires data annotation in the field of knowledge graph construction to annotate entity types, entity relationship types and entity attributes in the text based on the knowledge graph meta-model.

[0009] (2) Existing data standard methods lack user experience, which seriously affects the efficiency and quality of data annotation. Data annotation results are often displayed in the form of tables, etc., and cannot be visualized on the original text in a way that combines background color, labels and lines.

[0010] (3) Existing annotation methods often require a large amount of training data to improve the model’s recognition performance and accuracy. They have strong domain characteristics and low efficiency and poor effect in cross-domain recognition.

[0011] (4) It is not bound to the domain ontology, and the labeled data cannot be directly applied to the training of the knowledge graph information extraction model. Additional data format and labeling system conversions are required.

[0012] (5) Existing annotation methods lack the ability to pre-annotate based on multimodal large models in the military field. The annotation of elements such as equipment, institutions, countries, regulations, and geographic information in the military field, including entities, attributes, relationships, and events, requires a lot of manpower and time. Summary of the Invention

[0013] The purpose of this invention is to improve the accuracy of entity and relationship recognition in knowledge graphs, enhance data standardization efficiency, and enable intelligent data annotation across domains (military domain).

[0014] To address the aforementioned technical problems, the present invention provides a method for visual annotation of text and image data, comprising the following steps:

[0015] Acquire the corpus, determine whether the corpus is an image, and if it is an image, convert it into text. Proofread the text corpus, remove markers, and extract descriptive information from the corpus.

[0016] A text segmentation model based on sentence-level sequence labeling is used to segment the corpus description information into sentences and add special tags to obtain character sequences. The embedding vector, position vector and segment vector are obtained through a character embedding layer, and the element-wise summation is performed to obtain the final character vector. The final character vector is mapped to each sentence through a BERT encoder. The K character output vectors corresponding to each sentence are processed by average pooling to obtain the final sentence vector. Each sentence is encoded and mapped through the output layer and softmax layer to classify whether each sentence is a paragraph boundary, so as to realize paragraph analysis.

[0017] Based on a pre-established terminology and domain vocabulary database, the sentences after paragraph analysis are segmented and denoised. The segmented words are weighted according to their importance in the sentence. The weighted words are converted into hash values, and weighted number strings are generated according to the weights of the hash values. Each weighted number string is accumulated to obtain a sequence number string, and dimensionality reduction is performed to obtain the final hash value. Multiple final hash values ​​are compared to determine the differences between sentences in order to achieve duplicate data processing.

[0018] Based on entity extraction, attribute extraction, and event extraction, construct a Prompt template that combines answer-type Prompt and task-type Prompt. Based on entity extraction, establish corresponding entity categories and entities; based on attribute extraction, establish corresponding entity categories, entities, and relationships; based on event extraction, establish corresponding event types, events, argument roles, and arguments.

[0019] The Prompt-based cross-domain few-shot data pre-labeling model includes an input layer, an encoding layer, and a UniLM decoding layer. The schema template is concatenated as the input to the input layer. The positional encoding part in the encoding layer uses RoPE rotational positional encoding. The UniLM decoding layer uses a Seq2Seq attention mask to input bidirectional modeling and output unidirectional modeling to achieve conditional generation.

[0020] The input sequence is divided into a source sequence, a target sequence, and a separator between the source and target sequences. A random mask matrix is ​​generated for the target sequence. Tokens that are not masked in the source and target sequences are selected to predict the tokens that will be masked, in order to perform information integration tasks.

[0021] Inputting a portion of the sentence into a Prompt-based cross-domain few-shot data pre-labeling model, and outputting pre-labeling results according to the task type;

[0022] The quantity, quality, and completion rate of labeled data are statistically analyzed for different tasks, and the results of multiple pre-labeling are presented in the form of knowledge graphs, tables, and charts.

[0023] Preferably, the initial context representation formula in the UniLM decoding layer is as follows:

[0024]

[0025] In the formula, the input sequence is X = {x0, x1, ..., x...} l}, x l This indicates the length of the sequence.

[0026] Preferably, the final representation of the initial context representation after n layers of stacking is as follows:

[0027]

[0028] Preferably, when inputting a portion of the sentence into a Prompt-based cross-domain few-shot data pre-labeling model, the label smoothing loss function with the following formula is used:

[0029]

[0030] In the formula, y i p represents the i-th element of the true label vector. i ε is the probability value of the i-th element predicted by the model, ε is a smoothing coefficient less than 1, and k is the number of categories.

[0031] This invention proposes a visual annotation method for text and image data. By annotating entity types, entity relationship types, and entity attributes in the text, and adding visual annotations to image data, it improves the accuracy and performance of the information extraction model for knowledge graph construction, while further realizing the annotation of image data, thus solving the problem of not being able to meet the application scenarios of multimodal knowledge graph construction. Attached Figure Description

[0032] Figure 1 The principles of data annotation technology;

[0033] Figure 2 A text segmentation model for sentence-level sequence annotation;

[0034] Figure 3 For the data deduplication detection process;

[0035] Figure 4 Enter a template for the Prompt;

[0036] Figure 5 A Prompt-based cross-domain few-shot data pre-labeling model;

[0037] Figure 6 Masking strategy for the UniLM model;

[0038] Figure 7 For the data annotation system architecture;

[0039] Figure 8 This explains the principles of data annotation technology. Detailed Implementation

[0040] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0041] The principle of the visual annotation method for text and image data provided in this embodiment of the invention is as follows: Figure 1 As shown, based on the traditional data annotation process, the data annotation processing and text data pre-annotation stages have been significantly added, greatly improving the data annotation efficiency. At the same time, the data annotation method has strong cross-domain intelligent data annotation performance. The principle of the data annotation process will be introduced below, with a focus on a detailed explanation of the implementation principles of data annotation processing and text data pre-annotation.

[0042] (1) Data collection of annotation

[0043] The annotation data collection function primarily provides annotation corpus management for tasks such as entity and entity relation extraction, including data maintenance and user service functions. Data maintenance mainly includes functions such as corpus entry, proofreading, storage, modification, deletion, removal of tags, and management of corpus descriptive information.

[0044] (2) Annotated data processing

[0045] Corpus annotation processing mainly automates the processing of corpora, including paragraph analysis and duplicate data processing. Paragraph analysis identifies and calculates the content structure of the corpus, enabling a comparative display of the corpus on the annotation interface. Duplicate data processing uses an improved plagiarism detection algorithm to quickly compare the collected corpus with the corpus stored in the system, obtaining results rapidly.

[0046] 1) Text segmentation model based on sentence-level sequence labeling

[0047] Document segmentation is defined as automatically predicting the boundaries of segments (paragraphs or chapters) in a document. Existing document segmentation work mainly focuses on written text, and includes two main categories of methods: unsupervised and supervised. Document segmentation is a task that heavily relies on the information of long text passages. Sentence-by-sentence classification models tend to face performance limitations when utilizing the semantic information of long texts, while hierarchical models suffer from high computational cost and slow inference speed. To address these issues, this paper proposes a text segmentation model based on sentence-level sequence annotation for data such as spoken ASR (Automatic Speech Recognition) manuscripts. The model framework diagram is shown below. Figure 2 As shown.

[0048] ① This model models document segmentation as a sentence-level sequence labeling task. First, the input document is segmented into sentences, and each sentence is segmented by a tokenizer and a special tag is added.

[0049] ②The character sequence after word segmentation obtains an embedding vector through the character embedding layer, and then sums the elements with the position vector and segment vector to obtain the final character vector;

[0050] ③ Next, the character vectors are input into the BERT encoder, and the character vectors output by the encoder are mapped to each sentence. The K character output vectors corresponding to each sentence are then processed by average pooling to obtain the final sentence vector. Finally, each sentence encoding is mapped through the output layer and the softmax layer to classify whether each sentence is a paragraph boundary.

[0051] 2) Duplicate Data Processing

[0052] In addition to text segmentation during data acquisition, a key focus is on deduplication checks to improve the quality of the collected data. The specific data deduplication process is as follows: Figure 3 As shown.

[0053] ① Word segmentation. Word segmentation needs to be able to identify domain-specific terms and concepts. This method improves the recognition effect of domain-specific vocabulary and enhances the accuracy of word segmentation by building a terminology lexicon and writing it into the word segmenter. After denoising based on the word segmentation results, words are weighted according to their importance in the sentence. The higher the importance of a word in the sentence, the greater its weight.

[0054] ② Hashing. A hash algorithm is used to convert each word into a hash value, transforming vocabulary into numbers that a computer can process, thereby enabling similarity calculations and improving the performance of similarity calculations.

[0055] ③ Weighting. Based on the hash result generated in the previous step, a weighted number string needs to be formed according to the weight of each word.

[0056] ④ Merge: Add up the sequence values ​​calculated for each word to get a sequence of numbers.

[0057] ⑥ Dimensionality reduction: Reduce the dimension of the calculation result from the previous step. If each bit is greater than 0, record it as 1, and if it is less than 0, record it as 0, to form the final hash value.

[0058] The above process transforms the problem of calculating the similarity of text content into a numerical calculation problem, where differences between text content are determined by comparing the differences between binary number strings.

[0059] (3) Text data pre-labeling

[0060] Building upon manual annotation capabilities, this paper provides intelligent pre-annotation functionality to quickly complete data annotation, improving efficiency and reducing manual annotation costs. Data pre-annotation can instantly annotate entities, attributes, relationships, and other information in user-uploaded documents. However, it needs to address challenges such as cold start, limited sample size, and cross-domain issues, necessitating a general pre-annotation model capable of pre-annotating basic text information with little or no training. To address this, a Prompt-based cross-domain limited-sample data pre-annotation technique is proposed. This technique utilizes a combined answer-based and task-based Prompt template for optimization: a task prompt is added at the beginning of the text, and a uniformly constructed schema template is added at the end. An example is provided below. Figure 4 As shown.

[0061] In the Prompt template, INS represents the task type, such as entity extraction, attribute extraction, event extraction, etc.; s-type is a pre-constructed schema type, such as fighter jet, ship, personnel, country, etc.; ele_n[un_n] represents an element. Here, ele_n is the text description of the element, and [un_n] is the sentinel token representing that element. For example, in entity recognition tasks, it can represent entity category and entity; in relation extraction tasks, it can represent entity category, entity, and relation; in event extraction tasks, it can represent event type, event, argument role, and argument, respectively. Different elements correspond to different sentinel tokens, thus distinguishing different elements.

[0062] The Prompt-based cross-domain few-shot data pre-labeling model mainly consists of an input layer, an encoding layer, and a UniLM decoding layer. The input layer concatenates task prompts, original text, and a uniformly constructed schema template as input. The encoding layer uses three basic input representations, with RoPE rotational position encoding used for the positional encoding part. The UniLM layer employs a Seq2Seq attention mask method, i.e., bidirectional input modeling and unidirectional output modeling, to achieve conditional generation. The overall structure is as follows: Figure 5 As shown.

[0063] The input layer processes the raw text into a form that the model can understand, a crucial step in information extraction tasks. In the model, a unified schema is first constructed for all training data; this involves uniformly defining and constraining entities, attributes, and relationships across all information extraction subtasks, thus structuring the elements to be extracted. Then, the task description for information extraction, a pre-defined schema template, and the uploaded document are concatenated to generate the input sequence, guiding the model to produce the required structured text.

[0064] The encoding layer employs the Transformer architecture as its basic structure, using three input representations: word embeddings, segment embeddings, and position embeddings. Word embeddings map discrete word symbols to continuous real-valued vectors; segment embeddings represent the relationships between different text segments; and position embeddings represent the relative or absolute positional information of words in a sequence, helping the model capture the sequential relationships and contextual information between words. The model uses RoPE rotational position encoding as its positional embedding, which encodes absolute positions in a rotation matrix and incorporates explicit relative positional dependencies into the self-attention formula. This allows the model to support longer text sequences and achieves improved performance.

[0065] The decoding layer uses the UniLM model based on the Decoder-Only Transformer architecture. Different attention masks are modified to model various language models, including bidirectional, unidirectional, and seq2seq. The UniLM model consists of a multi-layer Transformer encoder. Let the input sequence be X = {x0, x1, ..., x...}. l}, x l This represents the length of the sequence. Inputting X into the first-level Transformer yields the initial context representation as follows:

[0066]

[0067] After N layers of Transformer stacking:

[0068] H n =Transformer n (H n ),n∈[1,N]

[0069] The final representation is obtained:

[0070]

[0071] UniLM uses different mask matrices to determine which tokens the model pays attention to in subsequent computations. Therefore, under the same set of parameters, UniLM can directly change the model's operating mode by altering the masking approach. UniLM models typically use three different masking strategies to adapt to different tasks, specifically as follows: Figure 6 As shown.

[0072] Since sequence-to-sequence prediction divides the input sequence into two parts, a source sequence and a target sequence, with a separator between them, and the source sequence is not masked, while the target sequence is randomly masked with some tokens, the model can use the unmasked tokens in both the source and target sequences to predict the masked tokens. This approach is more suitable for information organization tasks, therefore, the sequence-to-sequence masking method is chosen for modeling.

[0073] A small amount of training data is input into the extraction model based on the Prompt template for training, using the label smoothing loss function, as shown in the following formula:

[0074]

[0075] Among them, y i p represents the i-th element of the true label vector. i This represents the probability value of the i-th element predicted by the model, ε is a smoothing coefficient less than 1, and k is the number of classes. Finally, the pre-labeled results are output based on the task type.

[0076] (4) Statistics of annotation results

[0077] The data annotation results statistics can be performed separately for different tasks, including the quantity, quality, and completion rate of the annotated data. It also supports statistics on the quantity, quality, and completion rate of different tags for the same task. A visualization function for the annotated corpus is provided, allowing the annotation results to be displayed in the form of knowledge graphs, tables, and charts.

[0078] Data annotation system architecture as follows Figure 7 As shown.

[0079] Among them, (1) annotation corpus management

[0080] Corpus management is used to manage corpora for tasks such as classification, entity relation extraction, and tag extraction. It includes data maintenance, automatic corpus processing, and user service functions. Data maintenance mainly includes functions such as corpus entry, proofreading, storage, modification, deletion, format conversion, merging, tag removal, and management of corpus description information. Automatic corpus processing mainly realizes the automated processing of corpora, including word segmentation, annotation, text segmentation, merging, corpus alignment, and tag processing. User service functions in corpus management mainly include querying, retrieval, count statistics, sharing, and downloading.

[0081] (2) Task management

[0082] The annotation task management system includes functions such as task creation, editing, deletion, querying, and task assignment, supporting centralized annotation and multi-person collaborative annotation. Annotation task information includes task name, task description, task type, recommendation model, collaborating annotators, and allocated annotation volume. Annotation task types include entity annotation, semantic relationship annotation, one-way text annotation, multi-way text annotation, and sentiment annotation. Data annotation tasks support multi-person collaborative annotation, allowing users to view annotation progress and review status, and to audit collaborative annotation results. Administrators can assign annotation tasks to different users or teams, specifying the scope and requirements of annotation. The task assignment function ensures the orderly progress of annotation work. A progress tracking function is provided, allowing administrators to view the completion status of annotation tasks in real time. Progress tracking helps managers promptly identify and resolve problems that arise during the annotation process.

[0083] (3) Dataset Management

[0084] The dataset management system provides unified management of labeled corpora, supporting querying, filtering, and exporting of annotation results. Users can choose from various export formats, such as CSV, JSON, and XML. The system provides an export wizard to help users quickly export annotation results. The system automatically saves annotation results, recording the annotation process for each piece of data, including the annotator and time, ensuring the traceability of the annotation process. It supports building labeled datasets by filtering the labeled data records for use in different algorithm model training scenarios.

[0085] (4) Labeling quality inspection

[0086] The annotation quality check supports manual review by auditors to ensure annotation quality. After annotators submit their data, auditors can manually review the annotations according to a set sampling ratio. This allows for item-by-item quality review, rejection of substandard annotations for re-annotation, and a review progress display. Administrators can review the data quality of annotations submitted by regular users. An audit status query function is provided, displaying the number of reviewed and unreviewed data. An annotation corpus audit function is also provided, displaying annotated text paragraphs and allowing for item-by-item and batch audits. It displays the percentage of duplicate annotated text paragraphs and consistency statistics for duplicate annotation results.

[0087] (5) Intelligent pre-labeling

[0088] Building upon manual annotation capabilities, this system offers intelligent pre-annotation, enabling rapid data annotation, improving efficiency, and reducing manual annotation costs. Intelligent pre-annotation refers to the process of generating annotation results using existing algorithms within the system, based on the labels and data learned and trained at the current annotation stage. When selecting a pre-annotation algorithm model, it is crucial to ensure that the algorithm type matches the annotation type of the dataset.

[0089] (6) Statistics of data annotation results

[0090] The data annotation results statistics can be performed separately for different tasks, including the quantity, quality, and completion rate of the annotated data. It also supports statistics on the quantity, quality, and completion rate of different tags for the same task. A visualization function for the annotated corpus is provided, allowing the annotation results to be displayed in the form of knowledge graphs, tables, and charts.

[0091] This invention provides intelligent pre-annotation and collaborative annotation for text and image data of weapons and equipment in the military field. The main steps include ontology management, annotation corpus management, annotation task management, annotation quality management, and annotation result statistics. The process is as follows: Figure 8 As shown.

[0092] (1) Ontology Management Phase

[0093] Before data annotation, users need to use ontology management to create, import, and modify military weapon and equipment ontology, including elements such as entities, relationships, and attributes, to form a conceptually complete, hierarchically distinct, and clearly differentiated weapon and equipment ontology to guide data annotation content.

[0094] (2) Annotated Corpus Management Stage

[0095] Once the ontology is determined, users access text and image data such as military news, white papers, blue papers, and regulations to be annotated using the annotated corpus management system, which includes relational databases, graph databases, and ontology file systems. Then, they use functions such as paragraph analysis and deduplication to process duplicate data, thereby recognizing and processing the paragraph structure, punctuation, and other content of the corpus content, improving the text display effect and annotation efficiency.

[0096] (3) Labeling Task Management

[0097] After data processing, administrators use the annotation task management function to create annotation tasks and choose whether to enable intelligent pre-annotation. This system has a built-in self-developed multimodal large model for the military domain. Without training, intelligent pre-annotation of entities, relationships, attributes, and other elements in the military corpus to be annotated can be performed simply through Prompt settings, improving annotation efficiency. After integrating the multimodal large model for the military domain, administrators assign annotation tasks to team members for system annotation, and administrators can view the annotation progress in real time.

[0098] (4) Labeling quality management

[0099] After the data annotation is completed, the administrator uses the annotation quality management function to review the annotation results. The annotated data that passes the review is used to form training, testing, and evaluation datasets, while the annotated data that fails the review is returned to the team members for modification or re-annotation.

[0100] (5) Statistics of annotation results

[0101] After the annotation task is completed, administrators and users can view the annotation statistics, including task-dimensional statistics, tag-dimensional statistics, and visualized annotation corpora, to intuitively understand the completion status and results of the annotation task.

[0102] The beneficial effects of the embodiments of the present invention are as follows:

[0103] (1) The embodiments of this invention are important components of data analysis platforms such as multimodal knowledge graph platforms, deep learning algorithm platforms, data science platforms, and knowledge middleware platforms. They can access data from professional domain data analysis platforms and complete the annotation of raw corpus data. Data annotation is the foundation of deep learning algorithm training. By providing high-quality and large amounts of training data for unstructured data processing and analysis algorithm models such as text and images, it improves the accuracy and performance of data processing algorithm models and enhances the data analysis platform's capabilities in professional domain data compilation, data knowledge generation, data analysis, and data intelligence applications.

[0104] (2) This embodiment of the invention provides a data pre-annotation function on the basis of traditional manual data annotation. It can automatically annotate the corpus using existing algorithm models in the data annotation system, and then manually modify and improve the annotations to finally complete the data corpus annotation. Simultaneously, the data annotation system supports assigning data annotation tasks to different users, enabling collaborative data annotation work. Compared with traditional data annotation methods, data pre-annotation and collaborative data annotation can greatly improve data annotation efficiency and save data annotation costs. Furthermore, this embodiment of the invention provides functions such as data annotation review, dataset management, and data statistical analysis, further improving the functionality of the data annotation system. The drag-and-drop visual data annotation method greatly enhances the UI interaction experience of data annotation, providing real-time and intuitive display of data annotation content and progress, thereby improving data annotation efficiency and quality.

[0105] Implementation Example 1: Data Labeling Based on Knowledge Graph Metamodel

[0106] (1) Modeling is performed under the metamodel management function module of the knowledge graph platform, and the entity types, relation types and entity attributes of the knowledge graph are designed.

[0107] (2) In the entity and relation annotation module of the data annotation system, entity types and relation types can be displayed and distinguished by different colors;

[0108] (3) Select the leaf node of the classification tree on the left side of the page. The labeled corpora are all attached to the leaf nodes, and the non-leaf nodes are the classifications of the corpora. You can also enter a new labeled corpus and mark the words in the corpus by clicking the entity type label with the mouse. The labeling system adds a background color to the marked words and adds a type label;

[0109] (4) Click the relationship type label with the mouse. First, click on a marked entity, then drag the mouse. The annotation system will display an arc with an arrow, which will then connect to another marked entity, forming a complete line. The starting point of the line is the head entity, and the ending point of the arrow is the tail entity. The name in the middle of the line is the relationship between the head and tail entities. The annotation system can automatically calculate the height based on the annotation status of entities and relationships, and re-format the paragraphs in the annotated corpus according to the calculation results to achieve the best display effect.

[0110] (5) In the entity attribute annotation interface, select a leaf node of the corpus classification tree on the left. The right side of the page will display the detailed content of the corpus and the annotated entities. Select the text corresponding to the attribute value with the mouse, and the annotation system will add a background color and an attribute value label. Select an annotated entity, drag the mouse, and an arrowed arc will appear. As the mouse is dragged, the arc will connect to an annotated attribute value, forming a complete line. The starting position of the line is the entity, the ending position of the arrow is the attribute value, and the name in the middle of the line is the attribute type name. The annotation system can also automatically calculate the display height based on the annotation status of entity attributes and reformat the paragraphs of the annotated corpus to achieve the best display effect.

[0111] Example 2: Image Annotation

[0112] (1) In the image annotation module, the left side of the page displays the image annotation corpus, including the image corpus name, description, and thumbnail. The top of the page displays image annotation tags, the bottom displays the image content, and the right side displays the image annotation toolbar;

[0113] (2) Click on the image label (for example: wheel, track, antenna, etc.) and complete the labeling on the image by dragging the box. After the labeling is completed, the labeling system can color and mark the labeled area on the image and add the corresponding label.

[0114] (3) Right-clicking on the marked area allows you to delete the marked content, including the deletion of color, area, and label;

[0115] (4) The annotation toolbar on the right side of the page provides functions such as zooming in, zooming out, restoring, and deleting images, which facilitates the annotation of image corpora.

[0116] This invention, while providing both manual and automatic annotation functions, focuses on improving the UI interaction of data annotation. It enables manual data annotation through drag-and-drop, and enhances the visualization of annotation results through colors, lines, and labels, thereby improving the efficiency and user experience of the data annotation process. At the same time, modules such as annotation data quality detection, annotation result statistical analysis, and dataset management further improve key aspects of data annotation, optimizing and detecting the data annotation process, and improving the quality and efficiency of annotated data.

[0117] This invention creatively integrates the knowledge graph meta-model into the annotation system, providing data classification labels for data annotation. This ensures that the annotated dataset maintains a high degree of consistency with the knowledge graph meta-model, enabling it to be directly used for knowledge graph construction tasks such as entity extraction, relation extraction, and entity attribute extraction, thereby improving the accuracy and performance of the knowledge graph construction information extraction model.

[0118] This invention improves the data annotation component based on the d3 visualization library. It can distinguish the annotated data content by color according to different data labels, and intuitively display the relationship between data labels through connecting lines. At the same time, it displays the relationship category name on the connecting lines. It supports direct deletion of annotated phrases and relationships by right-clicking the mouse. The improved annotation control makes the data annotation process more intuitive and user-friendly, and indirectly improves the data annotation efficiency.

[0119] This invention provides multiple manual data annotation modes and supports AI annotation based on pre-trained models, reducing the cost of manual annotation. It proposes a cross-domain few-sample data pre-annotation technology based on Prompt, and optimizes it by using a Prompt template that combines answer-type Prompt and task-type Prompt. This achieves good label classification and recognition results in different domains, giving the data annotation system strong cross-domain intelligent data annotation performance.

Claims

1. A method for visual annotation of text and image data, characterized in that, Includes the following steps: Acquire the corpus, determine whether the corpus is an image, and if it is an image, convert it into text. Proofread the text corpus, remove markers, and extract descriptive information from the corpus. A text segmentation model based on sentence-level sequence labeling is used to segment the corpus description information into sentences and add special tags to obtain character sequences. The embedding vector, position vector and segment vector are obtained through a character embedding layer, and the element-wise summation is performed to obtain the final character vector. The final character vector is mapped to each sentence through a BERT encoder. The K character output vectors corresponding to each sentence are processed by average pooling to obtain the final sentence vector. Each sentence is encoded and mapped through the output layer and softmax layer to classify whether each sentence is a paragraph boundary, so as to realize paragraph analysis. Based on a pre-established terminology and domain vocabulary database, the sentences after paragraph analysis are segmented and denoised. The segmented words are weighted according to their importance in the sentence. The weighted words are converted into hash values, and weighted number strings are generated according to the weights of the hash values. Each weighted number string is accumulated to obtain a sequence number string, and dimensionality reduction is performed to obtain the final hash value. Multiple final hash values ​​are compared to determine the differences between sentences in order to achieve duplicate data processing. Based on entity extraction, attribute extraction, and event extraction, construct a Prompt template that combines answer-type Prompt and task-type Prompt. Based on entity extraction, establish corresponding entity categories and entities; based on attribute extraction, establish corresponding entity categories, entities, and relationships; based on event extraction, establish corresponding event types, events, argument roles, and arguments. The Prompt-based cross-domain few-shot data pre-labeling model includes an input layer, an encoding layer, and a UniLM decoding layer. The schema template is concatenated as the input to the input layer. The positional encoding part in the encoding layer uses RoPE rotational positional encoding. The UniLM decoding layer uses a Seq2Seq attention mask to input bidirectional modeling and output unidirectional modeling to achieve conditional generation. The input sequence is divided into a source sequence, a target sequence, and a separator between the source and target sequences. A random mask matrix is ​​generated for the target sequence. Tokens that are not masked in the source and target sequences are selected to predict the tokens that will be masked, in order to perform information integration tasks. Inputting a portion of the sentence into a Prompt-based cross-domain few-shot data pre-labeling model, and outputting pre-labeling results according to the task type; The quantity, quality, and completion rate of labeled data are statistically analyzed for different tasks, and the results of multiple pre-labeling are presented in the form of knowledge graphs, tables, and charts.

2. The method for visualizing and annotating text and image data as described in claim 1, characterized in that, The initial context representation formula in the UniLM decoding layer is as follows: In the formula, the input sequence is X = {x0, x1, ..., x...} l }, x l This indicates the length of the sequence.

3. The method for visualizing and annotating text and image data as described in claim 2, characterized in that, The final representation of the initial context representation after n layers of stacking is as follows:

4. The method for visualizing and annotating text and image data as described in claim 1, characterized in that, When inputting partial sentences into a Prompt-based cross-domain few-shot data pre-labeling model, the following label smoothing loss function is used: In the formula, y i p represents the i-th element of the true label vector. i ε is the probability value of the i-th element predicted by the model, ε is a smoothing coefficient less than 1, and k is the number of categories.

Images