Multi-modal labeling method based on deep learning

Abstract

The invention discloses a multi-modal labeling method based on deep learning, and the method comprises the steps: 1), labeling multi-modal data through a multi-modal labeling module, and obtaining a labeled data set; 2) training a deep learning model by using the data set, and adjusting the weight and parameters of the deep learning model through a back propagation algorithm and an optimizer, so that the deep learning model gradually learns the features and rules of the annotated content; and 3) inputting a piece of to-be-labeled multi-modal data into the trained deep learning model for reasoning and prediction, and generating a labeling result of the to-be-labeled multi-modal data. By means of the multi-modal data annotation method and device, efficient and accurate annotation can be conducted on data including images, texts and other modalities, and therefore an efficient and reliable solution is provided for annotation of the multi-modal data.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a system and method for multimodal data annotation, which uses a target detection algorithm of a deep learning model to speed up the efficiency of annotators in annotating data to be annotated. Background Art

[0002] Multimodal data annotation is an important research area in computer vision and data science. It involves assigning semantic labels, metadata or annotations to data containing multiple types of information such as text, images, audio, video, etc. These annotations enable computer systems to understand and process this information. The goal of multimodal data annotation is to associate data with semantic information, such as classification labels, descriptive text, sentiment sentiment polarity, etc., so that computer systems can better understand and analyze this data. This annotation is usually done by human annotators, who add appropriate labels and annotations to the data based on the content and characteristics of the data, but manual annotation comes with high costs, time consumption, difficulty in data integration, consistency and standardization, etc. The importance of multimodal data annotation lies in improving the understanding ability of machines. When computer systems can understand data of different modalities, they can better process and analyze this data, thereby improving performance in various applications.

[0003] With the rapid development of digital technology, the acquisition and application of multimodal data has become a common situation in real life and various industries. These multimodal data usually include information in different forms such as images and texts. These information are related to each other. The comprehensive use of this information can help achieve more accurate and comprehensive data analysis, identification and application. However, the labeling of multimodal data faces many challenges. Traditional labeling methods usually require a lot of manpower input, and subjective errors are prone to occur during the labeling process, making it difficult to guarantee labeling efficiency and accuracy. In addition, the complex correlation between different modal data also increases the complexity of the labeling task.

[0004] Existing technologies often use manual annotation. Even if automated annotation tools are used to a certain extent, there are still problems such as low annotation efficiency and poor accuracy. Therefore, there is an urgent need for a method and system that can efficiently and accurately complete multimodal data annotation. Providing an automated, efficient, and accurate multimodal data annotation system and method has become an urgent task to meet the growing demand for multimodal data annotation and promote the application and development of multimodal data in various fields.

[0005] The patent publication number CN113535949A discloses a multimodal joint event detection method based on images and sentences. To identify events from images and sentences at the same time, the existing unimodal data sets can be used to learn image and text event classifiers respectively; on the other hand, the image sentence matching module is trained using the existing image and title pairs to find the images and sentences with the highest semantic similarity in the multimodal articles, thereby obtaining the feature representation of image entities and words in the public space. Finally, the model is tested using a small amount of multimodal annotated data, and the events and their types described by the images and sentences are obtained using the shared event classifier.

[0006] The patent with publication number CN115311512A discloses a data annotation method, device, equipment and storage medium. The data set to be annotated is input into a preset automatic annotation model to obtain a first data set; the annotated data of the same frame in the first data set are fused and completed to obtain a second data set; the annotated data with redundant annotations in the second data set is determined, and the second data set is cross-validated and integrated according to the annotated data with redundant annotations to obtain a completed data set. Point cloud and image data are obtained through a collector, multimodal annotations are used to cross-validate and enrich semantic information, and multimodal annotations are fused.

[0007] The patent with publication number CN115937738A discloses a training method, device, equipment and storage medium for a video annotation model. The method includes: obtaining video data, extracting key frames from the video data; extracting features from frames to obtain feature data of frames in different modes; constructing subgraphs corresponding to different modes based on feature data in different modes; performing aggregation operations on subgraphs corresponding to different modes based on graph neural networks to obtain target graphs; obtaining predicted annotation results of key frames by graph neural networks based on the target graphs; training the graph neural network until convergence based on the predicted annotation results and the actual annotation results of key frames to obtain a video annotation model.

[0008] Comprehensively considering the existing multimodal annotation tools, the development of intelligent models is relatively slow. Most tools can only annotate single files such as text, pictures or videos. The file processing process is also very complicated, and the annotation form is also very simple. It is impossible to add information such as the relationship between annotation labels. Annotation efficiency is also an important evaluation indicator of a tool. The development of intelligent annotation with existing tools is relatively limited. Adding deep learning models can largely automate annotation, improve accuracy, perform multimodal fusion, and enhance real-time and adaptability.

[0009] Different file contents, such as text and images, have completely different requirements when being annotated, and annotation models need to be specially trained for different contents. Based on this situation, the present invention provides a multimodal annotation method based on deep learning, which can directly perform multimodal information annotation on data in various forms, train models, optimize models, and display the annotated information in real time. Summary of the invention

[0010] To solve the above problems, the present invention provides a multimodal annotation method based on deep learning. The present invention uses deep learning technology to efficiently and accurately annotate data including multiple modalities such as images and texts.

[0011] The technical solution of the present invention is:

[0012] A multimodal annotation method based on deep learning, the steps of which include:

[0013] 1) Use the multimodal annotation module to annotate data content:

[0014] 11) Labeling task definition: clearly define the specific content and standards of the labeling task, determine the data type and label system to be labeled, and ensure that the labelers understand the requirements of the labeling task;

[0015] 12) Selection of annotation form: Select an appropriate multimodal annotation form to ensure that the annotation form can meet the requirements of the annotation task.

[0016] 13) Integration of annotation results: Integrate the annotation results into a unified data set to ensure that the organization and format of the data meet the requirements of subsequent tasks.

[0017] 2) Use deep learning models to learn the annotated content:

[0018] 21) Collect annotated data sets, including input data and corresponding annotation information. Ensuring the quality of the data set and the accuracy of the annotations is crucial for the training of deep learning models;

[0019] 22) Select a suitable deep learning model and determine the structure and parameters of the model based on the characteristics of the public dataset arXiv Dataset and the complexity of the annotation task. The arXiv Dataset is a database of 1.7 million articles, including relevant features such as article title, author, category, abstract, PDF full text, etc. The data is stored in JSON format;

[0020] 23) Use the prepared data set to train the deep learning model. During the training process, the weights and parameters of the model are adjusted through the back propagation algorithm and optimizer, so that the model gradually learns the characteristics and rules of the annotation content. In order to improve the model's learning ability of the annotation content when processing multimodal data, the loss function is improved so that it can better capture the relationship and characteristics between each modality. The present invention uses a weighted fusion multimodal loss function:

[0021] L=α·L text +β·L img +γ·L fusion

[0022] Among them, L text is the loss for text modality, L img is the loss of image data, L fusion is the loss of modal fusion, and α, β, and γ are different weight coefficients.

[0023] 24) Use the validation data set to validate the trained model, evaluate the performance and generalization ability of the model, and tune the model based on the validation results to improve the performance of the model;

[0024] 25) Apply the trained deep learning model to new unlabeled data for inference and prediction. The model will automatically label or classify the input data and generate corresponding output results.

[0025] Furthermore, different annotation methods are provided for the annotation of multimodal data forms, such as text, image, audio, etc., which are annotated using different annotation methods, and annotated data sets are constructed by manual annotation.

[0026] Furthermore, it is necessary to select a suitable annotation tool and use this system as an annotation tool to perform data annotation. This system provides annotation methods for different data, such as picture types, which can perform target recognition and classification, key point annotation, image description, region annotation, and attribute annotation.

[0027] Furthermore, the deep learning module trains, predicts and evaluates the annotated data, and performs incremental training on the deep learning model based on the updated data obtained by the annotators modifying the information identified by the intelligent annotation.

[0028] Furthermore, the labeled data is collected and stored in a designated database for the next step of model training. The storage database uses a non-relational database.

[0029] Furthermore, the YOLO algorithm is used on the collected dataset to pre-train the deep learning model, train and tune the model parameters, and the YOLO algorithm is used in combination with the manually annotated labels to train the model to detect objects in the image and identify their categories and attributes.

[0030] Furthermore, the trained deep learning model is applied to integrate YOLO and the annotation model, and the multimodal framework of the YOLO target detection and annotation model is integrated. YOLO and the intelligent annotation model are integrated into a multimodal annotation framework, which can be used to perform multimodal annotation on new unlabeled files. It has the functions of using the YOLO model to detect objects in the image, extracting the features of each detected object, and using the text annotation model to predict the category and attributes of each object. The framework uses the YOLO algorithm for image modality target detection and recognition, and the recognition results are transmitted to the database in JSON format, and the annotation model is displayed on the front end; the text modality directly extracts the annotation entities through the deep learning model, and also saves the data in JSON format.

[0031] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the above method.

[0032] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0033] The present invention is divided into four modules: a multimodal annotation module, a model training module, an intelligent annotation module and an annotation echo module. Figure 1 Demonstrates modules for multimodal annotation methods.

[0034] The multimodal annotation module uses various annotation methods to annotate content, annotate text, tables, and pictures, locate their positions and information, and store the annotated information. The model training module trains the model based on the annotated data in the database, learns and tunes the model parameters. The intelligent annotation module uses the trained model to automatically annotate unannotated files and stores the annotation results in the database. The annotation echo module displays the annotation results at the corresponding position of the document based on the model output results.

[0035] The four modules of the present invention are described in detail below.

[0036] 1. Multimodal Annotation Module

[0037] The multimodal annotation module needs to complete the annotation of various data formats in the document, and different data types should be distinguished. First, data collection is carried out, and the parts of the document content that should be annotated are described in detail.

[0038] Define the annotation scheme and determine the type and format of annotation. Text annotation types include: classification, assigning text to predefined categories. Entity recognition: identifying specific types of entities in the text, such as names, places, or organizations. Relationship extraction: identifying the relationship between entities in the text. Then use the annotation module to annotate the text. For pictures and other forms of data, the annotation scheme is similar, but the form of annotation needs to be changed accordingly. Pictures, as a visual medium containing digital data, cannot be directly expressed intuitively using text. We stipulate that the multimodal annotation module uses coordinate selection to annotate and can be associated with relationships.

[0039] Classification annotation is a text data annotation technology that divides text data into different categories or labels and assigns a label to each category. To assign a corresponding category label to each text data, the annotator needs to classify the text according to a predefined classification system. The storage of classification annotations is usually relatively simple, and only the text content and text labels need to be stored, and other information can be added according to the actual scenario.

[0040] Bounding box annotation is an image annotation technique that involves drawing rectangular boxes around objects in an image and assigning class labels to the objects. It is one of the most commonly used annotation types in computer vision and is used to train object detection and recognition models. Use the selection tools in the annotation module to draw bounding boxes on the image and assign a class label to each bounding box, indicating the class to which the object belongs. Class labels can be predefined or custom. Save the annotations in the desired format.

[0041] The bounding box is usually represented as a quadruple (x, y, w, h),

[0042] in:

[0043] x and y are the coordinates of the upper left corner of the bounding box;

[0044] w and h are the width and height of the bounding box.

[0045] Polygon annotation is an image annotation technique that involves using polygonal shapes to outline objects or areas in an image. It is more sophisticated than bounding box annotation and can more accurately represent the shape of an object. Use the polygon drawing tool in the annotation module to draw bounding boxes on the image and assign a class label to each bounding box, indicating the class to which the object belongs. A polygon can be represented by a set of ordered vertices:

[0046] P=[(x1,y1),(x2,y2),...,(xn,yn)],

[0047] in:

[0048] (xi,yi) is the coordinate of the i-th vertex;

[0049] n is the number of vertices of the polygon.

[0050] As mentioned above, after the original data documents are annotated, the annotated data is stored in the database. The database is responsible for managing and processing data, including operations such as storage, retrieval, update, and deletion. The database spontaneously organizes and indexes data to achieve fast and efficient access. The annotated data is stored in the database using a non-relational database, and each piece of data is stored in the database in JSON format. JSON data is in the form of key-value pairs, which is very convenient to store in a non-relational database. At the same time, converting the data into JSON format when retrieving it makes it easier for downstream modules to use and analyze it, and it also greatly improves the training speed of the model.

[0051] Attached Figure 2 Shows the architecture of the multimodal annotation module.

[0052] 2. Model Training Module

[0053] In order to process multimodal input of image and text data, we can design a joint deep learning model, which contains two branches: image processing branch and text processing branch. These two branches are responsible for processing image and text data respectively, and fuse their features in subsequent layers to achieve joint multimodal processing.

[0054] 1. Image processing branch

[0055] Image feature extractor: Uses convolutional neural network as the base model of the image processing branch to extract features of image data. Pooling layer: Pools the feature maps output by the convolution layer to reduce feature dimensions and improve the robustness of the model. Fully connected layer: Connects the output of the pooling layer to one or more fully connected layers to learn higher-level image feature representations.

[0056] 2. Text processing branch

[0057] Word embedding layer: Represent text data as word embedding vectors, using a pre-trained word embedding model or by training your own word embedding model. Recurrent neural network: Use a recurrent neural network model to process the word embedding vector sequence to capture the semantic information of the text data. Pooling layer or global average pooling: Pool the output of the recurrent neural network to obtain a fixed-length representation of the text data.

[0058] 3. Multimodal Fusion

[0059] Feature fusion layer: The features of the image processing branch and the text processing branch are fused by concatenation, addition, weighted averaging, etc. Fully connected layer: The fused features are input into the fully connected layer to learn the joint representation of multimodal features.

[0060] Model training and optimization methods use training sets with image and text data annotations to train the model, and use loss functions and optimizers to optimize parameters. The model is verified and tuned through the validation set to prevent overfitting and improve the generalization ability of the model. This model can process image and text data at the same time and effectively fuse their features, thereby realizing multi-modal input processing and joint learning.

[0061] Attached Figure 3 This is the overall architecture of the model training module.

[0062] 3. Intelligent Labeling Module

[0063] The intelligent annotation module intelligently annotates the text, pictures and other information in the document that needs to be annotated. Based on the combination of the multimodal annotation module, it uses the trained intelligent model for automatic annotation. The quality of intelligent annotation depends on the accuracy of the model. After the annotation is completed, the annotation results can be manually reviewed and modified.

[0064] Multimodal annotation is performed according to the model, and the entity information and relationship information obtained by annotation are put into the database using a unified format and displayed in the document. Annotation screening is performed according to pre-defined knowledge ontology to realize an automatic annotation tool that is easy for users to use.

[0065] This module can automatically create annotation entities and relationship information, and can also manually add missing entity and relationship information after module processing. Unused annotation entities and relationship information after annotation is completed can be manually deleted.

[0066] Text information annotation uses the Word2Vec model to vectorize the defined entity names and relationship names, and calculates the entity and relationship categories marked in the intelligent annotation model for each name (including entity names and relationship names) based on the cosine similarity of the vectors. The selected entity and relationship names are filtered for output by the intelligent annotation model. The output results of the intelligent annotation module model are entity dictionaries and relationship dictionaries, and the categories required for the annotation project are filtered according to the category names in the dictionary. Generate annotations and output the annotations or descriptions of the text content to the user to assist the user in understanding the text content or performing related applications.

[0067] Multimedia information annotation involves different types of data such as images, videos, audio, and text, and uses a common multimedia information conversion model. First, the image is converted into text, and the image-to-text conversion model is used to convert the content in the image into a text description to assist in image content recognition. Secondly, a convolutional neural network and a recurrent neural network are combined, and a pre-trained convolutional neural network model is used to input the image into the network, convert the image into digital features, and extract the implicit features of the image. The image features extracted by the convolutional neural network are used as the initial hidden state of the recurrent neural network, and then the text sequence is gradually generated through the recurrent neural network.

[0068] Attached Figure 4 This is the overall architecture of the intelligent labeling module.

[0069] 4. Annotation Echo Module

[0070] The entity information and entity relationship information obtained according to the model are put into the database in a unified format and displayed on the document, and the annotation and screening are performed according to the pre-defined knowledge ontology, so that users can use the automatic annotation tool conveniently. This module plays a role in timely feedback and confirmation of the annotation results during the annotation process, and displays the annotation results in an intuitive way, such as drawing a bounding box on the image, displaying the annotated part in the text, etc., so that users can clearly see the annotated content.

[0071] 1. The user establishes the annotation project ontology and relationships in the annotation tool, including the entity categories that need to be annotated in the annotation project and the relationship categories between entities.

[0072] 2. Use the Word2Vec model to vectorize the user-defined entity names and relationship names, and calculate the entity and relationship categories marked in the intelligent labeling model corresponding to each name based on the cosine similarity of the vectors.

[0073] 3. Filter the output of the intelligent annotation model according to the entity and relationship names selected in step 2. The output results of the intelligent annotation module model are entity dictionaries and relationship dictionaries. The categories required for the annotation project are filtered out according to the category names in the dictionaries.

[0074] 4. Position the selected entities and relationships according to the document's text and text coordinate information dictionary, and locate them at the coordinates on the document.

[0075] 5. Establish an intelligent annotation layer on the original document, construct an annotation box based on the coordinates, and mark out the entity categories and relationship categories.

[0076] Attached Figure 5 This is the overall architecture of the annotation echo module.

[0077] Compared with the prior art, the present invention has the following positive effects:

[0078] The present invention realizes automatic labeling through deep learning technology. Compared with traditional labeling methods, the system and method of the present invention have the advantages of high labeling efficiency and high accuracy. It can be widely used in text labeling, image labeling, knowledge graph construction and other fields, and provides an efficient and reliable solution for the labeling of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 Block diagram for the multimodal annotation method.

[0080] Figure 2 Architecture diagram of the multimodal annotation module.

[0081] Figure 3 This is the overall architecture diagram of the model training module.

[0082] Figure 4 This is the overall architecture diagram of the intelligent labeling module.

[0083] Figure 5 This is the overall architecture diagram of the label echo module.

[0084] Figure 6 This is the overall architecture diagram of the general training model. DETAILED DESCRIPTION

[0085] The details of the present invention are described in detail below.

[0086] 1. Multimodal Data Conversion

[0087] 1. Text data conversion

[0088] Text vectorization uses the word embedding model Word2Vec to convert text into vector representation. As a deep learning-driven word embedding algorithm, Word2Vec successfully captures the semantic relationship and contextual information between words by mapping them to a low-dimensional real vector space, providing a powerful and flexible tool for subsequent text analysis tasks. Word2Vec mainly implements word embedding through two models: Continuous Bag of Words (CBOW) and Skip-Gram. CBOW: predicts a word itself based on its contextual vocabulary, emphasizing the aggregate semantics of the word. Skip-Gram: given a word, predicts the contextual vocabulary around it, focusing on the diffusion semantics of the word.

[0089] The computational method used by the model in text sequences uses the basic solution framework of statistical language models.

[0090] S = w1, w2, w3, ..., w T

[0091] Its probability can be expressed as:

[0092]

[0093] That is, the joint probability of the sequence is converted into the product of a series of conditional probabilities. The question becomes how to predict the conditional probabilities of these given words:

[0094] p(w t |w1,w2,...,w t-1 )

[0095] Due to its huge parameter space, such a primitive model wastes a lot of resources in practice. We chose a method that reduces the error and greatly reduces the waste of resources:

[0096] p(w t |w1,w2,...,w t-1 )≈p(w t |w t-n+1 , ..., w t-1 )

[0097] The input layer is the One-Hot encoded word vector of the context word, V is the number of words in the vocabulary, N is the dimension of the custom word vector, and C is the number of context words. Initialize a weight matrix W v×N , multiply the matrix by the left of all the input One-Hot encoded word vectors to obtain vectors w1, w2, w3, w with a dimension of N c , N is set according to the task requirements. The resulting vectors w1, w2, w3, w c Add and average as the hidden layer vector h. Initialize another weight matrix W' N×V , multiply this matrix by the hidden layer vector h, and then process it through the activation function to get the V-dimensional vector y, where each element of y represents the probability distribution of each corresponding word.

[0098] 2. Image data conversion

[0099] The objects in the image not only have visual features, but also contain semantic information. Converting them into text data can provide additional contextual information for the object detection model, which helps to more accurately identify and locate the object. Through cross-modal fusion, the model can learn the relationship between images and text, thereby improving the performance of object detection.

[0100] In the YOLO target detection algorithm, the purpose of normalizing the input layer data is to map the pixel value range of the input image to a smaller range, usually between 0 and 1. Normalization can ensure that the input data is within the same scale range and avoid excessive differences in pixel values ​​between different images. It helps the network model to better learn image features and improve the stability and convergence speed of the model.

[0101] Calculate the mean and variance, normalize and calculate the mean and variance of the input tensor in a specific dimension. Assume that the shape of the input tensor X is [N, D] (N is the batch size and D is the feature dimension), and calculate the mean μ and variance σ for each sample 2

[0102]

[0103]

[0104] Normalization: normalize each feature of each sample to make its mean 0 and variance 1

[0105]

[0106] Among them, ε is a very small number used to prevent division by zero. Scaling and offset, scaling and offset operations are performed on the normalized features, which are completed through the learned parameters γ (scaling) and β (offset)

[0107] y i，d =γx′ i，d +β

[0108] γ and β are trainable parameters whose dimensions are the same as the dimensions of the input features.

[0109] 2. General Model Training

[0110] YOLO is an object detection algorithm that can process data of different modalities at the same time. We have made some changes to YOLO so that it can better detect multi-modal data and obtain good model performance by training it on manually annotated data sets. Its model structure mainly includes the following components:

[0111] 1. Multi-channel input: Image data mainly consists of Mosaic image enhancement, adaptive anchor box calculation, and adaptive image scaling. Mosaic image enhancement is a commonly used data enhancement method in target detection. It can generate new training images by combining multiple different images. Randomly select four different images, reset the long side of the image, and keep the image ratio unchanged to generate a large image. Randomly generate a center point in the large image. Based on the center point in the large image, splice the four images together, perform M matrix transformation on the spliced ​​image, crop based on the upper left corner of the large image, and transform the label coordinates of each independent image into label coordinates based on the large image. After M matrix transformation, the final coordinates are obtained. Finally, the synthesized image and the corresponding label box are used for model training.

[0112] Mosaic data enhancement helps improve the robustness of the model because it requires the model to be able to handle overlaps and background information between different objects. This data enhancement technique is often used in object detection tasks in the YOLO series to improve the performance and generalization ability of the model.

[0113] For text data processing, use a pre-trained word embedding model or train your own word embedding model to represent text data as a sequence of word embedding vectors. Use a recurrent neural network model to process the word embedding vector sequence to capture the semantic information of the text data. Generate a subword sequence for each word in each sentence, and the model predicts a named entity tag for each subword in the sentence. The word segmentation result is saved as a subword list with a length equal to the number of words in the sentence. Then perform entity recognition to extract the named entities and their categories in the sentence, and save the name, length, offset in the sentence, entity type, and original sentence of each entity extracted in the sentence in the form of a dictionary.

[0114] 2. Backbone layer: The image operation of the Backbone layer consists of the Focus structure and the CSP structure, and the CLIP (Constrastive Language-Image Pre-training) model structure is added to match the relationship between the image and the text.

[0115] The Focus structure in YOLO is a convolutional neural network layer for feature extraction, which is used to compress and combine the information in the input feature map to extract higher-level feature representations. The Focus structure is a special convolution operation in YOLO, which is used as the first convolution layer in the network to downsample the input feature map to reduce the amount of calculation and parameters.

[0116] The Focus structure can divide the input feature into four sub-graphs and concatenate the four sub-graphs to obtain a smaller feature graph. Assuming the size of the input feature graph is N×N×C, where N is the size of the feature graph and C is the number of channels, the calculation process of the Focus structure is: separate the input feature graph into channels to obtain two sub-graphs of size The feature map is recorded as x and y; x and y are convolved with a step size of 2 in the horizontal and vertical directions respectively, and two sizes are obtained. The feature map is recorded as x' and y'; x' and y' are concatenated to obtain a size of The feature map is recorded as z; the horizontal and vertical convolution operations with a step size of 2 are performed on z to obtain a size of The feature map is the output of the Focus structure.

[0117] The CSP (Cross Stage Partial) structure is an important component of YOLO. The CSP structure can effectively reduce network parameters and computational complexity while improving the efficiency of feature extraction. The core idea of ​​the CSP structure is to divide the input features into two parts, one part is processed by a small convolutional network, and the other part is directly processed by the next layer. The two parts of the feature map are then spliced ​​together as the input of the Neck network of the next layer.

[0118] The CSP structure includes: dividing the input feature map into two parts, one part is processed by the sub-network, and the other part is directly processed by the next layer; in the sub-network, a convolution layer is first used to compress the input feature map, and then a series of convolution operations are performed, and finally a convolution layer is used to expand it. In this way, relatively few high-level features can be extracted; in the next layer of the Neck network, the feature map processed by the sub-network is spliced ​​with the directly processed feature map, and then a series of convolution operations are performed. In this way, low-level detail features and high-level abstract features can be combined to improve the efficiency of feature extraction.

[0119] The CLIP structure is added to match the relationship between image and text, and an automatic labeling method is introduced to generate region-text pairs, instead of directly using image-text pairs for pre-training, to detect regions on the image and identify different objects and their locations in the image. Specifically, the labeling method includes three steps: (1) Noun phrase extraction: We first use the n-gram algorithm to extract noun phrases from the text; (2) Pseudo-labeling: A pre-trained open vocabulary detector is used. The vocabulary detector contains multiple categories and is mainly used to compare the results. A pseudo box is generated for a given noun phrase of each image, thereby providing a rough region-text pair. (3) Filtering: Use the pre-trained CLIP to evaluate the relevance of image-text pairs and region-text pairs, and filter out pseudo annotations and images with low relevance. We further filter redundant bounding boxes by combining methods such as non-maximum suppression (NMS).

[0120] The Neck network aggregates feature maps of different scales through top-down and bottom-up paths to create a rich multi-scale feature pyramid. The purpose is to allow features at each level to have both high semantic information and accurate spatial location information, thereby improving the detection ability of small objects. The purpose is to process multi-scale targets and improve the detection ability of small and large targets. The network mainly performs a series of network layers that mix and combine image features and pass the image features to the output.

[0121] 3. Multi-channel output: The image data is mainly used in the output of YOLO as a prediction box. Each prediction box consists of confidence, category probability and bounding box position. The output of YOLO can output information such as the location, size and category of the target in the image for subsequent target recognition and tracking tasks.

[0122] The bounding box determination loss function in YOLO uses the IoU loss (Intersection over Union) function, which is mainly used to measure the difference between the predicted bounding box and the true bounding box. IoU loss is a variation of Intersection over Union (IoU), which is a metric used to measure the degree of overlap between the predicted bounding box and the true bounding box. In object detection, IoU is often used to evaluate the overlap between the predicted box and the true box to determine whether the predicted box is correct.

[0123] IoU is expressed as the intersection-over-union ratio of the real box and the predicted box in target detection, and is defined as follows:

[0124]

[0125] Where A and B represent the predicted bounding box and the true bounding box respectively. This formula calculates the ratio of the intersection area to the union area of ​​the two bounding boxes. When the true box and the predicted box completely overlap, the IoU is 1, so the bounding box regression loss function in object detection is defined as:

[0126] LossIoU=1-IoU

[0127] Specifically, for each predicted bounding box, we calculate its IoU value with all true bounding boxes, and then select the true bounding box with the largest IoU as its corresponding matching target, thereby calculating its IoU loss.

[0128] In the target detection task, an object may be detected by multiple prediction boxes. In order to avoid multiple detections of the same object, it is necessary to filter the repeated prediction boxes. This process is called non-maximum suppression (NMS). After outputting the results, YOLO will perform NMS processing on the overlapping target boxes to obtain the final detection results. NMS will compare the IoU value of the remaining prediction boxes except the retained prediction boxes with the prediction box with the highest confidence in the same category. If the IoU value of a prediction box and the selected prediction box exceeds the preset threshold, then the prediction box will be suppressed, that is, it is not considered a valid detection result.

[0129] NMS is mainly done by sorting all prediction boxes from high to low according to the confidence, and then starting from the prediction box with the highest confidence, traversing each prediction box in turn to determine whether the IOU value between the prediction box and all subsequent prediction boxes is greater than a certain threshold. If the IOU value is greater than the threshold, the prediction box is removed from the candidate box list, otherwise the prediction box is retained. Continue to traverse the next prediction box and repeat the above steps until all prediction boxes have been traversed. In the end, the prediction box that remains is the result after NMS processing, that is, each object corresponds to only one prediction box.

[0130] The text data is output probabilistically at the output end. After the probability prediction in the network, the recognition result with the highest probability is selected for output, and the prediction result is stored in the database in the form of a dictionary. The maximum probability entity information predicted by the named entity model is linked to the corresponding entity number in the entity dictionary in the database.

[0131] Attached Figure 6 The overall architecture of the general training model.

[0132] 3. Intelligent annotation display

[0133] The core function of intelligent annotation is to intelligently annotate various multimodal data in documents to improve the understanding and utilization efficiency of information. The design concept of this module is derived from the in-depth research and practice of multimodal data processing. It combines the multimodal annotation technology of the YOLO model to build an intelligent annotation system that can automatically identify entities in documents and analyze the relationship between entities.

[0134] First of all, the foundation comes from various data sets stored in the database, which contain a large number of manually annotated documents, providing sufficient data support for model training. On the data set, different types of data are converted in different forms, and then can be input into the model for training, and then the model is used for prediction.

[0135] For text data, the BIO sequence annotation rule is used. This rule provides a clear logical framework for entity annotation. The first word of the entity is annotated as B (Begin), non-first words are annotated as I (Inside), and non-entity words are annotated as O (Outside). In this way, each training sentence is represented as a sequence consisting of a BX label, an IX label, and an O label, where X represents the category of the entity. This annotation method is not only conducive to the model's understanding of the entity structure in the text, but also provides a strong training foundation for subsequent named entity recognition.

[0136] For image data, the target bounding box is usually represented by a rectangular box that surrounds the target object and is used to identify the location and size of the target. It is represented by the coordinates of the upper left corner of the rectangle (x1, y1) and the coordinates of the lower right corner (x2, y2). In addition, the center coordinates of the bounding box rectangle (x c ,y c ) and width and height (w,h). Through target bounding box annotation, the model can learn how to accurately locate targets in images and identify and classify them.

[0137] During the training phase, the YOLO model was used. The model performs well in processing various types of data, can effectively utilize the information of large-scale data, and improve the performance of the model in multimodal labeling tasks. The specific training process will be described in detail in the implementation method section of the manual, including the model architecture design, hyperparameter selection, and training strategy formulation.

[0138] After the model training is completed, the intelligent annotation module can intelligently annotate new documents. The annotation results will be stored in the database and displayed in the annotated document. Through the annotation function of the model, the annotation efficiency of various types of data in the document can be improved. After the annotation is completed, the existing deviation data can be manually corrected, and the accuracy of subsequent unlabeled data can be improved. In general, the design and implementation of the intelligent annotation module is based on in-depth mining and understanding of the data, providing users with efficient and accurate document annotation services.

[0139] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A multimodal annotation method based on deep learning, comprising the following steps: 1) Use the multimodal annotation module to annotate the multimodal data to obtain annotated data sets; 2) using the data set to train a deep learning model, and adjusting the weights and parameters of the deep learning model through a back propagation algorithm and an optimizer, so that the deep learning model gradually learns the characteristics and rules of the annotation content; 3) For a multimodal data to be labeled, it is input into the trained deep learning model for reasoning and prediction to generate the labeling result of the multimodal data to be labeled.

2. The method according to claim 1, characterized in that The method of annotating multimodal data using a multimodal annotation module is as follows: first, according to the specific content and standard of the set annotation task, determine the data type and label system to be annotated; then, according to the annotation task, select a multimodal annotation form, annotate the multimodal data, and obtain the data set.

3. The method according to claim 2, characterized in that The YOLO algorithm is used to train a deep learning model using the dataset.

4. The method according to claim 3, characterized in that The YOLO algorithm includes a multi-channel input end, a Backbone layer and a multi-channel output end; the Backbone layer includes a Focus structure, a CSP structure and a CLIP structure; wherein, for each annotated multimodal data in the data set, the training method is: 21) The multi-channel input end represents the text data in the multimodal data as a word embedding vector sequence, uses a recurrent neural network model to capture the semantic information of the text data from the word embedding vector sequence and predicts a named entity label for each subword of each sentence in the text data; the multi-channel input end sequentially performs Mosaic image enhancement, adaptive anchor frame calculation and adaptive image scaling on the image data in the multimodal data to generate a synthetic image and its label frame, and then executes step 22); 22) Using the Focus structure to extract low-level features of the synthesized image and input them into the CSP structure; 24) The CSP structure divides the input low-level features into two parts, processes one part through a convolutional network to obtain high-level features, and then splices the high-level features with the other part of the low-level features and inputs them into the Neck network; the Neck network extracts the input spliced ​​features and inputs them into the CLIP structure; 25) The CLIP structure generates region-text pairs according to input features and text data in the multimodal data, and the method for generating the region-text pairs is as follows: 251) extracting noun phrases from the text data using an n-gram algorithm; 252) using an open vocabulary detector to generate pseudo boxes for each given noun phrase to obtain image-text pairs; 253) evaluating the correlation between image-text pairs and region-text pairs, and filtering pseudo annotations and images whose correlation is lower than a set threshold to obtain prediction boxes for the multimodal data; 26) The multi-channel output end uses a non-maximum suppression method to process overlapping prediction boxes in the prediction boxes of the multimodal data to obtain a final detection result.

5. The method according to claim 1, 2 or 3, characterized in that: The loss function in the optimizer is LossIoU=1-IoU; IoU is the intersection-over-union ratio of the real box marked in a multimodal data and the predicted box of the multimodal data.

6. The method according to claim 1, 2 or 3, characterized in that: The structure and parameters of the deep learning model are determined according to the characteristics of the multimodal data and the labeling task.

7. The method according to claim 1, 2 or 3, characterized in that: The multimodal data includes images and texts.

8. A server, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images