Multi-modal data automatic labeling method based on semantic understanding
By generating fused feature representations using an improved ViLBERT model and Laplacian matrix, the problem of insufficient fusion of image and text features in multimodal data annotation is solved, achieving higher annotation accuracy and consistency, and is applicable to multimodal data annotation and image description generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-01
- Publication Date
- 2026-03-03
AI Technical Summary
Existing multimodal data annotation methods neglect the deep correlation between images and text, making it difficult to effectively capture fine-grained semantic information, resulting in insufficient accuracy and consistency of annotation results.
By employing an improved ViLBERT model, a two-stream network architecture, and a knowledge-aware encoder, and combining multi-scale image feature extraction, deep text semantic modeling, and cross-modal reasoning with Laplacian matrix generation to generate fused feature representations, deep fusion of image and text features is achieved.
It significantly improves the accuracy and consistency of image and text annotation, and the generated annotation results are more accurate compared with standard datasets, enhancing the model's generalization ability and making it suitable for fields such as multimodal data annotation, image description generation, and information retrieval.
Smart Images
Figure CN121600529A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data analysis technology, and in particular to an automatic annotation method for multimodal data based on semantic understanding. Background Technology
[0002] With the continuous development of artificial intelligence technology, multimodal data processing has become one of the important directions of current AI research. Multimodal data involves different types of data, such as images, text, and audio, among which images and text are among the most common multimodal data types. For automatic image and text annotation tasks, traditional methods mainly rely on image feature extraction and natural language processing techniques, generating annotations through predefined rules or shallow learning algorithms. However, these methods generally have certain limitations, especially in semantic understanding and fusion between images and text. Traditional multimodal annotation methods typically use independent models to process image and text data separately, and then combine their results through simple matching or direct mapping. However, this method ignores the deep correlation and mutual influence between images and text, resulting in annotation results that lack accuracy and consistency, making it difficult to meet the needs of practical applications.
[0003] Currently, many multimodal annotation methods primarily rely on traditional convolutional neural networks (CNNs) for image feature extraction and combine them with simple text processing models (such as recurrent neural networks (RNNs) or LSTMs) to generate annotations. However, these methods still suffer from two major drawbacks. First, image feature and text feature processing are performed separately, failing to fully integrate the semantic understanding between images and text, resulting in the model's inability to comprehensively understand the relationship between the two. Second, existing methods largely depend on traditional image recognition techniques and basic text generation methods, making it difficult to handle complex cross-modal semantic reasoning and effectively capture fine-grained semantic information between images and text. For example, detailed features in images and contextual information in text often cannot be fully modeled by a single model, thus affecting the accuracy of annotation. Furthermore, existing methods typically rely on simple similarity metrics (such as Euclidean distance and cosine similarity) to measure the relationship between images and text; however, these metrics often fail to adequately consider complex cross-modal relationships and semantic contextual information, thus limiting the accuracy and effectiveness of annotation generation. Therefore, how to enhance the model's understanding of the semantic relationships between images and text by combining more effective feature fusion methods with knowledge graphs remains a pressing problem that needs to be solved by current technology.
[0004] Therefore, how to provide an automatic annotation method for multimodal data based on semantic understanding is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose an automatic multimodal data annotation method based on semantic understanding. This invention effectively fuses image and text features through an improved ViLBERT model, a two-stream network architecture, and a knowledge-aware encoder, thereby improving the accuracy and consistency of image and text annotation. By employing multi-scale image feature extraction, deep text semantic modeling, and cross-modal reasoning, this method overcomes the limitations of traditional methods in semantic understanding, generating more accurate annotation results that can be compared with standard datasets, ensuring annotation quality. This method has broad application value in fields such as multimodal data annotation, image description generation, and information retrieval.
[0006] An automatic annotation method for multimodal data based on semantic understanding according to an embodiment of the present invention includes the following steps: Step 1: Collect and preprocess raw image and text data from multiple data sources to obtain image and text datasets with a unified structure; Step 2: Input the image dataset into the Inception network to extract the visual features of each image region and generate the corresponding image region feature vector set; Step 3: Convert each word in the text dataset into a word vector representation, and embed the positional information into the text features through positional encoding to obtain a text feature vector set; Step 4: Input the image region feature vector set and the text feature vector set into the improved ViLBERT model, and obtain the image text feature map through a two-stream network architecture and a knowledge-aware encoder; Step 5: Construct a Laplacian matrix based on the image text feature map, and generate a fused feature representation based on the Laplacian matrix; Step 6: Input the fused feature representation into the annotation generation module to generate image annotation results and text annotation results; Step 7: Compare the generated image annotation results and text annotation results with the manually annotated dataset or standard dataset, and generate an annotation report.
[0007] Optionally, the original image data includes static images and dynamic video frames; the original text data includes natural language descriptions, tags, question-and-answer text, and titles of the original images; the preprocessing steps include outlier removal, missing value imputation, timestamp alignment, and normalization of different types of data to obtain image datasets and text datasets with a unified structure.
[0008] Optionally, the step of inputting the image dataset into the Inception network, extracting the visual features of each image region, and generating a corresponding image region feature vector set is specifically as follows: The Inception network extracts features from images at different levels simultaneously at different kernel sizes through multi-scale convolution operations. It also performs non-linear transformations using the ReLU activation function, setting all negative values to zero while keeping positive values unchanged, thus capturing both local and global features of the image. The local features include texture and color, and the global features include object outline and shape; After the convolutional layers, the Inception network fuses the features extracted by convolutional kernels of different sizes and performs max pooling to reduce the spatial dimension of image features. The max pooling operation involves selecting the maximum value of each feature and aggregating each feature value; The Inception network's fully connected layers fuse the features extracted by the convolutional and pooling layers, transforming the features of each image region into a fixed-length set of image region feature vectors. The image region feature vector includes visual information of each region in the image, including the image's color, texture, object shape, and structure.
[0009] Optionally, the step of converting each word in the text dataset into a word vector representation and embedding the positional information into the text features through positional encoding to obtain a text feature vector set is as follows: The WordPiece word segmentation method is used to segment the text dataset into words or subwords, and each word or subword is mapped to a unique word ID through a vocabulary. The steps of the WordPiece word segmentation method are as follows: Remove spaces and punctuation marks from a text dataset; Each word in the text dataset is processed one by one from left to right, and a perfect match is found in the vocabulary. If a perfect match is found, the current word is treated as a single word unit. If a word cannot find a complete match in the vocabulary, the current word is split into multiple sub-word units and the search and matching are performed again. The splitting step is to find the longest sub-word match of the current word and use the longest matching sub-word to split it until all sub-words of the current word can be found in the vocabulary. Based on the word ID of each word, each word is mapped to a fixed-dimensional vector representation to obtain the word vector; Based on the word ID, a cosine function is used to generate a position encoding vector for each word's word vector. The position encoding vector is used to represent the word's order in the sentence. Each word's word vector is combined with its position index in the sentence to generate a unique numerical representation. The word vector of each word is added to the corresponding position encoding vector to generate a text feature vector with position information; By summing up all the text feature vectors, we obtain the text feature vector set.
[0010] Optionally, the step of inputting the image region feature vector set and the text feature vector set into the improved ViLBERT model, and obtaining the image text feature map through a two-stream network architecture and a knowledge-aware encoder, specifically involves: The improved ViLBERT model employs a two-stream network architecture to process image region feature vector sets and text feature vector sets separately. The two-stream network architecture includes a visual stream and a text stream. The visual stream processing steps are as follows: Each image region feature vector in the image region feature vector set is used as a node in the image region feature map to obtain the image node set; Calculate the Euclidean distance between the feature vectors of every two image regions. If the Euclidean distance is less than a preset distance threshold, establish an edge connection between the corresponding nodes to obtain the image edge set. Based on the correspondence between the image node set and the image edge set, construct the image region feature map; The specific steps of the text stream processing are as follows: Each text feature vector in the text feature vector set is used as a node in the text feature map to obtain a text node set; Calculate the Euclidean distance between any two text feature vectors. If the Euclidean distance is less than a preset distance threshold, establish an edge connection between the corresponding nodes to obtain the text edge set. Based on the correspondence between text node sets and text edge sets, construct a text feature map; The image region feature map and text feature map are input into a knowledge-aware encoder to perform cross-modal node matching to obtain the image text feature map. The cross-modal node matching step is as follows: Calculate the Manhattan distance between each pair of nodes in the image region feature map and the text feature map, whereby the Manhattan distance is used to measure the similarity between nodes in the image region feature map and the text feature map; If the Manhattan distance is less than the preset similarity threshold, it indicates that the image region and the text feature are semantically similar. Then, a cross-modal node connection is established between the corresponding image region feature vector and the text feature vector, and the weight value of the connection edge is the corresponding Manhattan distance. Connect all nodes that satisfy cross-modal node matching to obtain the image text feature map.
[0011] Optionally, the step of constructing a Laplacian matrix based on the image text feature map and generating a fused feature representation based on the Laplacian matrix specifically involves: An adjacency matrix is constructed based on the image text feature map. If there is no edge connecting two nodes in the adjacency matrix, the value of the corresponding element in the adjacency matrix is 0. If there is an edge connecting two nodes, then the element value in the corresponding adjacency matrix is the reciprocal of the weight of the connecting edge; Calculate the degree matrix of the image text feature map, where each element in the degree matrix is the degree of the corresponding node, and the degree is the number of connected edges of the node; Subtracting the degree matrix from the adjacency matrix yields the Laplace matrix; The Laplacian matrix is decomposed into eigenvalues to obtain the eigenvectors and eigenvalues of the Laplacian matrix. The eigenvectors represent the fusion features of nodes in the image text feature map, and the eigenvalues represent the similarity strength between nodes. Based on the magnitude of the eigenvalues, the eigenvectors corresponding to the k smallest eigenvalues are selected and weighted and summed to obtain the fused feature representation.
[0012] Optionally, the step of inputting the fused feature representation into the annotation generation module to generate image annotation results and text annotation results specifically involves: The fused feature representations are input into the annotation generation module for image annotation and text annotation tasks, respectively. In the image annotation task, the annotation generation module generates image annotation results by matching image region features with fused feature representations and combining them with image background information, which includes object category, location, and color. In the text annotation task, the annotation generation module compares and fuses the relationship between the feature representation and the image annotation results, and then matches the text annotations with the image regions to generate text annotation results that are related to the image content.
[0013] Optionally, the step of comparing the generated image annotation results and text annotation results with a manually annotated dataset or a standard dataset to generate an annotation report specifically involves: Obtain annotation data relevant to image and text annotation tasks from standard datasets or manually annotated datasets; For image annotation tasks, compare the similarity between the generated text descriptions and the description texts in manually annotated data; For text annotation tasks, the similarity between the generated text annotations and the annotated text in the standard dataset is compared. An annotation report is generated based on the comparison results. The annotation report includes similarity and error analysis. The error analysis is the difference between the annotation results and the standard dataset, including incorrect annotations, missing annotations, and over-annotations.
[0014] The beneficial effects of this invention are: This invention proposes a semantically understanding-based automatic multimodal data annotation method that significantly improves the semantic understanding and fusion capabilities between images and text by combining an improved ViLBERT model, a two-stream network architecture, and a knowledge-aware encoder. Compared to traditional multimodal data annotation methods, this invention effectively solves the challenge of deep semantic understanding between images and text by introducing deeper feature fusion and cross-modal reasoning. Traditional methods often rely on simple image feature extraction and text processing models, neglecting the correlation between images and text, resulting in significant discrepancies in the accuracy and consistency of the generated annotation results. This invention, however, employs the Inception network for multi-scale extraction of image features and combines it with the BERT model for deep semantic modeling of text, ensuring the full fusion of image and text information. Especially with the introduction of the knowledge-aware encoder, image regions and text features are not only matched through a self-attention mechanism but also incorporate structured knowledge from external knowledge graphs, making the model more accurate and comprehensive in understanding the semantic relationships between images and text.
[0015] By constructing an image-text feature map and generating a fused feature representation using the Laplacian matrix, this invention further improves the accuracy of image and text annotation. During the image-text fusion process, eigenvalue decomposition of the Laplacian matrix enables the model to capture the deep-seated relationships between image regions and text features, thus facilitating better cross-modal inference. This innovative fusion method avoids the separation of image and text processing in traditional methods, effectively improving the performance and accuracy of multimodal data annotation tasks. The annotation generation module, by inputting the fused feature representation, generates high-quality image and text annotation results. These results are not only closely related to the actual image content but can also be effectively compared with standard datasets or manually annotated datasets, ensuring the quality and consistency of the annotation results. The application of this invention in image and text annotation tasks not only improves the accuracy of annotation generation but also enhances the model's generalization ability through cross-modal feature fusion, overcoming the problem of insufficient accuracy in annotation generation in existing technologies. This makes this method of significant application value in multimodal data annotation, image description generation, and text annotation, especially demonstrating strong technical advantages and broad application prospects in practical applications such as intelligent image processing, content generation, and information retrieval. Attached Figure Description
[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0017] Figure 1 This is an overall flowchart of a multimodal data automatic annotation method based on semantic understanding proposed in this invention; Figure 2 This is a schematic diagram illustrating the text feature vector generation steps of a multimodal data automatic annotation method based on semantic understanding proposed in this invention. Figure 3 This is a flowchart illustrating the improved ViLBERT model processing steps of the automatic annotation method for multimodal data based on semantic understanding proposed in this invention. Detailed Implementation
[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0019] refer to Figure 1-3 An automatic annotation method for multimodal data based on semantic understanding includes the following steps: Step 1: Collect and preprocess raw image and text data from multiple data sources to obtain image and text datasets with a unified structure; Step 2: Input the image dataset into the Inception network to extract the visual features of each image region and generate the corresponding image region feature vector set; Step 3: Convert each word in the text dataset into a word vector representation, and embed the positional information into the text features through positional encoding to obtain a text feature vector set; Step 4: Input the image region feature vector set and the text feature vector set into the improved ViLBERT model, and obtain the image text feature map through a two-stream network architecture and a knowledge-aware encoder; Step 5: Construct a Laplacian matrix based on the image text feature map, and generate a fused feature representation based on the Laplacian matrix; Step 6: Input the fused feature representation into the annotation generation module to generate image annotation results and text annotation results; Step 7: Compare the generated image annotation results and text annotation results with the manually annotated dataset or standard dataset, and generate an annotation report.
[0020] In this embodiment, the original image data includes static images and dynamic video frames; the original text data includes natural language descriptions, tags, question-and-answer text, and titles of the original images; the preprocessing steps include outlier removal, missing value imputation, timestamp alignment, and normalization of different types of data to obtain image datasets and text datasets with a unified structure.
[0021] In this embodiment, the step of inputting the image dataset into the Inception network, extracting the visual features of each image region, and generating a corresponding image region feature vector set specifically involves: The Inception network extracts features from images at different levels simultaneously at different kernel sizes through multi-scale convolution operations. It also performs non-linear transformations using the ReLU activation function, setting all negative values to zero while keeping positive values unchanged, thus capturing both local and global features of the image. The local features include texture and color, and the global features include object outline and shape; After the convolutional layers, the Inception network fuses the features extracted by convolutional kernels of different sizes and performs max pooling to reduce the spatial dimension of image features. The max pooling operation involves selecting the maximum value of each feature and aggregating each feature value; The Inception network's fully connected layers fuse the features extracted by the convolutional and pooling layers, transforming the features of each image region into a fixed-length set of image region feature vectors. The image region feature vector includes visual information of each region in the image, including the image's color, texture, object shape, and structure.
[0022] This step uses the Inception network for image feature extraction, employing multi-scale convolutional operations at different kernel sizes to extract local and global features, effectively capturing visual information such as texture, color, object contours, and shape. A ReLU activation function is used for non-linear transformation, enhancing the expressive power of the features. After the convolutional layers, the features are compressed in spatial dimension through max pooling, preserving the most important feature information. Fully connected layers fuse the extracted image features, generating a fixed-length set of image region feature vectors. This method improves the accuracy of image annotation by precisely extracting visual information from image regions, providing high-quality image feature support for subsequent image and text multimodal annotation tasks.
[0023] In this embodiment, the step of converting each word in the text dataset into a word vector representation and embedding the positional information into the text features through positional encoding to obtain a text feature vector set is specifically as follows: The WordPiece word segmentation method is used to segment the text dataset into words or subwords, and each word or subword is mapped to a unique word ID through a vocabulary. The steps of the WordPiece word segmentation method are as follows: Remove spaces and punctuation marks from a text dataset; Each word in the text dataset is processed one by one from left to right, and a perfect match is found in the vocabulary. If a perfect match is found, the current word is treated as a single word unit. If a word cannot find a complete match in the vocabulary, the current word is split into multiple sub-word units and the search and matching are performed again. The splitting step is to find the longest sub-word match of the current word and use the longest matching sub-word to split it until all sub-words of the current word can be found in the vocabulary. Based on the word ID of each word, each word is mapped to a fixed-dimensional vector representation to obtain the word vector; Based on the word ID, a cosine function is used to generate a position encoding vector for each word's word vector. The position encoding vector is used to represent the word's order in the sentence. Each word's word vector is combined with its position index in the sentence to generate a unique numerical representation. The word vector of each word is added to the corresponding position encoding vector to generate a text feature vector with position information; By summing up all the text feature vectors, we obtain the text feature vector set.
[0024] This step employs the WordPiece word segmentation method to finely segment the text data, effectively improving its representation capabilities. Removing spaces and punctuation ensures standardized text processing. During segmentation, finding exact matches in the vocabulary and performing longest subword matching addresses the limitation of traditional segmentation methods in handling unknown words, thus enhancing vocabulary coverage and model generalization ability. Each word is mapped to a fixed-dimensional word vector, and positional encoding is used to embed positional information into the word vectors, providing more accurate contextual semantic information for subsequent modeling. Ultimately, the generated text feature vector set contains the positional information and contextual semantics of each word in the text, enhancing the model's understanding of the text and significantly improving the accuracy and consistency of image and text multimodal annotation.
[0025] In this embodiment, the step of inputting the image region feature vector set and the text feature vector set into the improved ViLBERT model, and obtaining the image text feature map through a two-stream network architecture and a knowledge-aware encoder, specifically involves: The improved ViLBERT model employs a two-stream network architecture to process image region feature vector sets and text feature vector sets separately. The two-stream network architecture includes a visual stream and a text stream. The visual stream processing steps are as follows: Each image region feature vector in the image region feature vector set is used as a node in the image region feature map to obtain the image node set; Calculate the Euclidean distance between the feature vectors of every two image regions. If the Euclidean distance is less than a preset distance threshold, establish an edge connection between the corresponding nodes to obtain the image edge set. Based on the correspondence between the image node set and the image edge set, construct the image region feature map; The specific steps of the text stream processing are as follows: Each text feature vector in the text feature vector set is used as a node in the text feature map to obtain a text node set; Calculate the Euclidean distance between any two text feature vectors. If the Euclidean distance is less than a preset distance threshold, establish an edge connection between the corresponding nodes to obtain the text edge set. Based on the correspondence between text node sets and text edge sets, construct a text feature map; The image region feature map and text feature map are input into a knowledge-aware encoder to perform cross-modal node matching to obtain the image text feature map. The cross-modal node matching step is as follows: Calculate the Manhattan distance between each pair of nodes in the image region feature map and the text feature map, whereby the Manhattan distance is used to measure the similarity between nodes in the image region feature map and the text feature map; If the Manhattan distance is less than the preset similarity threshold, it indicates that the image region and the text feature are semantically similar. Then, a cross-modal node connection is established between the corresponding image region feature vector and the text feature vector, and the weight value of the connection edge is the corresponding Manhattan distance. Connect all nodes that satisfy cross-modal node matching to obtain the image text feature map.
[0026] This step employs a modified ViLBERT model with a two-stream network architecture to process image region features and text features separately. Through processing via visual and text streams, more accurate image and text feature fusion is achieved. The visual stream constructs image region feature maps to capture spatial relationships between image regions, while the text stream constructs text feature maps to establish semantic associations between text features. By calculating the Manhattan distance between image and text nodes, the model effectively performs cross-modal node matching of image regions and text features, generating an image-text feature map. This innovative method significantly improves the accuracy and consistency of image and text annotation, enabling better understanding and generation of annotated content. It addresses the insufficient semantic fusion of images and text in existing methods, enhancing the performance and application effectiveness of multimodal data annotation.
[0027] In this embodiment, the step of constructing a Laplacian matrix based on the image text feature map and generating a fused feature representation based on the Laplacian matrix specifically involves: An adjacency matrix is constructed based on the image text feature map. If there is no edge connecting two nodes in the adjacency matrix, the value of the corresponding element in the adjacency matrix is 0. If there is an edge connecting two nodes, then the element value in the corresponding adjacency matrix is the reciprocal of the weight of the connecting edge; Calculate the degree matrix of the image text feature map, where each element in the degree matrix is the degree of the corresponding node, and the degree is the number of connected edges of the node; Subtracting the degree matrix from the adjacency matrix yields the Laplace matrix; The Laplacian matrix is decomposed into eigenvalues to obtain the eigenvectors and eigenvalues of the Laplacian matrix. The eigenvectors represent the fusion features of nodes in the image text feature map, and the eigenvalues represent the similarity strength between nodes. Based on the magnitude of the eigenvalues, the eigenvectors corresponding to the k smallest eigenvalues are selected and weighted and summed to obtain the fused feature representation.
[0028] This step constructs the adjacency matrix and degree matrix of the image-text feature map, and uses the Laplacian matrix to perform deep fusion and feature extraction of image and text features. By calculating the inverse of the edge weights between nodes in the adjacency matrix and combining it with the node degree, a degree matrix and a Laplacian matrix that reflect the similarity and connection between the image and text are constructed. Eigenvalue decomposition is performed on the Laplacian matrix, and the generated eigenvectors can effectively represent the fusion information of image and text features, while the eigenvalues reflect the similarity strength between nodes. By using a weighted summation method, the eigenvectors corresponding to the k smallest eigenvalues are selected to generate the fused feature representation, thereby improving the consistency and accuracy of image and text annotation and enhancing the ability to process cross-modal data. This method not only effectively improves the semantic fusion of image and text features, but also provides richer and higher-quality feature support for subsequent annotation tasks.
[0029] In this embodiment, the step of inputting the fused feature representation into the annotation generation module to generate image annotation results and text annotation results specifically involves: The fused feature representations are input into the annotation generation module for image annotation and text annotation tasks, respectively. In the image annotation task, the annotation generation module generates image annotation results by matching image region features with fused feature representations and combining them with image background information, which includes object category, location, and color. In the text annotation task, the annotation generation module compares and fuses the relationship between the feature representation and the image annotation results, and then matches the text annotations with the image regions to generate text annotation results that are related to the image content.
[0030] In this embodiment, the step of comparing the generated image annotation results and text annotation results with a manually annotated dataset or a standard dataset to generate an annotation report specifically involves: Obtain annotation data relevant to image and text annotation tasks from standard datasets or manually annotated datasets; For image annotation tasks, compare the similarity between the generated text descriptions and the description texts in manually annotated data; For text annotation tasks, the similarity between the generated text annotations and the annotated text in the standard dataset is compared. An annotation report is generated based on the comparison results. The annotation report includes similarity and error analysis. The error analysis is the difference between the annotation results and the standard dataset, including incorrect annotations, missing annotations, and over-annotations.
[0031] Example 1: To verify the feasibility of this invention in practice, it was applied to a modern intelligent vision system. Multimodal image and text annotation tasks have wide applications, especially in automatic image description generation and text annotation. Traditional annotation methods typically rely on separate image feature extraction and text feature modeling processes. A common approach is to use convolutional neural networks (CNNs) to extract image features and long short-term memory networks (LSTMs) to process text data. However, this method often neglects the deep semantic relationships between images and text, leading to insufficient accuracy and consistency in the generated annotation results. To address this issue, this invention proposes an automatic annotation method based on image-text feature fusion. Utilizing an improved ViLBERT model and a Laplacian matrix fusion strategy, it successfully improves the effectiveness of image and text annotation.
[0032] In this embodiment, we applied this method to process a dataset of 1000 images and their corresponding text descriptions, covering different scenes and object types, such as landscape images, animal images, and human images. Through the image region feature extraction module, the Inception network was used to perform multi-scale convolution operations on the image regions, generating a high-quality set of image region feature vectors. Simultaneously, the text was processed using the WordPiece word segmentation method, transforming the text dataset into a set of text feature vectors with semantic information. These image and text features were then input into an improved ViLBERT model, processed through a two-stream network architecture and a knowledge-aware encoder, resulting in an image-text feature map, effectively capturing the deep semantic relationships between images and text.
[0033] In the annotation generation module, we generate text descriptions of images based on image text feature maps and generate text annotations related to the image content through cross-modal reasoning. By comparing the generated annotation results with a manually annotated dataset, we can accurately evaluate the effectiveness of this method. Experimental results show that the method of this invention significantly outperforms traditional methods in terms of the accuracy and consistency of annotation generation, especially demonstrating higher accuracy and stronger semantic consistency in image description generation and text annotation tasks.
[0034] To verify the superiority of our proposed method, we compared it with three different traditional methods: a traditional method based on a combination of Convolutional Neural Networks (CNN) and Long Short-Term Memory Networks (LSTM), an image text annotation method based on a pre-trained BERT model, and a traditional method based on a two-stream network. Experiments show that our proposed method outperforms traditional methods in multiple evaluation metrics, including precision, recall, and F1 score, particularly demonstrating significant improvements in the accuracy of image description generation and text annotation. The recall rate is the ratio of correctly labeled positive samples to the total number of actual positive samples, and the F1 score is the ratio of twice the precision multiplied by the recall rate to the sum of precision and recall.
[0035] Table 1 Comparison of Image Text Annotation Methods
[0036] As shown in Table 1, the method of this invention significantly outperforms traditional methods in all evaluation metrics. Particularly in precision and recall, the method improves by approximately 5 percentage points, demonstrating more accurate semantic understanding of images and text during annotation generation. Furthermore, the improved F1 score proves that this method better balances accuracy and recall, resulting in more reliable and consistent annotation results. This experiment verifies the effectiveness of the method in multimodal data annotation tasks, particularly in image description generation and text annotation applications, significantly improving the quality of annotation results and providing strong technical support for subsequent automated annotation systems.
[0037] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for automatic annotation of multimodal data based on semantic understanding, characterized in that, Includes the following steps: Step 1: Collect and preprocess raw image and text data from multiple data sources to obtain image and text datasets with a unified structure; Step 2: Input the image dataset into the Inception network to extract the visual features of each image region and generate the corresponding image region feature vector set; Step 3: Convert each word in the text dataset into a word vector representation, and embed the positional information into the text features through positional encoding to obtain a text feature vector set; Step 4: Input the image region feature vector set and the text feature vector set into the improved ViLBERT model, and obtain the image text feature map through a two-stream network architecture and a knowledge-aware encoder; Step 5: Construct a Laplacian matrix based on the image text feature map, and generate a fused feature representation based on the Laplacian matrix; Step 6: Input the fused feature representation into the annotation generation module to generate image annotation results and text annotation results; Step 7: Compare the generated image annotation results and text annotation results with the manually annotated dataset or standard dataset, and generate an annotation report.
2. The automatic annotation method for multimodal data based on semantic understanding according to claim 1, characterized in that, The original image data includes static images and dynamic video frames; the original text data includes natural language descriptions, tags, question-and-answer text, and titles of the original images; the preprocessing steps include outlier removal, missing value imputation, timestamp alignment, and normalization of different types of data to obtain image datasets and text datasets with a unified structure.
3. The automatic annotation method for multimodal data based on semantic understanding according to claim 1, characterized in that, Step two specifically involves: The Inception network extracts features from images at different levels simultaneously at different kernel sizes through multi-scale convolution operations. It also performs non-linear transformations using the ReLU activation function, setting all negative values to zero while keeping positive values unchanged, thus capturing both local and global features of the image. The local features include texture and color, and the global features include object outline and shape; After the convolutional layers, the Inception network fuses the features extracted by convolutional kernels of different sizes and performs max pooling to reduce the spatial dimension of image features. The max pooling operation involves selecting the maximum value of each feature and aggregating each feature value; The Inception network's fully connected layers fuse the features extracted by the convolutional and pooling layers, transforming the features of each image region into a fixed-length set of image region feature vectors. The image region feature vector includes visual information of each region in the image, including the image's color, texture, object shape, and structure.
4. The automatic annotation method for multimodal data based on semantic understanding according to claim 1, characterized in that, Step three specifically involves: The WordPiece word segmentation method is used to segment the text dataset into words or subwords, and each word or subword is mapped to a unique word ID through a vocabulary. The steps of the WordPiece word segmentation method are as follows: Remove spaces and punctuation marks from a text dataset; Each word in the text dataset is processed one by one from left to right, and a perfect match is found in the vocabulary. If a perfect match is found, the current word is treated as a single word unit. If a word cannot find a complete match in the vocabulary, the current word is split into multiple sub-word units and the search and matching are performed again. The splitting step is to find the longest sub-word match of the current word and use the longest matching sub-word to split it until all sub-words of the current word can be found in the vocabulary. Based on the word ID of each word, each word is mapped to a fixed-dimensional vector representation to obtain the word vector; Based on the word ID, a cosine function is used to generate a position encoding vector for each word's word vector. The position encoding vector is used to represent the word's order in the sentence. Each word's word vector is combined with its position index in the sentence to generate a unique numerical representation. The word vector of each word is added to the corresponding position encoding vector to generate a text feature vector with position information; By summing up all the text feature vectors, we obtain the text feature vector set.
5. The automatic annotation method for multimodal data based on semantic understanding according to claim 1, characterized in that, The improved ViLBERT model is specifically as follows: The improved ViLBERT model employs a two-stream network architecture to process image region feature vector sets and text feature vector sets separately. The two-stream network architecture includes a visual stream and a text stream. The visual stream processing steps are as follows: Each image region feature vector in the image region feature vector set is used as a node in the image region feature map to obtain the image node set; Calculate the Euclidean distance between the feature vectors of every two image regions. If the Euclidean distance is less than a preset distance threshold, establish an edge connection between the corresponding nodes to obtain the image edge set. Based on the correspondence between the image node set and the image edge set, construct the image region feature map; The specific steps of the text stream processing are as follows: Each text feature vector in the text feature vector set is used as a node in the text feature map to obtain a text node set; Calculate the Euclidean distance between any two text feature vectors. If the Euclidean distance is less than a preset distance threshold, establish an edge connection between the corresponding nodes to obtain the text edge set. Based on the correspondence between text node sets and text edge sets, construct a text feature map; The image region feature map and text feature map are input into a knowledge-aware encoder to perform cross-modal node matching to obtain the image text feature map. The cross-modal node matching step is as follows: Calculate the Manhattan distance between each pair of nodes in the image region feature map and the text feature map, whereby the Manhattan distance is used to measure the similarity between nodes in the image region feature map and the text feature map; If the Manhattan distance is less than the preset similarity threshold, it indicates that the image region and the text feature are semantically similar. Then, a cross-modal node connection is established between the corresponding image region feature vector and the text feature vector, and the weight value of the connection edge is the corresponding Manhattan distance. Connect all nodes that satisfy cross-modal node matching to obtain the image text feature map.
6. The automatic annotation method for multimodal data based on semantic understanding according to claim 1, characterized in that, Step five specifically involves: An adjacency matrix is constructed based on the image text feature map. If there is no edge connecting two nodes in the adjacency matrix, the value of the corresponding element in the adjacency matrix is 0. If there is an edge connecting two nodes, then the element value in the corresponding adjacency matrix is the reciprocal of the weight of the connecting edge; Calculate the degree matrix of the image text feature map, where each element in the degree matrix is the degree of the corresponding node, and the degree is the number of connected edges of the node; Subtracting the degree matrix from the adjacency matrix yields the Laplace matrix; The Laplacian matrix is decomposed into eigenvalues to obtain the eigenvectors and eigenvalues of the Laplacian matrix. The eigenvectors represent the fusion features of nodes in the image text feature map, and the eigenvalues represent the similarity strength between nodes. Based on the magnitude of the eigenvalues, the eigenvectors corresponding to the k smallest eigenvalues are selected and weighted and summed to obtain the fused feature representation.
7. The automatic annotation method for multimodal data based on semantic understanding according to claim 1, characterized in that, Step six specifically involves: The fused feature representations are input into the annotation generation module for image annotation and text annotation tasks, respectively. In the image annotation task, the annotation generation module generates image annotation results by matching image region features with fused feature representations and combining them with image background information, which includes object category, location, and color. In the text annotation task, the annotation generation module compares and fuses the relationship between the feature representation and the image annotation results, and then matches the text annotations with the image regions to generate text annotation results that are related to the image content.
8. The automatic annotation method for multimodal data based on semantic understanding according to claim 1, characterized in that, Step seven specifically involves: Obtain annotation data relevant to image and text annotation tasks from standard datasets or manually annotated datasets; For image annotation tasks, compare the similarity between the generated text descriptions and the description texts in manually annotated data; For text annotation tasks, the similarity between the generated text annotations and the annotated text in the standard dataset is compared. An annotation report is generated based on the comparison results. The annotation report includes similarity and error analysis. The error analysis is the difference between the annotation results and the standard dataset, including incorrect annotations, missing annotations, and over-annotations.