Multi-modal ethnic culture entity identification method
Through the multimodal ethnic cultural entity recognition method, the image information is mapped to text representation using bridge network and contrast learning technology, and the large model is trained through instruction fine-tuning, which solves the problem that the existing technology is difficult to identify and distinguish cultural symbols of different ethnic groups, and realizes the in-depth exploration and inheritance of ethnic cultural knowledge.
Patent Information
- Application Number
- CN202411725264.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-21
AI Technical Summary
The existing technology is difficult to effectively identify and distinguish similar ethnic cultural symbols among different ethnic groups, and lacks a deep understanding of the ambiguity and complexity of ethnic symbols, and cannot support the exploration of correlation between symbols.
The multimodal ethnic cultural entity recognition method is adopted to map the image representations obtained by the CLIP model to the text representation of the large model by designing a bridge network, and a comparison learning data set is constructed. The semantic understanding of the network's image alignment is strengthened through comparison learning, and the big model is trained in combination with instructions to improve entity recognition capabilities.
The integration of multimodal information has been realized, the ability to identify subtle cultural symbols has been improved, and the semantic consistency across modalities has been enhanced, so that the model can extract relevant entity information more accurately and effectively in complex national cultural scenarios, and support the in-depth exploration and inheritance of national cultural knowledge.
Smart Images
Figure CN119669683A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of national culture identification, and in particular to a multimodal national culture entity identification method. Background Art
[0002] National cultural symbols contain rich modal features, covering multiple forms such as images, physical objects, and sounds, but there are relatively few written records. These symbols reflect the unique history and customs of each ethnic group and are of great significance in cultural protection and inheritance. However, with the passage of time and the exchange and integration between ethnic groups, similarities and correlations have emerged between the cultural symbols of different ethnic groups. This complexity makes it difficult for traditional recognition methods to cope with it, so multimodal recognition of national cultural symbols has become an urgent problem to be solved.
[0003] Traditional entity recognition methods mainly model text information and then identify entity information contained in the text. For example, the Bert+CRF method uses Bert to learn the identity of each word in a sentence, and then uses CRF to assign a BIO label to each word, thereby achieving entity classification and extraction. However, with the development of deep learning technology, more and more researchers have begun to explore entity recognition methods based on deep neural networks. For example, Transformer-based pre-trained models, such as BERT, RoBERTa, ChatGLM, etc., can capture rich contextual information through pre-training on large-scale corpora, thereby achieving significant performance improvements in entity recognition tasks.
[0004] Current entity recognition technologies are usually based on single-modal (such as text or image) data, and lack full integration and understanding of multimodal data. Especially in the field of ethnic culture, existing technologies are difficult to use multiple modal information such as images and physical objects at the same time, and cannot effectively identify and distinguish similar symbols between different ethnic groups, and thus cannot support the mining of associations between symbols. In addition, existing methods lack a deep understanding of the ambiguity and complexity of ethnic symbols, which limits the in-depth interpretation of ethnic culture.
[0005] In order to more comprehensively and accurately identify national cultural symbols and support the research on national cultural relevance, the present invention proposes a multimodal entity recognition method oriented to national culture. Summary of the invention
[0006] In response to the technical problems raised in the background technology, the present invention proposes a multimodal ethnic cultural entity recognition method.
[0007] The present invention adopts the following technical solution:
[0008] The multimodal ethnic cultural entity recognition method includes the following steps:
[0009] Step S1: First, a bridge network is designed to map the image representation obtained by CLIP to the text representation of the large model. Then, a contrastive learning dataset is constructed to align the output features of the bridge network with the text embedding. During the training process, contrastive learning is used to strengthen the network's semantic understanding of image-text alignment, so that the image features are more in line with the semantic requirements of the text description.
[0010] Step S2: Secondly, fine-tune the large model through instruction fine-tuning training to improve its entity recognition ability;
[0011] Step S3: Finally, the image and text descriptions are input into the big model, and the big model is asked to identify the entities and their types.
[0012] As a further technical solution of the present invention, in step S1, constructing a comparative learning data set includes the following steps:
[0013] Step S101: Data collection:
[0014] First, we collect a large amount of ethnic culture-related image-text pairing data. Each pair of data includes an image and a corresponding text description. These image-text pairs form the basis of the training dataset.
[0015] Step S102: Extraction of image embedding representation:
[0016] The image encoder using the CLIP model represents each image as a vector. Assuming the image is I, the CLIP model converts it into a vector v i ,The CLIP model is pre-trained on large-scale image-text pairing data and can effectively align images and texts across modalities;
[0017] Step S103: construct negative samples:
[0018] For each picture-text pair (I i , T i ), generate negative sample pairs to enhance the model’s ability to distinguish; specifically, we can find negative samples in the training set that are similar to the target image I i Images with similar content but different text descriptions Generate negative sample pairs This construction of negative samples can enable the model to learn subtle differences. For example, for a picture of Tibetan clothing I i and description text T i , select pictures of Mongolian costumes As negative samples, it helps the model distinguish the clothing features of different ethnic groups;
[0019] Step S104: Rewriting of text description:
[0020] In order to further expand the contrastive learning dataset, a large language model is used to rewrite the original text description. During the rewriting, the entity words are replaced with other ethnic names to form new negative samples.
[0021] The goal of contrastive learning is to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs, using the InfoNCE contrastive loss function for training:
[0022]
[0023] Among them, sim(v i , t i ) represents the similarity between image and text embedding, and τ is the temperature coefficient.
[0024] As a further technical solution of the present invention, in step S1, a bridge network is designed based on a Transformer and a fully connected layer (MLP) to map the image embedding generated by CLIP to a vector space of text embedding;
[0025] Use contrastive learning dataset to train bridge network and embed CLIP images into V i After inputting the bridge network, the mapped feature z is obtained i , whose goal is to i Alignment,optimize the parameters of the bridge network by contrastive loss function;
[0026]
[0027] Through this training process, the bridge network embeds the image into v i Transformed into semantically similar text embedding i Aligned representation z i , thus achieving cross-modal fusion.
[0028] As a further technical solution of the present invention, in step S2, the step of fine-tuning the large model through instruction fine-tuning training includes:
[0029] LoRA (Low-Rank Adaptation) is used to fine-tune the large model. LoRA decomposes the low-rank matrix and represents the weight matrix of the large model as the product of smaller matrices, thereby reducing the number of training parameters and accelerating the fine-tuning process. During training, the output embedding of the bridge network and the text embedding are input into the large model together, and fine-tuned through LoRA to improve the performance of the model in the task of ethnic cultural entity recognition.
[0030] As a further technical solution of the present invention, the bridging network adopts the Transformer+fully connected network method, and the large model is Qwen2.5-72B.
[0031] Compared with the prior art, the present invention has the following beneficial effects:
[0032] The present invention makes full use of the powerful image representation capabilities of the trained image model, and seamlessly maps the image information to the text representation space through a bridging network, thereby realizing the fusion of multimodal information. The bridging network combines Transformer and multi-layer perceptron, which can capture the contextual relationship of image features and efficiently convert them into text features. This multimodal alignment method can combine image and text information while utilizing the natural language understanding capabilities of a large model to accurately identify entities in images and texts in national culture. Compared with traditional methods, the method of the present invention not only improves the ability to recognize subtle cultural symbols, but also enhances cross-modal semantic consistency, enabling the model to more accurately and effectively extract relevant entity information in complex national cultural scenes, achieving in-depth mining and inheritance of national cultural knowledge, and effectively promoting the application of multimodal knowledge. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 Schematic diagram of the overall training framework in an embodiment of the present invention.
[0034] Figure 2 This is a graphic comparison data diagram of ethnic costumes in an embodiment of the present invention.
[0035] Figure 3 The following is a flow chart of the model application in the embodiment of the present invention. DETAILED DESCRIPTION
[0036] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] The overall process of the multimodal knowledge extraction model is as follows Figure 1 As shown, first you need to select picture-text pairing data containing content related to ethnic culture, such as ethnic totems, ethnic costumes, ethnic architecture, etc. Figure 2 The example data of a national culture is shown, which includes a picture and a corresponding text description. The text details the cultural entity information involved in the picture. In the overall process, the multimodal knowledge extraction capability of the model is realized through three steps: comparative learning data construction, bridge network training, and entity recognition fine-tuning.
[0038] See also Figures 1 to 3, an embodiment of the present invention provides a multimodal ethnic cultural entity recognition method, comprising the following steps:
[0039] Step S1: First, a bridge network is designed to map the image representation obtained by CLIP to the text representation of the large model. Then, a contrastive learning dataset is constructed to align the output features of the bridge network with the text embedding. During the training process, contrastive learning is used to strengthen the network's semantic understanding of image-text alignment, so that the image features are more in line with the semantic requirements of the text description.
[0040] Step S2: Secondly, fine-tune the large model through instruction fine-tuning training to improve its entity recognition ability;
[0041] Step S3: Finally, the image and text descriptions are input into the big model, and the big model is asked to identify the entities and their types.
[0042] In this embodiment, in model training, the construction of a contrastive learning dataset is a key step in improving the model's sensitivity to details and multimodal alignment capabilities. In step S1, the construction of the contrastive learning dataset includes the following steps:
[0043] Step S101: Data collection:
[0044] First, we collect a large amount of image-text pairing data related to ethnic culture. Each pair of data includes an image (such as an image of ethnic clothing) and a corresponding text description (such as a detailed introduction to the clothing). These image-text pairs form the basis of the training data set.
[0045] Step S102: Extraction of image embedding representation:
[0046] The image encoder using the CLIP model represents each image as a vector. Assuming the image is I, the CLIP model converts it into a vector v i ,The CLIP model is pre-trained on large-scale image-text pairing data and can effectively align images and texts across modalities;
[0047] Step S103: construct negative samples:
[0048] For each picture-text pair (I i , T i ), generate negative sample pairs to enhance the model’s ability to distinguish; specifically, we can find negative samples in the training set that are similar to the target image I i Images with similar content but different text descriptions Generate negative sample pairs This construction of negative samples can enable the model to learn subtle differences. For example, for a picture of Tibetan clothing I i and description text T i , select pictures of Mongolian costumes As negative samples, it helps the model distinguish the clothing features of different ethnic groups;
[0049] Step S104: Rewriting of text description:
[0050] In order to further expand the contrastive learning dataset, a large language model (such as GPT-3) is used to rewrite the original text description; when rewriting, the entity words (such as "Tibetan") are replaced with other ethnic names to form new negative samples; for example, "Tibetan clothing" is replaced with "Kazakh clothing" to generate a new picture-text pair. This data augmentation strategy can improve the robustness of the model;
[0051] The goal of contrastive learning is to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs, using the InfoNCE contrastive loss function for training:
[0052]
[0053] Among them, sim(v i , t i ) represents the similarity between image and text embedding (such as cosine similarity), and τ is the temperature coefficient.
[0054] In this embodiment, the bridge network plays a role in mapping the image modality to the text modality, so that the image and text can be represented in the same vector space. In step S1, the bridge network is designed based on Transformer and a fully connected layer (MLP) to map the image embedding generated by CLIP to the vector space of text embedding. Assuming that the dimension of the text embedding is d, the output dimension of the bridge network also needs to be adjusted to d to ensure the alignment effect;
[0055] Use contrastive learning dataset to train bridge network and embed CLIP images into V i After inputting the bridge network, the mapped feature z is obtained i , whose goal is to i Alignment,optimize the parameters of the bridge network by contrastive loss function;
[0056]
[0057] Through this training process, the bridge network embeds the image into v i Transformed into semantically similar text embedding i Aligned representation z i , thus achieving cross-modal fusion.
[0058] In this embodiment, after the bridging network conversion, the image data has been mapped to the text embedding space, and the image features and text features can be input into the entity recognition model together to extract knowledge in the field of national culture.
[0059] The instruction learning dataset is constructed using labeled data. Each sample in the instruction dataset contains input data (text or image) and its corresponding target output (i.e., the national cultural entity to be identified), ensuring that the model can better understand the task requirements through instructions.
[0060] In this embodiment, in step S2, the step of fine-tuning the large model through instruction fine-tuning training includes:
[0061] LoRA (Low-Rank Adaptation) is used to fine-tune the large model. LoRA decomposes the low-rank matrix and represents the weight matrix of the large model as the product of smaller matrices, thereby reducing the number of training parameters and accelerating the fine-tuning process. During training, the output embedding of the bridge network and the text embedding are input into the large model together, and fine-tuned through LoRA to improve the performance of the model in the task of ethnic cultural entity recognition.
[0062] In the testing phase, the model identifies the national cultural entity information based on the input national cultural image or text description. Through the bridge network, the input of the image modality is converted to the text embedding space, and then the knowledge extraction model completes the recognition and knowledge extraction.
[0063] Application process such as Figure 3 As shown in the figure, the application process of the model is as follows: when an image in the field of ethnic culture is input, CLIP is first used to extract the image embedding, and then it is converted to the text embedding space through the bridge network. Finally, the fine-tuned knowledge extraction model (large model) is used to identify and extract entity information related to ethnic culture from the image description, such as totems and clothing styles of a specific ethnic group. Through multimodal fusion, the model can accurately identify and extract rich knowledge in the field of ethnic culture. Contrastive learning improves the model's sensitivity to detailed features, the bridge network realizes the alignment of image and text modalities, and LoRA fine-tuning strengthens the entity recognition capability.
[0064] The CLIP model used in the present invention in image representation can also be a model such as ALIGN, FLAVA, etc. of the pelican eye;
[0065] The bridge network uses the Transformer + fully connected network approach, or the fully connected network can be used directly;
[0066] The large model used in the present invention is Qwen2.5-72B, which can also be replaced by other types of open source large models, or commercial large models that support fine-tuning.
[0067] The present invention uses LoRA to fine-tune the model, which can also be replaced by methods such as P-tuning.
[0068] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.
[0069] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A multimodal ethnic cultural entity recognition method, characterized in that: The following steps are involved: Step S1: First, a bridge network is designed to map the image representation obtained by CLIP to the text representation of the large model. Then, a contrastive learning dataset is constructed to align the output features of the bridge network with the text embedding. During the training process, contrastive learning is used to strengthen the network's semantic understanding of image-text alignment, so that the image features meet the semantic requirements of the text description. Step S2: Secondly, fine-tune the large model through instruction fine-tuning training to improve its entity recognition ability; Step S3: Finally, the image and text descriptions are input into the big model, and the big model is asked to identify the entities and their types.
2. The multimodal ethnic cultural entity recognition method according to claim 1, characterized in that: In step S1, constructing a comparative learning dataset includes the following steps: Step S101: Data collection: First, a large amount of picture-text pairing data related to ethnic culture is collected. Each pair of data includes an image and a corresponding text description. Step S102: Extraction of image embedding representation: The image encoder using the CLIP model represents each image as a vector. Assuming the image is I, the CLIP model converts it into a vector v i ; Step S103: construct negative samples: For each picture-text pair (I i , T i ), generate negative sample pairs to enhance the model’s ability to distinguish; find the image I in the training set that matches the target image i Images with similar content but different text descriptions Generate negative sample pairs Step S104: Rewriting of text description: In order to further expand the contrastive learning dataset, a large language model is used to rewrite the original text description. During the rewriting, the entity words are replaced with other ethnic names to form new negative samples. The goal of contrastive learning is to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs, using the InfoNCE contrastive loss function for training: Among them, sim(v i , t i ) represents the similarity between image and text embedding, and τ is the temperature coefficient.
3. The multimodal ethnic cultural entity recognition method according to claim 2, characterized in that: In step S1, the bridge network is designed based on Transformer and fully connected layers to map the image embedding generated by CLIP to the vector space of text embedding; Use contrastive learning dataset to train bridge network and embed CLIP images into V i After inputting the bridge network, the mapped feature z is obtained i , whose goal is to i Alignment,optimize the parameters of the bridge network by contrastive loss function; Through this training process, the bridge network embeds the image into v i Transformed into semantically similar text embedding i Aligned representation z i , thus achieving cross-modal fusion.
4. The multimodal ethnic cultural entity recognition method according to claim 1, characterized in that: In step S2, the step of fine-tuning the large model through instruction fine-tuning training includes: LoRA is used to fine-tune the large model. LoRA decomposes the low-rank matrix and represents the weight matrix of the large model as the product of smaller matrices, thereby reducing the number of training parameters and accelerating the fine-tuning process. During training, the output embedding of the bridge network and the text embedding are input into the large model together, and fine-tuned through LoRA to improve the performance of the model in the task of ethnic cultural entity recognition.
5. The multimodal ethnic cultural entity recognition method according to claim 1, characterized in that: The bridge network adopts the Transformer+fully connected network approach, and the large model is Qwen2.5-72B.