A multi-modal retrieval method combining image features and semantic understanding
By generating visual fragment features and semantic framework sequences through image segmentation and semantic role labeling, a semantically anchored visual codebook and cross-modal index are constructed, solving the problems of information loss and low accuracy in multimodal retrieval, and realizing deep fusion and accurate matching of images and text.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING AUGUST MELON TECHNOLOGY CO LTD
- Filing Date
- 2026-03-17
- Publication Date
- 2026-06-19
AI Technical Summary
Existing multimodal retrieval methods fail to fully consider the differences in deep semantic information between images and text, resulting in information loss and reduced retrieval accuracy, thus affecting the accuracy and efficiency of multimodal retrieval.
Image segmentation technology is used to generate visual fragment feature sets, and semantic role annotation is combined to generate semantic framework sequences. Visual codeword sets are generated through role mapping records and K-means clustering. A semantically anchored visual codebook and cross-modal anchored index are constructed to achieve deep fusion of images and text.
It improves the accuracy and efficiency of multimodal retrieval, alleviates the problem of information loss through deep fusion of images and text, and achieves more accurate cross-modal matching and indexing.
Smart Images
Figure CN121858757B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information retrieval technology, and in particular to a multimodal retrieval method that combines image features and semantic understanding. Background Technology
[0002] With the rapid development of information technology, multimodal data generated on the internet and various digital platforms is growing explosively, especially the proportion of image and text data. In traditional information retrieval methods, image and text data are usually processed independently. Image data often relies on computer vision technology to extract image features, while text data relies on natural language processing technology for understanding. This independent processing approach may lead to information loss in practical applications, failing to achieve effective integration and deep understanding of images and text, thus affecting the accuracy and efficiency of multimodal retrieval.
[0003] Existing multimodal retrieval methods primarily focus on feature alignment and matching between images and text, but most rely on simple vector space models, calculating the similarity between images and text for matching. However, this approach does not fully consider the differences in deep semantic information between images and text, particularly how to more accurately combine the content in the image with the description in the text. Furthermore, the retrieval accuracy of existing methods is often affected by information loss and modal inconsistencies, leading to reduced cross-modal matching accuracy. Summary of the Invention
[0004] To address the technical problems existing in the background art, this invention proposes a multimodal retrieval method that combines image features and semantic understanding.
[0005] This invention proposes a multimodal retrieval method that combines image features and semantic understanding, comprising the following steps:
[0006] S1. Obtain the multimodal data to be retrieved, which includes image data and text data;
[0007] For each image in the image data, image segmentation techniques are used to generate a visual segment feature set corresponding to each image;
[0008] Semantic role labeling is applied to each text in the text data to generate a semantic framework sequence corresponding to each text.
[0009] S2. Based on the semantic frame sequence, using the semantic frame sequence as the original semantic frame sequence, generate each permuted semantic frame sequence and role mapping record corresponding to the original semantic frame sequence.
[0010] Based on each permuted semantic framework sequence and role mapping record corresponding to the original semantic framework sequence, a set of visual codewords is generated.
[0011] Based on the role mapping record and the visual codeword set, a semantically anchored visual codebook is generated;
[0012] S3. Generate a cross-modal anchored index that can be used for retrieval based on the visual codeword set and semantic frame sequence.
[0013] Preferably, in S1, image segmentation technology is applied to each image in the image data to generate a visual segment feature set corresponding to each image, as follows:
[0014] Each image in the image data is segmented into multiple visual segments using image segmentation techniques, and each visual segment is appended with a segment location description and segment type identifier; a convolutional neural network is used to obtain the segment feature vector corresponding to each visual segment;
[0015] Obtain all visual segments and their corresponding segment feature vectors to form a visual segment feature set.
[0016] Preferably, in S1, semantic role labeling is applied to each text in the text data to generate a semantic frame sequence corresponding to each text, as follows:
[0017] Semantic role labeling is applied to each text in the text data to generate multiple semantic frames corresponding to each text. The multiple semantic frames are arranged in the order of their appearance to form a semantic frame sequence.
[0018] The semantic framework includes at least one semantic role slot, a semantic role tag corresponding to the role type of the semantic role slot, and the slot content corresponding to the semantic role slot; wherein, the semantic role tag is generated from the field representing the role type in the semantic role annotation result.
[0019] Preferably, in S2, based on the semantic frame sequence, using the semantic frame sequence as the original semantic frame sequence, each permutation semantic frame sequence and role mapping record corresponding to the original semantic frame sequence are generated, as follows:
[0020] Based on the semantic role slots in the semantic frame sequence, the semantic role labels of at least two semantic role slots in each semantic frame sequence are interchanged to generate at least one permuted semantic frame sequence with the same slot content as the original semantic frame sequence but different semantic role label correspondences. The semantic role label correspondences before and after the interchange are recorded to form a role mapping record.
[0021] Preferably, in S2, a visual codeword set is generated based on each permutation semantic frame sequence and role mapping record corresponding to the original semantic frame sequence, as follows:
[0022] Using the semantic role slots in the original semantic frame sequence and each permuted semantic frame sequence as anchor points, the visual segment feature vectors in the same visual segment feature set are clustered using the K-means clustering algorithm to generate multiple cluster center vectors. The cluster center vectors are used as visual codewords, and multiple visual codewords form a visual codeword set. Each visual codeword is associated with the semantic role slot that was anchored when the visual codeword was generated and the corresponding role mapping record.
[0023] Preferably, in S2, a semantically anchored visual codebook is generated based on the role mapping record and the visual codeword set, as follows:
[0024] Based on the role mapping record, visual segment feature vectors that are still merged into the same visual codeword under different semantic role slot anchoring are obtained. The visual segment feature vectors and their corresponding semantic role slots are used as supervised samples to perform learning vector quantization processing to generate multiple prototype vectors. The prototype vectors are used to aggregate visual segment feature vectors with the same semantic role slots, and multiple prototype vectors form a prototype vector set. The semantic role slots of the visual segment feature vectors aggregated by the prototype vectors are used as the role labels of the prototype vectors. The role labels are associated with the corresponding visual codewords to form discriminative visual codewords.
[0025] All discriminative visual codewords are acquired to form a semantically anchored visual codebook;
[0026] The purpose of semantically anchored visual codebooks is to ensure that visual codewords corresponding to different semantic role slots remain distinguishable during subsequent index calls.
[0027] Preferably, in S3, a cross-modal anchoring index that can be used for retrieval is generated based on the visual codeword set and the semantic frame sequence, as follows:
[0028] Extract the feature vector of each visual codeword in the visual codeword set;
[0029] The overall features of the visual codeword set are obtained by global average pooling of the feature vectors of all visual codewords; the overall features of the visual codeword set are used as the image-side anchoring vector.
[0030] The semantic role slots in the semantic framework sequence are generated into slot embedding vectors using the ELMo algorithm, and the embedding vectors of all slots are pooled to obtain the text-side anchoring vectors.
[0031] For illustration, the ELMo algorithm is an existing language model;
[0032] Image-side anchor vectors and text-side anchor vectors are written together into an index, where each image-side anchor vector is associated with its corresponding semantic role label, and each text-side anchor vector is associated with its corresponding semantic role slot and semantic role label, forming a cross-modal anchor index.
[0033] The multimodal retrieval method combining image features and semantic understanding proposed in this invention has the following beneficial technical effects:
[0034] This application employs a multimodal retrieval method that combines image features with semantic understanding. Specifically, it uses image segmentation to generate a visual segment feature set for each image in the image data, and semantic role annotation to generate a semantic frame sequence for each text in the text data. This design fully leverages the semantic information in both image and text data, achieving deep fusion of images and text by generating semantically anchored visual codebooks and cross-modal anchoring indexes. Specifically, it generates a permutation semantic frame sequence and role mapping records based on the semantic frame sequence, and then generates a set of visual codewords, which in turn generates a semantically anchored visual codebook based on the role mapping records. This method can more accurately correspond and align visual and semantic information in images and text, alleviating the information loss and low retrieval accuracy problems caused by independent processing of image and text data in existing technologies, significantly improving the accuracy and efficiency of multimodal retrieval. Simultaneously, by using anchor vectors from both the image and text sides for indexing, it tightly integrates visual codewords and semantic role slots, establishing more accurate cross-modal matching for matching and indexing between different modalities. Attached Figure Description
[0035] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0036] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0037] like Figure 1 The multimodal retrieval method shown includes the following steps: (Image features combined with semantic understanding)
[0038] S1. Obtain the multimodal data to be retrieved, which includes image data and text data;
[0039] For each image in the image data, image segmentation techniques are used to generate a visual segment feature set corresponding to each image;
[0040] Semantic role labeling is applied to each text in the text data to generate a semantic framework sequence corresponding to each text.
[0041] In an optional embodiment, in S1, image segmentation technology is applied to each image in the image data to generate a visual segment feature set corresponding to each image, as follows:
[0042] Each image in the image data is segmented into multiple visual segments using image segmentation techniques, and each visual segment is appended with a segment location description and segment type identifier; a convolutional neural network is used to obtain the segment feature vector corresponding to each visual segment;
[0043] Obtain all visual segments and their corresponding segment feature vectors to form a visual segment feature set;
[0044] In an optional embodiment, in S1, semantic role labeling is applied to each text in the text data to generate a semantic framework sequence corresponding to each text, as follows:
[0045] Semantic role labeling is applied to each text in the text data to generate multiple semantic frames corresponding to each text. The multiple semantic frames are arranged in the order of their appearance to form a semantic frame sequence.
[0046] The semantic framework includes at least one semantic role slot, a semantic role tag corresponding to the role type of the semantic role slot, and the slot content corresponding to the semantic role slot; wherein, the semantic role tag is generated from the field representing the role type in the semantic role annotation result;
[0047] This method processes image and text data independently, extracting visual fragment feature sets for each image and semantic frame sequences for each text, thus facilitating subsequent semantic fusion of image and text data. The visual fragment feature sets generated from image data through image segmentation effectively capture both local and overall image information, providing detailed visual features for subsequent image-text matching. The semantic frame sequences generated from text data through semantic role annotation more accurately describe the semantic structure of the text and provide explicit semantic role labels and slot content for image-text matching. This image and text feature processing improves subsequent retrieval accuracy and semantic matching precision.
[0048] S2. Based on the semantic frame sequence, using the semantic frame sequence as the original semantic frame sequence, generate each permuted semantic frame sequence and role mapping record corresponding to the original semantic frame sequence.
[0049] Based on each permuted semantic framework sequence and role mapping record corresponding to the original semantic framework sequence, a set of visual codewords is generated.
[0050] Based on the role mapping record and the visual codeword set, a semantically anchored visual codebook is generated;
[0051] In an optional embodiment, in S2, based on the semantic frame sequence, using the semantic frame sequence as the original semantic frame sequence, each permutation semantic frame sequence and role mapping record corresponding to the original semantic frame sequence are generated, as follows:
[0052] Based on the semantic role slots in the semantic frame sequence, the semantic role labels of at least two semantic role slots in each semantic frame sequence are interchanged to generate at least one permuted semantic frame sequence with the same slot content as the original semantic frame sequence but different semantic role label correspondences. The semantic role label correspondences before and after the interchange are recorded to form a role mapping record.
[0053] In an optional embodiment, in S2, a visual codeword set is generated based on each permutation semantic frame sequence and role mapping record corresponding to the original semantic frame sequence, as follows:
[0054] Using the semantic role slots in the original semantic framework sequence and each of the permuted semantic framework sequences as anchor points, the K-means clustering algorithm is used to cluster the visual segment feature vectors in the same visual segment feature set to generate multiple cluster center vectors. The cluster center vectors are used as visual codewords, and multiple visual codewords form a visual codeword set. Each visual codeword is associated with the semantic role slot that was anchored when the visual codeword was generated and the corresponding role mapping record.
[0055] In an optional embodiment, in S2, a semantically anchored visual codebook is generated based on the role mapping record and the visual codeword set, as follows:
[0056] Based on the role mapping record, visual segment feature vectors that are still merged into the same visual codeword under different semantic role slot anchoring are obtained. The visual segment feature vectors and their corresponding semantic role slots are used as supervised samples to perform learning vector quantization processing to generate multiple prototype vectors. The prototype vectors are used to aggregate visual segment feature vectors with the same semantic role slots, and multiple prototype vectors form a prototype vector set. The semantic role slots of the visual segment feature vectors aggregated by the prototype vectors are used as the role labels of the prototype vectors. The role labels are associated with the corresponding visual codewords to form discriminative visual codewords.
[0057] All discriminative visual codewords are acquired to form a semantically anchored visual codebook;
[0058] The purpose of semantically anchored visual codebooks is to ensure that visual codewords corresponding to different semantic role slots remain distinguishable during subsequent index calls.
[0059] This method further improves the semantic alignment between images and text by permuting semantic framework sequences and generating role mapping records. By mapping different semantic role slots to visual segments in the visual segment feature set, visual codewords are generated using the K-means clustering algorithm. Furthermore, a semantically anchored visual codebook is generated based on the semantic role slots and role mapping records. This visual codebook not only aggregates visual segments with the same semantic role slots but also preserves semantic distinctions, enabling each visual codeword to be mapped more accurately to its corresponding semantic role label. Ultimately, generating a semantically anchored visual codebook allows subsequent cross-modal retrieval to perform more precise indexing based on semantic information.
[0060] S3. Generate a cross-modal anchored index that can be used for retrieval based on the visual codeword set and semantic frame sequence.
[0061] In an optional embodiment, in S3, a cross-modal anchoring index for retrieval is generated based on the visual codeword set and the semantic frame sequence, as follows:
[0062] Extract the feature vector of each visual codeword in the visual codeword set;
[0063] The overall features of the visual codeword set are obtained by global average pooling of the feature vectors of all visual codewords; the overall features of the visual codeword set are used as the image-side anchoring vector.
[0064] The semantic role slots in the semantic framework sequence are generated into slot embedding vectors using the ELMo algorithm, and the embedding vectors of all slots are pooled to obtain the text-side anchoring vectors.
[0065] For illustration, the ELMo algorithm is an existing language model;
[0066] Image-side anchor vectors and text-side anchor vectors are written together into an index, where each image-side anchor vector is associated with its corresponding semantic role label, and each text-side anchor vector is associated with its corresponding semantic role slot and semantic role label, forming a cross-modal anchor index.
[0067] By combining visual codeword sets and semantic frame sequences, this method significantly improves retrieval efficiency and accuracy by generating a cross-modal anchoring index. Anchor vectors on both the image and text sides are extracted using global average pooling, and the ELMo algorithm is used to generate slot embedding vectors for the text, ensuring good alignment between the semantic representations of images and text. The anchor vectors of both images and text are jointly written into the index, maintaining their association with semantic role labels, thus achieving cross-modal retrieval of image and text data. This index not only improves accuracy during the retrieval process but also ensures semantic consistency, making the retrieval results more accurate and reliable.
[0068] This application employs a multimodal retrieval method that combines image features with semantic understanding. Specifically, it uses image segmentation to generate a visual segment feature set for each image in the image data, and semantic role annotation to generate a semantic frame sequence for each text in the text data. This design fully leverages the semantic information in both image and text data, achieving deep fusion of images and text by generating semantically anchored visual codebooks and cross-modal anchoring indexes. Specifically, it generates a permutation semantic frame sequence and role mapping records based on the semantic frame sequence, and then generates a set of visual codewords, which in turn generates a semantically anchored visual codebook based on the role mapping records. This method can more accurately correspond and align visual and semantic information in images and text, alleviating the information loss and low retrieval accuracy problems caused by independent processing of image and text data in existing technologies, significantly improving the accuracy and efficiency of multimodal retrieval. Simultaneously, by using anchor vectors from both the image and text sides for indexing, it tightly integrates visual codewords and semantic role slots, establishing more accurate cross-modal matching for matching and indexing between different modalities.
[0069] For clarification, "acquisition" in this application refers to obtaining the required content or data using existing technical means.
[0070] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.
[0071] In the embodiments provided by this invention, it should be understood that the disclosed system or method can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for instance, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.
[0072] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0073] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.
[0074] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic characteristics of the present invention.
[0075] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A multimodal retrieval method combining image features and semantic understanding, characterized in that, Includes the following steps: S1. Obtain the multimodal data to be retrieved, which includes image data and text data; apply image segmentation technology to each image in the image data to generate a visual segment feature set corresponding to each image; apply semantic role labeling to each text in the text data to generate a semantic frame sequence corresponding to each text. S2. Based on the semantic frame sequence, using the semantic frame sequence as the original semantic frame sequence, generate each permuted semantic frame sequence and role mapping record corresponding to the original semantic frame sequence; based on each permuted semantic frame sequence and role mapping record corresponding to the original semantic frame sequence, generate a visual codeword set; based on the role mapping record and visual codeword set, generate a semantically anchored visual codebook. S3. Generate a cross-modal anchored index that can be retrieved based on the visual codeword set and semantic frame sequence; In S1, semantic role labeling is applied to each text in the text data to generate a semantic frame sequence corresponding to each text, as follows: Semantic role labeling is applied to each text in the text data to generate multiple semantic frames corresponding to each text. The multiple semantic frames are arranged in the order of their appearance to form a semantic frame sequence. The semantic framework includes at least one semantic role slot, a semantic role tag corresponding to the role type of the semantic role slot, and the slot content corresponding to the semantic role slot; wherein, the semantic role tag is generated from the field representing the role type in the semantic role annotation result; In S2, based on the semantic frame sequence, and using the semantic frame sequence as the original semantic frame sequence, each permutation semantic frame sequence and role mapping record corresponding to the original semantic frame sequence are generated, as follows: Based on the semantic role slots in the semantic frame sequence, the semantic role labels of at least two semantic role slots in each semantic frame sequence are interchanged to generate at least one permuted semantic frame sequence with the same slot content as the original semantic frame sequence but different semantic role label correspondences. The semantic role label correspondences before and after the interchange are recorded to form a role mapping record. In S2, a visual codeword set is generated based on each permutation semantic frame sequence and role mapping record corresponding to the original semantic frame sequence, as follows: Using the semantic role slots in the original semantic framework sequence and each of the permuted semantic framework sequences as anchor points, the K-means clustering algorithm is used to cluster the visual segment feature vectors in the same visual segment feature set to generate multiple cluster center vectors. The cluster center vectors are used as visual codewords, and multiple visual codewords form a visual codeword set. Each visual codeword is associated with the semantic role slot that was anchored when the visual codeword was generated and the corresponding role mapping record. In S2, a semantically anchored visual codebook is generated based on the role mapping record and the visual codeword set, as follows: Based on the role mapping record, visual segment feature vectors that are still merged into the same visual codeword under different semantic role slot anchoring are obtained. The visual segment feature vectors and their corresponding semantic role slots are used as supervised samples to perform learning vector quantization processing to generate multiple prototype vectors. The prototype vectors are used to aggregate visual segment feature vectors with the same semantic role slots, and multiple prototype vectors form a prototype vector set. The semantic role slots of the visual segment feature vectors aggregated by the prototype vectors are used as the role labels of the prototype vectors. The role labels are associated with the corresponding visual codewords to form discriminative visual codewords. All discriminative visual codewords are acquired to form a semantically anchored visual codebook.
2. The multimodal retrieval method combining image features and semantic understanding according to claim 1, characterized in that, In S1, image segmentation techniques are applied to each image in the image data to generate a visual segment feature set corresponding to each image, as follows: Each image in the image data is segmented into multiple visual segments using image segmentation techniques, and each visual segment is appended with a segment location description and segment type identifier; a convolutional neural network is used to obtain the segment feature vector corresponding to each visual segment; Obtain all visual segments and their corresponding segment feature vectors to form a visual segment feature set.
3. The multimodal retrieval method combining image features and semantic understanding according to claim 1, characterized in that, In S3, a cross-modal anchored index is generated based on the visual codeword set and the semantic frame sequence, as follows: Extract the feature vector of each visual codeword in the visual codeword set; The overall features of the visual codeword set are obtained by global average pooling of the feature vectors of all visual codewords; the overall features of the visual codeword set are used as the image-side anchoring vector. The semantic role slots in the semantic framework sequence are generated into slot embedding vectors using the ELMo algorithm, and the embedding vectors of all slots are pooled to obtain the text-side anchoring vectors. Image-side anchor vectors and text-side anchor vectors are written together into an index, where each image-side anchor vector is associated with its corresponding semantic role label, and each text-side anchor vector is associated with its corresponding semantic role slot and semantic role label, forming a cross-modal anchor index.