A multimodal data retrieval method and device based on metric index

Abstract

The present invention discloses a multimodal data retrieval method and device based on metric index, the method comprising: acquiring multimodal data, each multimodal data comprising data elements of different modes and a timestamp generated by the multimodal data, the multimodal data comprising several of text modal data, image modal data, video modal data, and geographic information modal data; extracting key attribute information of the multimodal data; optimizing the key attribute information to obtain optimized features, thereby forming a multimodal embedding vector based on the key attribute information and the optimized features; constructing a multi-index structure framework based on SPBTree, Mtree and M3Tree according to the multimodal embedding vector; acquiring query information input by a user, extracting and optimizing key attribute information of the query information to obtain a query vector, matching the query vector based on the multi-index structure framework, thereby realizing multimodal data retrieval.

Description

Technical Field

[0001] The present invention belongs to the technical field of data retrieval, and in particular to a multimodal data retrieval method and device based on metric indexing. Background Art

[0002] In recent years, with the rapid development of information technology, the generation and application of multimodal data have become more and more extensive. Multimodal data refers to a collection of data consisting of multiple different types, such as text, images, videos, and audio. These data play an important role in many application scenarios. For example, in social media analysis, user posts, pictures, and videos are all multimodal. Posts published on social media software are also typical representatives of multimodal data.

[0003] With the continuous growth of multimodal data, how to effectively retrieve and utilize this data has become an important research topic. Traditional data retrieval techniques mainly focus on single-modal data, such as text retrieval, image retrieval, etc. These technologies have high efficiency and accuracy when processing single-type data, but they are powerless when facing multimodal data. This is because multimodal data has significant differences in features, structure, and content, and traditional retrieval techniques find it difficult to simultaneously process and integrate these differences.

[0004] The main challenge of multimodal data retrieval is how to effectively fuse and utilize data features of different modalities to achieve efficient and accurate retrieval. Most of the existing multimodal data retrieval technologies adopt methods based on feature extraction and fusion, which extract data features of different modalities and map these features into the same feature space for similarity measurement. However, these methods have many problems in practical applications. First, the data features of different modalities have essential differences in structure and content, and how to design effective feature extraction and fusion methods is a difficulty. Secondly, existing methods usually require a large amount of labeled data for training, and data labeling is costly and difficult to obtain. In addition, with the continuous growth of data volume, the efficiency and accuracy of retrieval are also facing huge challenges.

[0005] In response to these problems, some new multimodal data retrieval methods have been proposed in recent years. For example, the method based on graph neural network constructs the graph structure of multimodal data and uses graph neural network for feature extraction and similarity measurement. This method can effectively integrate data features of different modalities and improve the accuracy and efficiency of retrieval. However, these methods still have some problems in practical applications, such as high computational complexity and limited processing capabilities for large-scale data. Therefore, how to design an efficient and accurate multimodal data retrieval method is still a difficult problem that needs to be solved urgently. Summary of the invention

[0006] In view of the deficiencies in the prior art, the purpose of the embodiments of the present application is to provide a multimodal data retrieval method and device based on metric indexing to improve the efficiency and accuracy of information retrieval.

[0007] According to a first aspect of an embodiment of the present application, a multimodal data retrieval method based on a metric index is provided, comprising:

[0008] Acquire multimodal data, each piece of multimodal data including data elements of different modalities and a timestamp of generation of the multimodal data, the multimodal data including several of text modal data, image modal data, video modal data, and geographic information modal data;

[0009] Extracting key attribute information of the multimodal data;

[0010] Optimizing the key attribute information to obtain optimized features, thereby forming a multimodal embedding vector based on the key attribute information and the optimized features;

[0011] According to the multimodal embedding vector, a multi-index structure framework is constructed based on SPBTree, Mtree and M3Tree;

[0012] The query information input by the user is obtained, key attribute information is extracted and optimized for the query information to obtain a query vector, and the query vector is matched based on the multi-index structure framework, thereby realizing multimodal data retrieval.

[0013] Furthermore, extracting key attribute information of the multimodal data includes one or more of the following:

[0014] For text modality data, conduct word frequency analysis, word meaning association analysis, grammatical structure analysis, and sentiment analysis;

[0015] For image modality data, color distribution is analyzed using color histogram or color moment, and texture features are extracted through gray-level co-occurrence matrix or local binary pattern;

[0016] For video modality data, extract the image features of each key frame and analyze the motion patterns and time series changes in the video, including identifying scene changes and activity patterns through motion vectors and inter-frame differences, and capturing key objects and human expressions in the video through face and object recognition;

[0017] For geographic location data, use GIS techniques to identify and parse data in geographic coordinate systems, including extracting location information from metadata and using geocoding techniques to convert place names into coordinates;

[0018] The key attribute information of each mode is intensively integrated to obtain multimodal comprehensive features.

[0019] Furthermore, the key attribute information is optimized, including one or more of the following:

[0020] Visual feature optimization: Use machine learning to generate an embedding vector for key attribute information of image modality data and / or video modality data, and use an autoencoder or a generative adversarial network to process the embedding vector to obtain optimized visual features;

[0021] Feature dimensionality reduction: Reduce the dimensionality of key attribute information of image modality data and video modality data;

[0022] Feature fusion: For key attribute information corresponding to video modal data, long short-term memory networks or 3D convolutional networks are used to fuse the spatial features and temporal features. For key attribute information corresponding to text modal data, time label analysis is performed. For key attribute information corresponding to image modal data, sequence image analysis and dynamic scene reconstruction are performed. For key attribute information corresponding to geographic information modal data, spatiotemporal trajectory analysis is performed.

[0023] Furthermore, based on the multimodal embedding vector, a multi-level index structure framework is constructed, including:

[0024] Indexing the multimodal embedding vector;

[0025] Based on the indexed multimodal embedding vector, an initial index structure is constructed, wherein the initial index structure includes an SPBTree index structure, an Mtree index structure and an M3Tree index structure, wherein the M3Tree index structure is used for indexing multimodal comprehensive features in the multimodal embedding vector, the SPBTree index structure is used for indexing features corresponding to geographic information modality data in the multimodal embedding vector, and the Mtree index structure is used for indexing remaining features in the multimodal embedding vector;

[0026] The initial index structure is optimized based on performance to form a dynamically updated index structure.

[0027] Furthermore, the multimodal embedding vector is indexed, specifically:

[0028] An identifier of a shift is set for the multimodal embedding vector, and an index entry is constructed based on the identifier and the multimodal embedding vector.

[0029] Furthermore, matching the query vector is performed based on the multi-index structure framework, specifically:

[0030] Based on the multi-index structure framework, evaluating the similarity between the query vector and each multimodal embedding vector;

[0031] Among them, for the query vector corresponding to the query information in the text modality, the similarity is evaluated by the edit distance method; for the query vector corresponding to the query information in the image modality and the video modality, the similarity is evaluated by the cosine distance, Manhattan distance or Euclidean distance; for the query vector corresponding to the query information in the geographic information modality, the similarity is evaluated by the Euclidean distance.

[0032] According to a second aspect of an embodiment of the present application, a multimodal data retrieval device based on a metric index is provided, comprising:

[0033] An acquisition module, used to acquire multimodal data, each piece of multimodal data includes data elements of different modalities and a timestamp when the multimodal data is generated, and the multimodal data includes several of text modal data, image modal data, video modal data, and geographic information modal data;

[0034] An attribute extraction module, used to extract key attribute information of the multimodal data;

[0035] A feature optimization module, used to optimize the key attribute information to obtain optimized features, thereby forming a multimodal embedding vector based on the key attribute information and the optimized features;

[0036] An index building module, used for building a multi-index structure framework based on SPBTree, Mtree and M3Tree according to the multimodal embedding vector;

[0037] The retrieval module is used to obtain the query information input by the user, extract and optimize the key attribute information of the query information, obtain the query vector, and match the query vector based on the multi-index structure framework, thereby realizing multimodal data retrieval.

[0038] According to a third aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0039] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, including:

[0040] one or more processors;

[0041] A memory for storing one or more programs;

[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0043] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0044] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0045] As can be seen from the above embodiments, the present application optimizes and refines the attribute representation of multimodal data, and constructs a multi-level index structure and a metric index model to achieve efficient information retrieval and matching. By constructing a multi-level metric index structure, the computational complexity of data retrieval is effectively reduced. In this structure, the top-level index quickly filters most of the irrelevant data, while the deeper index accurately handles complex queries. This hierarchical approach significantly improves query efficiency and data processing speed. Through refined multimodal data attribute analysis and optimization, the system can more accurately capture and characterize key information in multimodal data. This not only improves the organization of data, but also reduces unnecessary computational burdens in the indexing process, thereby reducing the overall computational complexity. The system uses advanced query processing algorithms, combined with dynamic index updates and intelligent caching mechanisms, to further improve the performance when processing large-scale data sets. This enables the system to process larger data sets while maintaining high response speeds, expanding the scope of application and practicality of the system.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 The present invention is a flowchart of a multimodal data retrieval method based on metric index according to an exemplary embodiment.

[0049] Figure 2 It is a block diagram of a multimodal data retrieval device based on metric index according to an exemplary embodiment.

[0050] Figure 3 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0051] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0052] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0053] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0054] Figure 1 is a flowchart of a multimodal data retrieval method based on metric index according to an exemplary embodiment. Figure 1 As shown, the method is applied in a terminal and may include the following steps:

[0055] In the specific implementation of step S11, multimodal data is obtained, each piece of multimodal data includes data elements of different modalities and a timestamp when the multimodal data is generated, and the multimodal data includes text modal data, image modal data, video modal data, and geographic information modal data;

[0056] Specifically, in a multimodal information system, a large amount of different types of data are generated and interacted at every moment, and these data may involve multiple modalities such as text, images, and videos. In specific implementations, multimodal data is usually collected at regular intervals, and the specific collection cycle can be dynamically adjusted according to actual needs.

[0057] In social media analysis applications, users' posts can be actively and continuously tracked and collected, and data sent by users can also be received. Posts may contain text (text modality), uploaded pictures (image modality), published videos (video modality), and the user's location at the time of publication (geolocation modality data), which can be represented by a multi-tuple (text, image, video, geolocation, time).

[0058] In the specific implementation of step S12, key attribute information of the multimodal data is extracted;

[0059] Specifically, the key attribute information is used to subsequently construct a unique identifier or feature vector for each multimodal data, thereby making the multimodal data easier to index and query in the multimodal database.

[0060] Before extracting key attribute information, data cleaning and preprocessing are required, such as removing noise, standardizing the format, extracting data from various modalities, etc., to improve the accuracy and efficiency of attribute extraction. In addition, various data enhancement techniques can be used to expand the original feature set. This may include generating synthetic data (such as by rotating and scaling images), or applying techniques such as synonym replacement in text to increase the diversity of data.

[0061] Specifically, this step is performed according to pre-set extraction rules and extraction algorithms, where the extraction rules define which attributes need to be extracted, such as keywords, author information, publication date, text length, language type, and text structure in text modality data; color histogram, texture features, shape descriptors, and object tags in image modality data; frame rate, resolution, key frame content, motion vectors, and geographic location data in video modality data.

[0062] For text modal data, natural language processing algorithms can be used to analyze text content, extract keywords and topics, etc., which include but are not limited to: word frequency analysis: calculate the frequency of occurrence of each word in the text and identify high-frequency words, which helps to initially understand the theme or content focus of multimodal data; word meaning association analysis: through natural language processing technology, such as Word2Vec or BERT, analyze the semantic relationship between words and identify semantic patterns in multimodal data; grammatical structure analysis: evaluate the grammatical structure of the text, such as the organization of phrases and clauses, to further reveal the information structure of the text; sentiment analysis: perform sentiment tendency analysis on the text and identify positive or negative emotional expressions, which is particularly important when processing user-generated content.

[0063] For image modal data, image processing algorithms can be used to extract color and texture features. In one embodiment, a color histogram or color moment is used to analyze the color distribution in an image, and texture features are extracted through methods such as gray-level co-occurrence matrices or local binary patterns, thereby helping to identify the visual style and subject matter of the image.

[0064] For video modality data, video analysis techniques can be used to identify key frames, motion trajectories, and scene changes in the video. Not only can the image features of each key frame be extracted, but also the motion patterns and time series changes in the video can be analyzed. For example, scene changes and activity patterns can be identified through motion vectors and inter-frame differences, which helps to understand the dynamic nature of the video content. In addition, facial and object recognition is performed to capture key objects and character expressions in the video and enhance the contextual understanding of the content.

[0065] For geographic location data, Geographic Information System (GIS) technology is used to identify and interpret data in a geographic coordinate system. This includes extracting location information from metadata and using geocoding technology to convert place names into precise coordinates.

[0066] Based on the above cross-modal attribute extraction of multimodal data, the key attribute information corresponding to each modal data is integrated to form a unified comprehensive representation, which helps to conduct in-depth analysis and respond more accurately to complex query requirements. Such multimodal analysis ensures that the system can fully interpret and utilize the information in the data from all angles, providing a solid foundation for subsequent indexing and retrieval tasks.

[0067] In specific implementations, the attribute analysis algorithm can be designed modularly, allowing the addition or modification of attribute extraction rules and algorithms to adapt to changing business needs.

[0068] It should be noted that the attributes to be extracted as defined in the extraction rules can be optimized and verified: the extracted attributes can be refined and adjusted, and the contribution of each attribute to the model and the correlation between them can be evaluated through statistical analysis methods (such as correlation coefficient calculation, feature importance evaluation, and regression analysis). At the same time, sample tests (including cross-validation and repeated experiments) can be used to verify the consistency and stability of the extraction of these attributes on different data sets, ensuring the effectiveness and accuracy of the extracted attributes in practical applications.

[0069] In step S13, the key attribute information is optimized and refined to obtain optimized features, thereby forming a multimodal embedding vector based on the key attribute information and the optimized features;

[0070] Specifically, this step aims to eliminate information redundancy, highlight key features, and improve the distinguishing ability of features, which is crucial for subsequent multimodal data matching and similarity calculation, especially when facing large-scale multimodal data.

[0071] For the key attribute information extracted in step S12, the quality of the embedded representation can be further optimized by the following methods:

[0072] Feature weighting and selection: Adjust the weights according to the importance of key attribute information (for example, based on the score of information gain or feature selection algorithm). The weight adjustment will directly affect the accuracy and discrimination of the embedded representation of multimodal data, thereby optimizing the efficiency of index construction and query response, so that the system can match and retrieve relevant multimodal data more accurately and quickly. For example, key objects in object recognition (such as brand logos or facial information) may be more important than background elements, so they should be given higher weights, which will have a greater impact on the search results (such as similarity calculation results) during the search process. In addition, feature selection algorithms (such as methods based on information gain or mutual information) are used to screen out the most representative and discriminative visual features and remove redundant and irrelevant features.

[0073] Visual feature optimization: Use advanced machine learning techniques (such as multi-layer perceptrons or deep convolutional networks) to integrate key attribute information of image modality data and / or video modality data to generate an embedding vector that can express complex visual and dynamic information. This integrated approach helps capture deep information and subtle connections in image and video modality data. In addition, autoencoders or generative adversarial networks (GANs) are used to further refine and abstract feature vectors to improve their expressiveness and make them more suitable for complex query needs.

[0074] Feature dimensionality reduction: Use PCA, t-SNE and other dimensionality reduction techniques to reduce the feature dimensions of image modality data and video modality data to reduce computational complexity while retaining the key information of the data. For data with lower dimensions such as text modality data and geographic information modality data, dimensionality reduction processing is not required.

[0075] Feature fusion: For video modality data, spatial features (such as objects and colors in image frames) are fused with temporal features (such as the motion or activity patterns of objects), using, for example, long short-term memory networks (LSTM) or 3D convolutional networks, to ensure that the continuity and context of the time series are effectively captured. This fusion ensures that the embedding vector reflects not only the information of a single frame, but also the dynamic changes and development trends of the video content. For text modality data, time label analysis is performed to analyze time-sensitive text content (such as social media dynamics posted by the same author at different times) to track the development trend of text content or time series analysis. For image modality data, sequence image analysis is performed, that is, time series images from the same location are analyzed, such as image sequences captured by surveillance cameras, to analyze scenes or activities under time changes; in addition, dynamic scene reconstruction is required: dynamic scenes are reconstructed through a collection of images marked with timestamps to understand how scenes change over time. For geographic data, spatiotemporal trajectory analysis is performed: the movement trajectory of individuals or groups in geographic space over time is analyzed, such as the spatiotemporal distribution of migration paths and traffic flow.

[0076] Based on the aforementioned extracted key attribute information and optimized features, a multimodal embedding vector is formed.

[0077] In step S14, a multi-index structure framework is constructed according to the multimodal embedding vector;

[0078] Specifically, the following sub-steps may be included:

[0079] S41: Indexing the multimodal embedding vector;

[0080] The multimodal embedding vector is converted into an index entry, each of which contains an identifier of the corresponding multimodal data and its embedding vector. The "identifier" in each index entry is a unique identification code, usually a multimodal data ID, a URL or a file path of the multimodal data. These identifiers enable the system to quickly locate and reference specific multimodal data, whether they are stored locally or on a network server. The multimodal data ID is usually a unique number or string used for precise queries in the database, while the URL or path provides a direct way to access the multimodal data, ensuring that the corresponding multimodal data can be quickly found and processed when performing search queries. This identification method not only optimizes the multimodal data management process, but also enhances the efficiency and responsiveness of the retrieval system. In this way, when a search query is performed, the most relevant multimodal data can be quickly retrieved through vector similarity calculation.

[0081] S42: constructing an initial index structure based on the indexed multimodal embedding vector, wherein the initial index structure includes an SPBTree index structure, an Mtree index structure, and an M3Tree index structure, wherein the M3Tree index structure is used for indexing multimodal comprehensive features in the multimodal embedding vector, and the SPBTree index structure and the Mtree index structure are used for indexing remaining features in the multimodal embedding vector;

[0082] In this step, the index structure is adjusted according to the real-time changes of social media data and query requirements, such as dynamically inserting new content and adjusting index parameters to maintain index efficiency and accuracy. Based on the multimodal embedding vector, a specific index structure is used to build the index:

[0083] Applications of SPBTree: SPBTree, or spatial-probabilistic B-tree, is suitable for processing data containing spatial or probabilistic information, and can effectively manage and index multimodal data with geographic location information or probabilistic distribution characteristics, such as location tags in image or video modal data. By combining the spatial attributes and content attributes of multimodal data, SPBTree can support complex query operations such as range search and nearest neighbor search. In one embodiment, SPBTree is used to index multimodal data containing spatial attributes, such as geo-tagged social media posts or location-related news articles.

[0084] Application of MTree: MTree is a metric-based tree structure used to manage distance-based similarity calculations, applicable to any form of modal data, such as text or images. Mtree uses the distance function between data points to organize data, so it is suitable for performing content-based search tasks, such as finding the multimodal dataset that is most similar to the content of a given multimodal data. This structure will be used to build an index that can effectively handle high-dimensional spatial data, thereby supporting complex similarity queries.

[0085] Application of M3Tree: M3Tree is an extension of MTree that supports multi-dimensional data indexing. In a multimodal system, M3Tree can effectively index multimodal data that combines multiple modal features, such as multimodal data that contains text, image, and video modal data at the same time. Therefore, M3Tree is used to index multimodal comprehensive features.

[0086] S43: Optimizing the initial index structure based on performance, dynamically updating the index structure, and forming a multi-index structure framework;

[0087] After the initial index is built, the index structure will be further optimized according to the actual retrieval requirements and query rules. Specifically, the retrieval performance of the existing index structure will be evaluated and optimized according to the requirements. This may include adjusting index parameters, such as the setting of branching factor in MTree and M3Tree, or optimizing data partitioning and index splitting strategies in SPBTree. In MTree and M3Tree, the adjustment of branching factor is the key, which determines the depth of the tree and the number of child nodes of a node. Properly increasing the branching factor can reduce the depth of the tree and improve the retrieval speed, but it may increase the node width, affecting the node loading and maintenance costs. Relatively speaking, reducing the branching factor will increase the depth of the tree, which may lead to a decrease in retrieval speed, but it will help reduce the amount of data in a single node and improve data loading efficiency. For SPBTree, optimizing data partitioning and index splitting strategies is mainly to manage spatial and probabilistic information more efficiently. By fine-tuning the data partitioning rules and improving the index splitting logic, the load of the index tree and the speed of responding to queries can be effectively balanced. These adjustments make the index structure more adaptable to different types of query requirements and improve the performance and scalability of the overall system.

[0088] It should be noted that the index structure formed should have the characteristics of dynamic adjustment to cope with the increase or change of data. This includes the strategy of inserting new nodes in MTree and M3Tree, and the method of managing the change of spatial and probabilistic information in SPBTree. In the multimodal attitude metric indexing system, dynamic index adjustment ensures that the system can adapt to the increase or change of data. Index updates are usually performed when new data is added to the system, existing data is modified or deleted, and during regular system maintenance. The specific update mechanism includes inserting new nodes in MTree and M3Tree to contain new or changed data, and adjusting data partitions and updating index structures in SPBTree to reflect changes in spatial and probabilistic information. Such dynamic updates not only ensure the timeliness and accuracy of the index, but also optimize the system's response speed and ability to handle large amounts of data, so that the system can efficiently handle various queries from simple keyword searches to similarity searches based on complex features. In this way, the multimodal attitude metric indexing system can effectively support various types of queries, from simple keyword searches to similarity searches based on complex features, while ensuring the scalability and efficiency of the system.

[0089] In a specific implementation of S15, query information input by a user is obtained, key attribute information is extracted and optimized for the query information to obtain a query vector, and the query vector is matched based on the multi-index structure framework, thereby realizing multimodal data retrieval;

[0090] S51: obtaining query information input by a user, extracting and optimizing key attribute information of the query information, and obtaining a query vector;

[0091] Specifically, the query information is processed in steps S12 and S13 to obtain a query vector.

[0092] S52: Based on the multi-index structure framework, evaluate the similarity between the query vector and each multimodal embedding vector to obtain a multimodal retrieval result;

[0093] Specifically, appropriate similarity metrics, such as cosine similarity or Euclidean distance, are used to evaluate and compare the similarity between embedding vectors. This ensures the relevance and accuracy of the retrieval results.

[0094] This application is applicable to two types of queries: range query and K nearest neighbor query, which can cope with different search requirements and optimize retrieval efficiency. Range query allows users to specify a specific query range, which is suitable for scenarios where all relevant items need to be found within a certain space, time or any measurement range, for example, finding events in a specific area in a geographic information system. K nearest neighbor query finds the K items closest to the query point based on the distance metric, which is suitable for use when the user needs to find the most similar objects, such as finding the content most similar to the target user's interests in a recommendation system. These two query methods provide flexible data access methods, allowing the system to more accurately meet the user's specific query needs.

[0095] In the specific implementation, according to the characteristics of the query vector, a suitable similarity measurement method is selected or specified by the user, such as cosine similarity, Euclidean distance or Manhattan distance, and based on an efficient retrieval algorithm (such as the nearest neighbor search algorithm), the multimodal embedding vector most relevant to the query vector is quickly located. These measurement methods will be used to evaluate the similarity between the query vector and each multimodal embedding vector. For example, for a composite query of text and image, the system will consider the feature vectors of both text and image, and use an index structure to accelerate the matching process of related multimodal data.

[0096] If MTree or M3Tree is used as the index framework, the parameters of the tree, such as node splitting strategy and balance conditions, will be adjusted according to the characteristics of the embedding vector of multimodal data. Choosing different measurement methods, such as cosine similarity or Euclidean distance, has a significant impact on the establishment of the index structure. Cosine similarity mainly measures the consistency of angles between vectors. It is suitable for multimodal data embedding that ignores size and focuses on directional similarity. It usually uses inverted indexes or graph structures to optimize the fast matching of similar vectors. In contrast, Euclidean distance focuses on the straight-line distance from point to point. It is suitable for situations where accurate measurement of difference size is required. Index structures such as MTree or M3Tree can adapt to this measurement method by adjusting the node splitting strategy and balance conditions to support complex spatial distance calculations and maintain the efficiency and balance of the index. Such structural adjustments ensure that the index system can effectively support different query requirements based on the characteristics of the embedding vector and improve the accuracy and response speed of retrieval. When establishing SPBTree, MTree and M3Tree index structures, the parameters of the tree, such as branching factor and node splitting strategy, are adjusted according to the characteristics of the embedding vector of multimodal data to adapt to the characteristics and requirements of different data. For example, SPBTree optimizes data partitioning and index splitting strategies for data with spatial or probabilistic information, while MTree and M3Tree adjust node capacity and branching factor for high-dimensional data to ensure that the index structure can not only efficiently handle complex similarity queries but also adapt to the growth of data volume and the complexity of multimodal data.

[0097] In specific implementations, you can also use an intelligent caching mechanism to implement intelligent caching strategies for common queries, reduce repeated access to the database, and improve the overall performance of the system.

[0098] It should be noted that if the user inputs a request to delete or modify data, it is also necessary to locate the corresponding data through the above-mentioned search process and then perform the deletion or modification operation.

[0099] Corresponding to the aforementioned embodiment of the multimodal data retrieval method based on metric index, the present application also provides an embodiment of the multimodal data retrieval device based on metric index.

[0100] Figure 2 is a block diagram of a multimodal data retrieval device based on a metric index according to an exemplary embodiment. Figure 2 , the device may include:

[0101] An acquisition module 21, used to acquire multimodal data, each piece of multimodal data includes data elements of different modalities and a timestamp of the generation of the multimodal data, and the multimodal data includes several of text modal data, image modal data, video modal data, and geographic information modal data;

[0102] An attribute extraction module 22, used to extract key attribute information of the multimodal data;

[0103] A feature optimization module 23, configured to optimize the key attribute information to obtain optimized features, thereby forming a multimodal embedding vector based on the key attribute information and the optimized features;

[0104] An index building module 24, configured to build a multi-index structure framework based on SPBTree, Mtree and M3Tree according to the multimodal embedding vector;

[0105] The retrieval module 25 is used to obtain the query information input by the user, extract and optimize the key attribute information of the query information, obtain the query vector, and match the query vector based on the multi-index structure framework, thereby realizing multimodal data retrieval.

[0106] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0107] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0108] Accordingly, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the multimodal data retrieval method based on metric index as described above.

[0109] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal data retrieval method based on metric index as described above. Figure 3 As shown in FIG. 1 , a hardware structure diagram of a multimodal data retrieval device based on metric index provided by an embodiment of the present invention is provided in any device with data processing capability, except Figure 3In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0110] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the multimodal data retrieval method based on metric index as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0111] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

Claims

1. A multimodal data retrieval method based on metric indexing, characterized in that: include: Acquire multimodal data, each piece of multimodal data including data elements of different modalities and a timestamp of generation of the multimodal data, the multimodal data including several of text modal data, image modal data, video modal data, and geographic information modal data; Extracting key attribute information of the multimodal data; Optimizing the key attribute information to obtain optimized features, thereby forming a multimodal embedding vector based on the key attribute information and the optimized features; According to the multimodal embedding vector, a multi-index structure framework is constructed based on SPBTree, Mtree and M3Tree; Acquire query information input by a user, extract and optimize key attribute information of the query information to obtain a query vector, and match the query vector based on the multi-index structure framework, thereby realizing multimodal data retrieval; According to the multimodal embedding vector, a multi-index structure framework is constructed based on SPBTree, Mtree and M3Tree, including: Indexing the multimodal embedding vector; Based on the indexed multimodal embedding vector, an initial index structure is constructed, wherein the initial index structure includes an SPBTree index structure, an Mtree index structure and an M3Tree index structure, wherein the M3Tree index structure is used for indexing multimodal comprehensive features in the multimodal embedding vector, the SPBTree index structure is used for indexing features corresponding to geographic information modality data in the multimodal embedding vector, and the Mtree index structure is used for indexing remaining features in the multimodal embedding vector; The initial index structure is optimized based on performance to form a dynamically updated index structure.

2. The method according to claim 1, characterized in that Extracting key attribute information of the multimodal data includes one or more of the following: For text modality data, conduct word frequency analysis, word meaning association analysis, grammatical structure analysis, and sentiment analysis; For image modality data, color distribution is analyzed using color histogram or color moment, and texture features are extracted through gray-level co-occurrence matrix or local binary pattern; For video modality data, extract the image features of each key frame and analyze the motion patterns and time series changes in the video, including identifying scene changes and activity patterns through motion vectors and inter-frame differences, and capturing key objects and human expressions in the video through face and object recognition; For geographic location data, use GIS techniques to identify and parse data in geographic coordinate systems, including extracting location information from metadata and using geocoding techniques to convert place names into coordinates; The key attribute information of each mode is intensively integrated to obtain multimodal comprehensive features.

3. The method according to claim 1, characterized in that Optimizing the key attribute information includes one or more of the following: Visual feature optimization: Use machine learning to generate an embedding vector for key attribute information of image modality data and / or video modality data, and use an autoencoder or a generative adversarial network to process the embedding vector to obtain optimized visual features; Feature dimensionality reduction: Reduce the dimensionality of key attribute information of image modality data and video modality data; Feature fusion: For key attribute information corresponding to video modality data, long short-term memory networks or 3D convolutional networks are used to fuse the spatial features with the temporal features. For the key attribute information corresponding to the text modal data, time tag analysis is performed; for the key attribute information corresponding to the image modal data, sequence image analysis and dynamic scene reconstruction are performed; for the key attribute information corresponding to the geographic information modal data, spatiotemporal trajectory analysis is performed.

4. The method according to claim 1, characterized in that The multimodal embedding vector is indexed as follows: An identifier of a shift is set for the multimodal embedding vector, and an index entry is constructed based on the identifier and the multimodal embedding vector.

5. The method according to claim 1, characterized in that The query vector is matched based on the multi-index structure framework, specifically: Based on the multi-index structure framework, evaluating the similarity between the query vector and each multimodal embedding vector; Among them, for the query vector corresponding to the query information in the text modality, the similarity is evaluated by the edit distance method; for the query vector corresponding to the query information in the image modality and the video modality, the similarity is evaluated by the cosine distance, Manhattan distance or Euclidean distance; for the query vector corresponding to the query information in the geographic information modality, the similarity is evaluated by the Euclidean distance.

6. A multimodal data retrieval device based on metric index, characterized in that: include: An acquisition module, used to acquire multimodal data, each piece of multimodal data includes data elements of different modalities and a timestamp when the multimodal data is generated, and the multimodal data includes several of text modal data, image modal data, video modal data, and geographic information modal data; An attribute extraction module, used to extract key attribute information of the multimodal data; A feature optimization module, used to optimize the key attribute information to obtain optimized features, thereby forming a multimodal embedding vector based on the key attribute information and the optimized features; An index building module, used for building a multi-index structure framework based on SPBTree, Mtree and M3Tree according to the multimodal embedding vector; A retrieval module is used to obtain query information input by a user, extract and optimize key attribute information of the query information, obtain a query vector, and match the query vector based on the multi-index structure framework, thereby realizing multimodal data retrieval; According to the multimodal embedding vector, a multi-index structure framework is constructed based on SPBTree, Mtree and M3Tree, including: Indexing the multimodal embedding vector; Based on the indexed multimodal embedding vector, an initial index structure is constructed, wherein the initial index structure includes an SPBTree index structure, an Mtree index structure and an M3Tree index structure, wherein the M3Tree index structure is used for indexing multimodal comprehensive features in the multimodal embedding vector, the SPBTree index structure is used for indexing features corresponding to geographic information modality data in the multimodal embedding vector, and the Mtree index structure is used for indexing remaining features in the multimodal embedding vector; The initial index structure is optimized based on performance to form a dynamically updated index structure.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images