Indexing method and device for mixed storage of multi-modal data and electronic equipment

Abstract

The invention relates to an indexing method and device for mixed storage of multi-modal data and electronic equipment. The method comprises the following steps: preprocessing a multi-modal original data set to obtain metadata and a semantic embedding vector of each piece of original data, and packaging each piece of original data and the metadata and the semantic embedding vector of the original data into a unified multi-modal data object; for each multi-modal data object, constructing a vector index and a structured index according to a modal type, and respectively storing each multi-modal data object to storage media of different hierarchies according to an access frequency and a storage value; analyzing the multi-modal query statement to determine an execution sequence of different types of indexes, and generating a corresponding query execution plan; and performing vector index retrieval and structured index scanning in parallel according to the query execution plan, and performing joint screening and result fusion on retrieval results based on the object identifier to obtain a multi-modal data result. According to the method and the device, the technical problem of index splitting during management of the multi-modal data is solved.

Description

Technical Field

[0001] This application relates to the field of database storage and retrieval technology, and in particular to an indexing method, apparatus, and electronic device for multimodal data hybrid storage. Background Technology

[0002] In current database applications, with the rapid growth of multimodal data, the industry commonly adopts a hybrid retrieval architecture of "vector database + structured database" in parallel collaboration to support the combined use of similarity retrieval for unstructured data such as text, images, and audio with traditional structured conditional queries. This approach typically stores vector and structured data in different database systems and aggregates the results at the application or middleware layer, thus achieving divide-and-conquer management and retrieval of multimodal data. However, this technical approach still has many limitations. First, the vector and structured databases are fragmented, failing to achieve unified object modeling and atomic management of multimodal data within a single database engine, making it difficult to meet ACID requirements such as transaction consistency, isolation, and durability. Second, existing solutions lack a unified query language and execution engine, typically relying on multiple query interfaces or extended syntaxes, making it impossible to achieve unified optimization and execution of cross-modal queries at the database kernel layer. Third, the indexing system is highly fragmented; vector indexes, inverted indexes, and B+ tree indexes are maintained by different engines, unable to share logs, caches, and storage structures, resulting in high index maintenance costs and low collaboration efficiency. Finally, the parallel deployment of multiple systems leads to severe redundancy in the storage architecture. Structured and unstructured data are stored separately, and there is a lack of unified hot and cold tiering, compression, and caching strategies, which further restricts the improvement of overall I / O efficiency and system performance. Summary of the Invention

[0003] This application provides an indexing method, apparatus, and electronic device for mixed storage of multimodal data to solve the technical problem of index fragmentation when managing multimodal data.

[0004] Firstly, this application provides an indexing method for hybrid storage of multimodal data, comprising: preprocessing the received multimodal raw dataset to obtain metadata and semantic embedding vectors for each raw data, and encapsulating each raw data and its metadata and semantic embedding vectors into a unified multimodal data object, wherein each multimodal data object is assigned a globally unique object identifier; for each multimodal data object, constructing a vector index and a structured index according to the modality type, associating the vector index and structured index with its object identifier through an index association mapping mechanism, and storing each multimodal data object in different locations according to access frequency and storage value. The system utilizes a hierarchical storage medium. Upon receiving a multimodal query statement from a user, which includes structured query conditions and semantic similarity query conditions, the system parses the multimodal query statement to determine the execution order of different types of indexes based on the selectivity of the structured query conditions, and generates a corresponding query execution plan. According to the query execution plan, vector index retrieval and structured index scanning are executed in parallel across all levels of storage media. The retrieval results are jointly filtered and fused based on object identifiers to obtain multimodal data results, which are then returned to the user. These multimodal data results contain all multimodal data objects that satisfy the query conditions of the multimodal query statement.

[0005] Secondly, this application provides an indexing device for hybrid storage of multimodal data, comprising: a preprocessing module, used to preprocess the received multimodal raw dataset to obtain metadata and semantic embedding vectors for each raw data, and to encapsulate each raw data and its metadata and semantic embedding vectors into a unified multimodal data object, wherein each multimodal data object is assigned a globally unique object identifier; and a construction module, used to construct vector indexes and structured indexes for each multimodal data object according to its modality type, so as to associate its vector indexes and structured indexes with its object identifier through an index association mapping mechanism, and to store each multimodal data object in different locations according to access frequency and storage value. The system comprises a hierarchical storage medium; a parsing module, which parses a multimodal query statement sent by a user, containing structured query conditions and semantic similarity query conditions, to determine the execution order of different types of indexes based on the selectivity of the structured query conditions and generate a corresponding query execution plan; and an indexing module, which performs parallel vector index retrieval and structured index scanning in all levels of storage medium according to the query execution plan, performs joint filtering and result fusion on the retrieval results based on object identifiers, obtains multimodal data results, and returns them to the user. These multimodal data results contain all multimodal data objects that satisfy the query conditions of the multimodal query statement.

[0006] As an optional example, the preprocessing module includes: a cleaning unit for performing format parsing and data cleaning on the multimodal raw data to eliminate noise data and unify the data representation format; and a first processing unit for determining the unprocessed raw data after data cleaning as the current raw data, and performing the following processing on the current raw data: performing modality type identification on the current raw data to determine the modality type of the current raw data, wherein the modality type is text modality, image modality, or audio modality; extracting metadata corresponding to the current raw data according to the modality type of the current raw data, wherein the metadata includes time information, source information, and content description information; and calling a semantic feature extraction model matching the modality type of the current raw data to perform semantic feature encoding on the current raw data and generate a semantic embedding vector of the current raw data.

[0007] As an optional example, the above-mentioned construction module includes: a third processing unit, configured to identify the unprocessed multimodal data object as the current multimodal data object, and perform the following processing on the current multimodal data object: obtain the metadata, semantic embedding vector, and modality type contained in the current multimodal data object; construct a vector index for semantic similarity retrieval for the current multimodal data object based on the semantic embedding vector of the current multimodal data object, and associate the entries of the vector index with the object identifier of the current multimodal data object; construct a structured index for precise conditional filtering for the current multimodal data object based on the metadata and modality type of the current multimodal data object, and associate the entries of the structured index with the object identifier of the current multimodal data object; and record the index entries of the vector index and the structured index in an index association mapping table.

[0008] As an optional example, the above-mentioned construction module includes: a second processing unit, configured to identify unprocessed multimodal data objects as current multimodal data objects, and perform the following processing on the current multimodal data objects: obtain the access frequency, modality type, and lifecycle characteristics of the current multimodal data objects, and evaluate the storage value of the current multimodal data objects based on the access frequency, modality type, and lifecycle characteristics; if the storage value of the current multimodal data objects is greater than a first threshold, store the current multimodal data objects in a high-performance storage medium; if the storage value of the current multimodal data objects is less than or equal to the first threshold, store the current multimodal data objects in a low-cost storage medium.

[0009] As an optional example, the above parsing module includes: a parsing unit, used to perform syntax parsing on the above multimodal query statement to identify the structured query conditions and semantic similarity query conditions; a calculation unit, used to calculate the selectivity of the above structured query conditions based on historical statistical information or index statistical information and using the above preset cost model; a first determining unit, used to determine the above execution order as performing a structured index scan first and then a vector index retrieval if the selectivity of the above structured query conditions is less than a second threshold; a second determining unit, used to determine the above execution order as performing a vector index retrieval first and then a structured index scan if the selectivity of the above structured query conditions is greater than or equal to the above second threshold; and a generation unit, used to generate a query execution plan including a vector index retrieval operator and a structured index scan operator according to the above execution order.

[0010] As an optional example, the above index module includes: a retrieval unit, used to distribute the above query execution plan and index association mapping table to the corresponding data nodes to trigger vector index retrieval and structured index scanning in parallel. In each data node, semantic similarity retrieval is performed based on the vector index, and structured condition filtering is performed based on the structured index to obtain the corresponding intermediate retrieval results; a filtering unit, used to aggregate all intermediate retrieval results according to object identifiers, and perform at least one of the following processing on the retrieval results from different indexes based on the object identifiers to complete the joint filtering; and a sorting unit, used to perform result fusion and sorting on the multimodal data objects that have completed the joint filtering to form a multimodal data result containing multimodal data objects that meet the query conditions of the above multimodal query statement.

[0011] As an optional example, the index module includes a return unit, which is used to return the multimodal data results to the user in a manner that combines structured information and multimedia information according to a preset result organization rule. The structured information includes the object identifier, time attribute and status information of the data object, and the multimedia information includes the image access link or audio playback address corresponding to the data object.

[0012] Thirdly, this application provides a storage medium storing a computer program, wherein the computer program is executed by a processor to perform the above-described indexing method for mixed storage of multimodal data.

[0013] Fourthly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described indexing method for multimodal data hybrid storage through the computer program.

[0014] The technical solutions provided in this application have the following advantages compared with the prior art: This application employs a preprocessing method to obtain metadata and semantic embedding vectors for each received multimodal raw dataset. Each raw dataset, along with its metadata and semantic embedding vectors, is then encapsulated into a unified multimodal data object, with each multimodal data object assigned a globally unique object identifier. Based on access frequency and storage value, each multimodal data object is stored in different levels of storage media. For each multimodal data object, vector and structured indexes are constructed according to modality type, and these indexes are associated with their object identifiers through an index association mapping mechanism. Upon receiving a multimodal query statement from a user containing structured query conditions and semantic similarity query conditions, the multimodal query statement is parsed, and the selection rate based on the structured query conditions is determined. The process involves determining the execution order of different index types and generating corresponding query execution plans. Based on these plans, vector index retrieval and structured index scanning are performed in parallel across all levels of storage media. The retrieval results are then jointly filtered and fused based on object identifiers to obtain multimodal data results, which are returned to the user. These multimodal data results include methods for handling all multimodal data objects that meet the query conditions of the multimodal query statement. This is achieved by preprocessing the original multimodal data, encapsulating the original data, metadata, and semantic embedding vectors into a unified multimodal data object, and assigning it a globally unique object identifier. Furthermore, the multimodal data objects are stored hierarchically based on access frequency and storage value, and vector and structured indexes are constructed, with index association achieved through object identifiers. During the query phase, multimodal query statements that simultaneously contain structured conditions and semantic similarity conditions are uniformly parsed and execution plans are generated. Index retrieval is then performed in parallel at each storage level, and the results are fused based on object identifiers. This achieves unified management and efficient joint retrieval of multimodal data within a single database system, improving query efficiency, storage utilization, and overall system performance. Consequently, it solves the technical problem of index fragmentation when managing multimodal data. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 This is a flowchart of an optional indexing method for multimodal data hybrid storage according to an embodiment of this application; Figure 2 This is a system architecture diagram of an optional multimodal data hybrid storage indexing method according to an embodiment of this application; Figure 3 This is a flowchart illustrating the implementation of an optional indexing method for multimodal data hybrid storage according to an embodiment of this application. Figure 4 This is a schematic diagram of the structure of an optional multimodal data hybrid storage indexing device according to an embodiment of this application; Figure 5 This is a schematic diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0021] According to a first aspect of the embodiments of this application, an indexing method for multimodal data hybrid storage is provided, optionally, as follows: Figure 1 As shown, the above method includes: S102, preprocess the received multimodal raw dataset to obtain the metadata and semantic embedding vector of each raw data, and encapsulate each raw data and its metadata and semantic embedding vector into a unified multimodal data object, wherein each multimodal data object is assigned a globally unique object identifier; S104. For each multimodal data object, construct vector index and structured index according to modality type, and associate the vector index and structured index with the object identifier through index association mapping mechanism. Then, store each multimodal data object in different levels of storage medium according to access frequency and storage value. S106, When receiving a multimodal query statement sent by the user that contains structured query conditions and semantic similarity query conditions, the multimodal query statement is parsed to determine the execution order of different types of indexes based on the selectivity of its structured query conditions and generate the corresponding query execution plan; S108. According to the query execution plan, vector index retrieval and structured index scanning are performed in parallel on all levels of storage media. The retrieval results are jointly filtered and fused based on object identifiers to obtain multimodal data results and return them to the user. The multimodal data results contain all multimodal data objects that meet the query conditions of the multimodal query statement.

[0022] Optionally, this embodiment proposes an indexing method for hybrid storage of multimodal data, aiming to solve the problems of scattered multimodal data storage, fragmented indexes, and weak query collaboration capabilities in existing database systems. This method achieves unified modeling, unified storage, and efficient joint retrieval of multimodal data within a single database system. It is suitable for application scenarios that simultaneously contain multiple modalities such as structured data, text, images, and audio, and can improve the overall efficiency of multimodal queries while ensuring transaction consistency and system performance.

[0023] In the specific implementation process, the received multimodal raw dataset is first preprocessed. This preprocessing includes format parsing, modality recognition, and feature extraction of the raw data, thereby generating metadata describing the basic attributes of the raw data, as well as semantic embedding vectors used to characterize the semantic features of the data. Subsequently, each raw data, along with its corresponding metadata and semantic embedding vector, is encapsulated to form a unified multimodal data object, and a globally unique object identifier is assigned to each multimodal data object. Through object-oriented modeling, a unified representation of different modalities of data at the logical level is achieved, providing a foundation for subsequent index construction and joint queries.

[0024] After constructing the multimodal data objects, tiered storage management is implemented based on their access frequency and storage value. Specifically, multimodal data objects are stored in different tiers of storage media according to their access frequency and business value, such as high-performance storage media and low-cost, high-capacity storage media. Simultaneously, for each multimodal data object, vector indexes and structured indexes are constructed based on its modality type and data attributes. Semantic embedding vectors are used to construct the vector indexes, while structured fields from the metadata are used to construct the structured indexes. By establishing an index association mapping mechanism, the vector indexes and structured indexes are uniformly associated with the corresponding object identifiers, thereby achieving unified management of multimodal indexes at the logical level.

[0025] When a multimodal query is received from a user, containing both structured query conditions and semantic similarity query conditions, the system first parses the query to identify different types of query conditions. Based on the selectivity of the structured query conditions, the filtering capabilities of various indexes are evaluated to determine the execution order of vector and structured indexes, generating a corresponding query execution plan. This execution plan fully leverages the advantages of different indexes during the query phase, achieving overall optimization of the query process.

[0026] During the query execution phase, according to the query execution plan, vector index retrieval and structured index scanning operations are performed in parallel across all levels of storage media to obtain candidate result sets. Subsequently, the results returned by different indexes are jointly filtered and fused based on object identifiers to ensure that the final returned multimodal data objects simultaneously satisfy both structured and semantic similarity conditions. Finally, the obtained multimodal data results are returned to the user, containing all multimodal data objects that meet the query conditions of the multimodal query statement.

[0027] Optionally, this embodiment achieves unified management of multimodal data within a single database system through a unified multimodal data object modeling and object identification mechanism, avoiding transaction consistency and operational complexity issues caused by the fragmentation of multiple systems; through the association mapping and joint execution of vector indexes and structured indexes, it achieves true multimodal joint retrieval and query optimization, improving execution efficiency in complex query scenarios; in addition, by combining hierarchical storage and parallel retrieval mechanisms, it effectively reduces system I / O overhead and improves storage resource utilization and overall system performance.

[0028] As an optional example, the received multimodal raw dataset is preprocessed to obtain metadata and semantic embedding vectors for each raw data point, including: The original multimodal data is parsed and cleaned to eliminate noisy data and unify the data representation format. The unprocessed, cleaned raw data is identified as the current raw data, and the following processing is performed on the current raw data: Modality type identification is performed on the current raw data to determine the modality type of the current raw data, where the modality type is text modality, image modality, or audio modality; Based on the modality type of the current raw data, extract the metadata corresponding to the current raw data. The metadata includes time information, source information, and content description information. Call the semantic feature extraction model that matches the modality type of the current original data to encode the semantic features of the current original data and generate the semantic embedding vector of the current original data.

[0029] Optionally, in this embodiment, the received multimodal raw dataset is preprocessed to obtain metadata and semantic embedding vectors corresponding to each raw data, thereby providing basic support for subsequent unified modeling and index construction. The preprocessing process first performs format parsing and data cleaning operations on the multimodal raw data. By uniformly parsing data from different sources and in different formats, duplicate data, missing fields, and abnormal noise data are eliminated, and the multimodal raw data is converted into a unified data representation format to ensure the consistency and stability of subsequent processing.

[0030] After data cleaning, a processing flow is executed for each piece of raw data. First, modality identification is performed on the current raw data. Based on its data characteristics and content format, its modality is determined. Modality types include, but are not limited to, text modality, image modality, or audio modality. Through modality identification, the system can select corresponding processing strategies and feature extraction methods for different modalities.

[0031] After determining the modality type, the corresponding metadata is extracted based on the modality type of the current raw data. Metadata describes the basic attributes and business characteristics of the current raw data, including but not limited to time information, source information, and content description information. Specifically, time information characterizes the time when the data was generated or collected, source information identifies the data's origin or business system, and content description information provides a structured summary of the data content, supporting subsequent structured queries and filtering.

[0032] Subsequently, a semantic feature extraction model matching the current original data modality is invoked to encode semantic features into the original data, generating corresponding semantic embedding vectors. Different semantic feature extraction models are used to process data of different modalities; for example, a text semantic encoding model is used for text modality data, an image feature extraction model is used for image modality data, and an audio semantic encoding model is used for audio modality data. By mapping data of different modalities to a unified vector space, a unified expression of semantic information of multimodal data is achieved.

[0033] As an optional example, for each multimodal data object, a vector index and a structured index are constructed based on the modality type. These vector and structured indexes are then associated with the object's identifier via an index association mapping mechanism, including: The unprocessed multimodal data object is identified as the current multimodal data object, and the following processing is performed on the current multimodal data object: Retrieve the metadata, semantic embedding vector, and modality type contained in the current multimodal data object; Based on the semantic embedding vector of the current multimodal data object, construct a vector index for semantic similarity retrieval for the current multimodal data object, and associate the entries of the vector index with the object identifier of the current multimodal data object; Based on the metadata and modality type of the current multimodal data object, construct a structured index for precise conditional filtering for the current multimodal data object, and associate the entries of the structured index with the object identifier of the current multimodal data object; Record the index entries of vector indexes and structured indexes in the index association mapping table.

[0034] Optionally, in this embodiment, for each multimodal data object, a vector index and a structured index are constructed according to the data modality types it contains. The vector index and structured index are then uniformly associated with the corresponding object identifier through an index association mapping mechanism to achieve efficient joint retrieval of multimodal data. Specifically, the index construction process is executed sequentially for each multimodal data object.

[0035] First, the metadata, semantic embedding vector, and modality type information contained in the current multimodal data object are obtained. The semantic embedding vector characterizes the semantic features of the current multimodal data object in a high-dimensional semantic space, the metadata describes the structured attribute information of the current multimodal data object, and the modality type indicates the data form category corresponding to the current multimodal data object.

[0036] After obtaining the above information, a vector index for semantic similarity retrieval is constructed based on the semantic embedding vector of the current multimodal data object. The vector index uses an approximate nearest neighbor search technique to build its index structure, supporting fast retrieval based on semantic similarity. During the construction of the vector index, each vector index entry is bound to the object identifier of the current multimodal data object, ensuring accurate location of the corresponding multimodal data object after vector retrieval.

[0037] Simultaneously, based on the metadata and modality type of the current multimodal data object, a structured index for precise conditional filtering is constructed. The structured index can select different index structures based on the type of the metadata fields to support query scenarios such as keyword matching, numerical range filtering, or state condition selection. Similarly, when constructing the structured index, the corresponding index entries are associated with the object identifier of the current multimodal data object.

[0038] After constructing the vector and structured indexes, the index entries are uniformly recorded in the index association mapping table. This table maintains the correspondence between different types of indexes and object identifiers, enabling cross-index result alignment, joint filtering, and rapid location during the query phase. Through this index construction and association mechanism, this embodiment achieves unified index management of multimodal data objects within the same database engine, providing a reliable index foundation for subsequent multimodal joint queries and efficient execution.

[0039] As an alternative example, storing each multimodal data object in different tiers of storage media based on access frequency and storage value includes: The unprocessed multimodal data object is identified as the current multimodal data object, and the following processing is performed on the current multimodal data object: Obtain the access frequency, modality type, and lifecycle characteristics of the current multimodal data object, and evaluate the storage value of the current multimodal data object based on the access frequency, modality type, and lifecycle characteristics; If the storage value of the current multimodal data object is greater than the first threshold, the current multimodal data object will be stored in a high-performance storage medium. If the storage value of the current multimodal data object is less than or equal to a first threshold, the current multimodal data object will be stored in a low-cost storage medium.

[0040] Optionally, in this embodiment, each multimodal data object is stored in a different tier of storage medium based on access frequency and storage value, thereby achieving rational utilization of storage resources and overall optimization of system performance. Specifically, a tiered storage decision process is executed for each multimodal data object.

[0041] First, the access frequency, modality type, and lifecycle characteristics of the current multimodal data object are obtained. Access frequency reflects the number of times the current multimodal data object is accessed or queried within a preset time window; modality type characterizes the data format contained in the current multimodal data object, such as text, image, or audio; lifecycle characteristics describe the activity cycle, retention period, or update pattern of the current multimodal data object in the business system. By comprehensively considering these multi-dimensional characteristics, the use value and storage cost of the current multimodal data object are evaluated, thereby calculating the corresponding storage value index.

[0042] After assessing the storage value, it is compared with a preset first threshold. When the storage value of the current multimodal data object exceeds the first threshold, the multimodal data object is determined to have high access demand or business value, and it is stored in a high-performance storage medium. The high-performance storage medium can be a low-latency, high-throughput storage device to meet the needs of frequent access and fast retrieval.

[0043] When the storage value of a multimodal data object is less than or equal to a first threshold, it is determined that the multimodal data object has a low access frequency or relatively low business importance, and it is stored in a low-cost storage medium. The low-cost storage medium can be a storage device with a large capacity but relatively low access performance, used for long-term storage or infrequently accessed data objects, thereby reducing the overall storage cost.

[0044] Through the above-described hierarchical storage method, this embodiment can dynamically match suitable storage media according to the actual usage characteristics of multimodal data objects. While ensuring the access performance of key data, it reduces the ineffective occupation of high-performance storage resources, improves the overall utilization and operating efficiency of the storage system, and provides a stable and efficient storage foundation for subsequent multimodal joint retrieval.

[0045] As an optional example, the multimodal query statement is parsed to determine the execution order of different types of indexes based on the selectivity of its structured query conditions, generating the corresponding query execution plan, including: Syntax parsing is performed on multimodal query statements to identify structured query conditions and semantic similarity query conditions; Based on historical or index statistics, the selectivity of structured query conditions is calculated using a pre-defined cost model. If the selectivity of the structured query condition is less than the second threshold, the execution order is determined to be to first perform the structured index scan and then perform the vector index retrieval. If the selectivity of the structured query condition is greater than or equal to the second threshold, the execution order is determined to be to first perform vector index retrieval and then perform structured index scanning. Based on the execution order, a query execution plan is generated that includes vector index retrieval operators and structured index scan operators.

[0046] Optionally, in this embodiment, multimodal query statements are uniformly parsed and optimized to dynamically determine the execution order of different types of indexes based on the selectivity of structured query conditions, thereby generating an efficient query execution plan. Multimodal query statements contain both structured query conditions and semantic similarity query conditions. If index retrieval is executed indiscriminately, it will lead to an expansion of the invalid scan range, increasing system computation and I / O overhead. Therefore, this embodiment introduces a selectivity-aware query optimization mechanism to intelligently schedule the multimodal query execution process.

[0047] Specifically, the received multimodal query statement is first parsed. A custom query parser or extended SQL parsing component is used to identify and split various query conditions in the query statement, thereby distinguishing between exact matching or range query conditions based on structured fields and similarity query conditions based on semantic embedding vectors.

[0048] After identifying the query conditions, the selectivity of the structured query conditions is calculated using a pre-defined cost model based on historical query statistics or current index statistics. The selectivity reflects the filtering ability of the structured query conditions on the dataset, i.e., the proportion of data objects that satisfy the structured query conditions in the overall dataset. The cost model comprehensively considers index size, field distribution characteristics, and historical query hit rates to improve the accuracy of the selectivity evaluation.

[0049] When the selectivity of the structured query conditions is less than the second threshold, it indicates that the structured query conditions have a strong filtering ability and can significantly narrow down the range of candidate data. Therefore, the query execution order is determined to be to first perform a structured index scan, and then perform a vector index retrieval on the filtered candidate data set, thereby reducing the computational overhead of vector retrieval.

[0050] Conversely, if the selectivity of the structured query condition is greater than or equal to the second threshold, it indicates that the filtering ability of the structured query condition is weak. If the structured index scan is performed first, it will lead to a large number of invalid scans. Therefore, the query execution order is determined to first perform vector index retrieval to quickly obtain semantically similar candidate data objects, and then perform a structured index scan on the candidate result set for accurate filtering.

[0051] Finally, based on the determined index execution order, a query execution plan is generated that includes vector index retrieval operators and structured index scan operators. The query execution plan clearly defines the execution order of each operator and the data flow method, providing an efficient and controllable execution path for subsequent multimodal joint retrieval, thereby improving overall query performance and system resource utilization.

[0052] As an optional example, based on the query execution plan, vector index retrieval and structured index scanning are performed in parallel across all levels of storage media. The retrieval results are then jointly filtered and fused based on object identifiers to obtain multimodal data results, including: The query execution plan and index association mapping table are distributed to the corresponding data nodes to trigger vector index retrieval and structured index scanning in parallel. In each data node, semantic similarity retrieval is performed based on the vector index, and structured condition filtering is performed based on the structured index to obtain the corresponding intermediate retrieval results. All intermediate search results are aggregated according to object identifiers, and at least one of the following processes—duplicate removal, intersection, and union—is performed on the search results from different indexes based on the object identifiers to complete the joint filtering; The results of the multimodal data objects that have completed the joint screening are fused and sorted to form a multimodal data result containing multimodal data objects that meet the query conditions of the multimodal query statement.

[0053] Optionally, in this embodiment, according to the generated query execution plan, vector index retrieval and structured index scanning are executed in parallel across all levels of storage media. The retrieval results are then jointly filtered and fused based on object identifiers to obtain the final multimodal data results. By introducing parallel execution and an object identifier-driven result fusion mechanism, distributed storage and computing resources can be fully utilized, improving the overall execution efficiency of multimodal joint retrieval.

[0054] Specifically, the query execution plan and index association mapping table are first distributed to the corresponding data nodes. Each data node is responsible for managing data from different data shards or storage levels, triggering vector index retrieval and structured index scanning tasks in parallel through this distribution mechanism. Within each data node, based on the execution order and operator types defined in the query execution plan, semantic similarity retrieval is performed using the vector index to obtain candidate multimodal data objects that match the query's semantic conditions. Simultaneously, structured condition filtering is performed on the multimodal data objects based on the structured index to select data objects that meet structured constraints such as time, status, category, or source. This results in a corresponding intermediate retrieval result set in each data node.

[0055] After completing the index retrieval for each data node, all intermediate retrieval results are aggregated according to object identifiers. The object identifier is a globally unique identifier for multimodal data objects, used to uniformly align retrieval results across different indexes and nodes. Based on the object identifier, at least one of the following processing methods—duplicate removal, intersection, and union—is performed on the intermediate retrieval results from different indexes and nodes to complete cross-index, cross-modal joint filtering operations, thereby ensuring that the ultimately retained data objects simultaneously satisfy semantic similarity and structured query conditions.

[0056] After joint screening, the selected multimodal data objects undergo result fusion and sorting. The result fusion process comprehensively considers the similarity scores of the multimodal data objects in vector retrieval, the matching of structured conditions, and the weight information of the objects themselves to generate a unified comprehensive score. Subsequently, the multimodal data objects are sorted according to the comprehensive score, forming the final multimodal data result set. The multimodal data result set contains all multimodal data objects that meet the query conditions of the multimodal query statement and can be returned to the user in order of relevance, thus providing the user with accurate and efficient multimodal joint retrieval results.

[0057] As an optional example, the following is returned to the user: According to the preset result organization rules, the multimodal data results are returned to the user in a combination of structured information and multimedia information. The structured information includes the object identifier, time attribute and status information of the data object, and the multimedia information includes the image access link or audio playback address corresponding to the data object.

[0058] Optionally, in this embodiment, after completing the joint retrieval, result fusion, and sorting of multimodal data, the final multimodal data results are returned to the user according to preset result organization rules. Result organization rules are used to standardize the presentation structure and output format of multimodal data results to balance machine parsingability and user understandability, thereby meeting the data consumption needs of different application scenarios.

[0059] Specifically, the multimodal data results are organized and returned to the user in a way that combines structured information and multimedia information. The structured information describes the basic attributes and management information of the multimodal data objects, including but not limited to the object identifier, time attribute, and status information. The object identifier is a globally unique identifier for the multimodal data object, used for unique location and reference in subsequent business processing; the time attribute characterizes the generation time, acquisition time, or update time of the multimodal data object; and the status information indicates the current valid status, processing status, or availability status of the multimodal data object.

[0060] Meanwhile, multimedia information is used to provide users with access to the raw payload associated with the multimodal data object. This multimedia information includes image access links or audio playback addresses corresponding to the multimodal data object, enabling users to browse image content online or play audio content in real time. By returning multimedia information through links or address references, large-scale multimedia data can be avoided by directly transmitting it in the query response, thereby reducing network transmission overhead and improving response efficiency.

[0061] In practical implementation, multimodal data results can be encapsulated according to a preset data format and returned to the user through a unified query interface. The query interface can be a network protocol-based interface or an application programming interface (API) to support access from different types of clients. The returned data results can be used for direct display or as input for subsequent data analysis or intelligent processing, thereby improving the system's versatility and scalability.

[0062] To illustrate with an example, this application relates to an indexing technique for multimodal data hybrid storage, such as... Figure 2 The system architecture diagram shown adopts a modular and layered design. The core components are built around a "multimodal unified database engine," forming a closed loop from data access to result output. The functions of each component are as follows: 1. Multimodal data source: The input end provides the raw multimodal data to be processed, including text, images, audio, etc.

[0063] 2. Data Access and Preprocessing Module: Responsible for the "standardization processing" of raw data. Core functions include: Data format conversion: Convert heterogeneous data from different sources (such as JSON, CSV, raw image files, and audio files) into system-compatible formats; Data cleaning: removing noisy data (such as blurry images and broken audio) and duplicate data; Modality recognition: Automatically identifies the modality type (text / image / audio) of the data; Metadata extraction: Extract basic descriptive information of the data (such as collection time, device ID, and data size).

[0064] Final output: Standardized raw data packet + modal label.

[0065] 3. Vector Generation Service: Responsible for converting unstructured data into high-dimensional vectors that can be used for semantic retrieval. The core logic is to call pre-trained deep learning models (such as BERT for text and ResNet for images) to generate semantic embedding vectors for standardized data of each modality; at the same time, the vector results of high-frequency samples are cached to avoid repeated calculations and improve vector generation efficiency.

[0066] 4. Multimodal Unified Database Engine (Core Component): The core of the system, integrating five sub-modules to achieve unified modeling, index management, query optimization, and storage collaboration for multimodal data. Unified Object Model Management: Defines and maintains the data structure of "multimodal objects", and is responsible for encapsulating raw data, metadata, and embedded vectors into unified objects, generating globally unique IDs, and ensuring transactional writes and consistent reads; Fusion Index Management: Maintain a "two-layer fusion index architecture", including vector index, inverted index, B+ tree index, and bitmap index; manage index association mapping tables; and perform incremental index merging and version control. Intelligent Query Optimizer: Parses extended SQL / MQL query statements, analyzes the distribution of query conditions (structured / semantic conditions), calculates the optimal execution path through a cost model, generates an execution plan, and schedules resources such as CPU and memory; Hybrid storage engine: Executes a tiered storage strategy, responsible for the automatic migration of data in hot / warm / cold layers (based on time period or access frequency), realizing the separate storage of raw payload and metadata / vectors, improving update flexibility and I / O efficiency; Metadata service: Centrally manages the schema (data structure definition), index status, snapshots, and access permissions of all multimodal objects. It implements high-availability registration and discovery based on Zookeeper to ensure data consistency and synchronization across nodes.

[0067] 5. Query Interface and Result Integration Module: The system's output end, responsible for integrating query results and external interaction. Core functions include: Provides multiple types of interfaces: JDBC / ODBC interfaces (compatible with traditional database tools), REST API (returns JSON response for third-party system calls), SDK (for AI systems to call for subsequent analysis), and Web front-end (visualizing image comparisons, audio waveforms, etc.). Results fusion and export: The search results are integrated into a format of "structured information (ID, time, status) + multimedia links (image URL, audio playback address)" and can be exported as JSON, CSV or visual reports.

[0068] The implementation process of this application forms a closed loop, covering two core links: data writing and query retrieval, such as... Figure 3 The specific implementation flowchart is shown below, and the specific steps are as follows: 1. Multimodal data access: Raw multimodal data from different sources can be uniformly accessed into the system through API interfaces (REST), message queues (Kafka), batch file import (CSV / JSON), gateway push, etc.; asynchronous streaming access is supported to ensure high throughput and low latency writing.

[0069] 2. Data preprocessing and modality recognition: Clean and standardize the raw data, automatically identify modality types, extract metadata, and output "standardized raw data package + modality label".

[0070] 3. Semantic embedding vector generation: The pre-trained model is called to generate semantic vectors for each modality of data, and high-frequency sample vectors are cached to avoid repeated calculations.

[0071] 4. Multimodal object construction and unified modeling: encapsulate raw data, metadata, and semantic vectors into unified multimodal objects, generate globally unique IDs, and support transaction writing.

[0072] 5. Hybrid Index Creation: Create composite indexes of multiple types within the same database engine to support efficient cross-modal retrieval; the index types and construction methods are shown in Table 1 below: Table 1

[0073] Build corresponding indexes for different fields of the unified object—build B+ tree / inverted indexes for structured fields (time, device type), build bitmap indexes for status fields, and build upper-level vector indexes for semantic vectors; at the same time, update the index association mapping table and start the background incremental index merging task to ensure index consistency.

[0074] 6. Persistent Data Storage: Based on data access frequency, data is automatically allocated to hot / warm / cold storage layers to achieve tiered storage, as shown in Table 2 below: Table 2

[0075] Storage strategy: Automatic lifecycle management: Automatic migration by time; Separate storage of raw payload, metadata, and vectors to improve I / O efficiency; Supports "on-demand loading", decompressing and loading when querying cold data.

[0076] 7. Intelligent query parsing and optimization: When a user submits an extended SQL / MQL query statement, the system extracts the query conditions through a custom parser, distinguishes between structured conditions and semantic conditions, calculates the optimal execution path through a cost model, and generates an execution plan.

[0077] 8. Joint Retrieval and Result Ranking: The coordinating node distributes the query plan to multiple data nodes, executing multi-index scans in parallel—structured index filtering (B+ tree filtering time / device type, bitmap filtering status) and vector index ANN search (image similarity matching); each node returns intermediate results. The intermediate results from all nodes are aggregated, deduplicated and merged according to a unified object ID, and a comprehensive relevance score is calculated for each object (combining structured condition matching degree and semantic similarity score), then sorted in descending order of score.

[0078] 9. Multimodal result return: The sorted results are returned to the user in the form of "structured information + multimedia links" through the corresponding interface, and export and visualization are supported.

[0079] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0080] According to another aspect of the embodiments of this application, an indexing device for multimodal data hybrid storage is also provided, such as... Figure 4 As shown, it includes: The preprocessing module 402 is used to preprocess the received multimodal raw dataset to obtain the metadata and semantic embedding vector of each raw data, and to encapsulate each raw data and its metadata and semantic embedding vector into a unified multimodal data object, wherein each multimodal data object is assigned a globally unique object identifier. Module 404 is used to construct vector indexes and structured indexes for each multimodal data object according to its modality type, so as to associate its vector indexes and structured indexes with its object identifier through an index association mapping mechanism, and to store each multimodal data object in different levels of storage media according to access frequency and storage value. The parsing module 406 is used to parse the multimodal query statement sent by the user, which contains structured query conditions and semantic similarity query conditions, so as to determine the execution order of different types of indexes based on the selectivity of its structured query conditions and generate the corresponding query execution plan. The index module 408 is used to perform vector index retrieval and structured index scanning in parallel across all levels of storage media according to the query execution plan, perform joint filtering and result fusion on the retrieval results based on object identifiers, obtain multimodal data results and return them to the user. The multimodal data results contain all multimodal data objects that satisfy the query conditions of the multimodal query statement.

[0081] It should be noted that the preprocessing module 402 in this embodiment can be used to execute step S102 in this application embodiment, the construction module 404 in this embodiment can be used to execute step S104 in this application embodiment, the parsing module 406 in this embodiment can be used to execute step S106 in this application embodiment, and the indexing module 408 in this embodiment can be used to execute step S108 in this application embodiment.

[0082] As an optional example, the preprocessing module includes: The cleaning unit is used to perform format parsing and data cleaning on the multimodal raw data to eliminate noisy data and unify the data representation format; The first processing unit is used to identify the unprocessed, cleaned raw data as the current raw data, and to perform the following processing on the current raw data: Modality type identification is performed on the current raw data to determine the modality type of the current raw data, where the modality type is text modality, image modality, or audio modality; Based on the modality type of the current raw data, extract the metadata corresponding to the current raw data. The metadata includes time information, source information, and content description information. Call the semantic feature extraction model that matches the modality type of the current original data to encode the semantic features of the current original data and generate the semantic embedding vector of the current original data.

[0083] As an optional example, the building blocks include: The third processing unit is used to identify the unprocessed multimodal data object as the current multimodal data object, and to perform the following processing on the current multimodal data object: Retrieve the metadata, semantic embedding vector, and modality type contained in the current multimodal data object; Based on the semantic embedding vector of the current multimodal data object, construct a vector index for semantic similarity retrieval for the current multimodal data object, and associate the entries of the vector index with the object identifier of the current multimodal data object; Based on the metadata and modality type of the current multimodal data object, construct a structured index for precise conditional filtering for the current multimodal data object, and associate the entries of the structured index with the object identifier of the current multimodal data object; Record the index entries of vector indexes and structured indexes in the index association mapping table.

[0084] As an optional example, the building blocks include: The second processing unit is used to identify the unprocessed multimodal data object as the current multimodal data object, and to perform the following processing on the current multimodal data object: Obtain the access frequency, modality type, and lifecycle characteristics of the current multimodal data object, and evaluate the storage value of the current multimodal data object based on the access frequency, modality type, and lifecycle characteristics; If the storage value of the current multimodal data object is greater than the first threshold, the current multimodal data object will be stored in a high-performance storage medium. If the storage value of the current multimodal data object is less than or equal to a first threshold, the current multimodal data object will be stored in a low-cost storage medium.

[0085] As an optional example, the parsing module includes: The parsing unit is used to perform syntax parsing on multimodal query statements to identify structured query conditions and semantic similarity query conditions. The calculation unit is used to calculate the selectivity of structured query conditions based on historical or index statistics and using a preset cost model. The first determining unit is used to determine the execution order as follows: first perform the structured index scan, then perform the vector index retrieval, when the selectivity of the structured query condition is less than the second threshold. The second determining unit is used to determine the execution order as follows: first execute vector index retrieval and then execute structured index scanning, when the selectivity of the structured query condition is greater than or equal to the second threshold. The generation unit is used to generate query execution plans, including vector index retrieval operators and structured index scan operators, based on the execution order.

[0086] As an optional example, the index module includes: The retrieval unit is used to distribute the query execution plan and index association mapping table to the corresponding data nodes to trigger vector index retrieval and structured index scanning in parallel. In each data node, semantic similarity retrieval is performed based on the vector index, and structured condition filtering is performed based on the structured index to obtain the corresponding intermediate retrieval results. The filtering unit is used to aggregate all intermediate search results according to the object identifier, and perform at least one of the following processing on the search results from different indexes based on the object identifier: deduplication, intersection, and union, in order to complete the joint filtering. The sorting unit is used to fuse and sort the results of the multimodal data objects that have completed the joint filtering, forming a multimodal data result containing multimodal data objects that meet the query conditions of the multimodal query statement.

[0087] As an optional example, the index module includes: The return unit is used to return multimodal data results to the user in a combination of structured information and multimedia information according to preset result organization rules. The structured information includes the object identifier, time attribute and status information of the data object, and the multimedia information includes the image access link or audio playback address corresponding to the data object.

[0088] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0089] Figure 5 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 5 As shown, it includes a processor 502, a communication interface 504, a memory 506, and a communication bus 508. The processor 502, communication interface 504, and memory 506 communicate with each other via the communication bus 508. Memory 506 is used to store computer programs; When processor 502 executes a computer program stored in memory 506, it performs the following steps: The received multimodal raw dataset is preprocessed to obtain metadata and semantic embedding vectors for each raw data. Each raw data and its metadata and semantic embedding vectors are then encapsulated into a unified multimodal data object, where each multimodal data object is assigned a globally unique object identifier. For each multimodal data object, a vector index and a structured index are constructed according to the modality type. The vector index and structured index are associated with the object identifier through an index association mapping mechanism. Based on the access frequency and storage value, each multimodal data object is stored in a different level of storage medium. When a multimodal query statement containing structured query conditions and semantic similarity query conditions is received from a user, the multimodal query statement is parsed to determine the execution order of different types of indexes based on the selectivity of its structured query conditions, and to generate the corresponding query execution plan. According to the query execution plan, vector index retrieval and structured index scanning are performed in parallel across all levels of storage media. The retrieval results are jointly filtered and fused based on object identifiers to obtain multimodal data results, which are then returned to the user. These multimodal data results contain all multimodal data objects that satisfy the query conditions of the multimodal query statement.

[0090] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0091] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0092] As an example, the memory 506 described above may include, but is not limited to, the preprocessing module 402, the construction module 404, the parsing module 406, and the indexing module 408 of the indexing device for multimodal data hybrid storage described above. Furthermore, it may include, but is not limited to, other module units in the indexing device for multimodal data hybrid storage described above, which will not be elaborated upon in this example.

[0093] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0094] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0095] Those skilled in the art will understand that Figure 5The structure shown is for illustrative purposes only. The device implementing the above-described multimodal data hybrid storage indexing method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet Devices (MID), PAD, etc. Figure 5 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0096] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0097] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which, when executed by a processor, performs the steps in the above-described indexing method for multimodal data mixed storage.

[0098] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0099] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0101] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An indexing method for multimodal data hybrid storage, characterized in that, include: The received multimodal raw dataset is preprocessed to obtain metadata and semantic embedding vectors for each raw data. Each raw data and its metadata and semantic embedding vectors are then encapsulated into a unified multimodal data object, where each multimodal data object is assigned a globally unique object identifier. For each multimodal data object, a vector index and a structured index are constructed according to the modality type. The vector index and structured index are associated with the object identifier through an index association mapping mechanism. Based on the access frequency and storage value, each multimodal data object is stored in a different level of storage medium. When a multimodal query statement containing structured query conditions and semantic similarity query conditions is received from a user, the multimodal query statement is parsed to determine the execution order of different types of indexes based on the selectivity of its structured query conditions, and to generate the corresponding query execution plan. According to the query execution plan, vector index retrieval and structured index scanning are performed in parallel across all levels of storage media. The retrieval results are jointly filtered and fused based on object identifiers to obtain multimodal data results, which are then returned to the user. The multimodal data results contain all multimodal data objects that satisfy the query conditions of the multimodal query statement.

2. The method according to claim 1, characterized in that, The received multimodal raw dataset is preprocessed to obtain metadata and semantic embedding vectors for each raw data point, including: The multimodal raw data is parsed and cleaned to eliminate noise and unify the data representation format. The unprocessed, cleaned raw data is identified as the current raw data, and the following processing is performed on the current raw data: Modality type identification is performed on the current raw data to determine the modality type of the current raw data, wherein the modality type is text modality, image modality, or audio modality; Based on the modality type of the current raw data, extract the metadata corresponding to the current raw data, wherein the metadata includes time information, source information and content description information; A semantic feature extraction model that matches the modality type of the current original data is invoked to encode the semantic features of the current original data and generate a semantic embedding vector of the current original data.

3. The method according to claim 1, characterized in that, For each multimodal data object, based on the modality type, a vector index and a structured index are constructed to associate the vector index and structured index with the object identifier through an index association mapping mechanism, including: The unprocessed multimodal data object is identified as the current multimodal data object, and the following processing is performed on the current multimodal data object: Obtain the metadata, semantic embedding vector, and modality type contained in the current multimodal data object; Based on the semantic embedding vector of the current multimodal data object, a vector index for semantic similarity retrieval is constructed for the current multimodal data object, and the entries of the vector index are associated with the object identifier of the current multimodal data object; Based on the metadata and modality type of the current multimodal data object, a structured index for precise conditional filtering is constructed for the current multimodal data object, and the entries of the structured index are associated with the object identifier of the current multimodal data object; The index entries of the vector index and the structured index are recorded in the index association mapping table.

4. The method according to claim 1, characterized in that, Based on access frequency and storage value, each multimodal data object is stored in different levels of storage media, including: The unprocessed multimodal data object is identified as the current multimodal data object, and the following processing is performed on the current multimodal data object: The access frequency, modality type, and lifecycle characteristics of the current multimodal data object are obtained, and the storage value of the current multimodal data object is evaluated based on the access frequency, modality type, and lifecycle characteristics. If the storage value of the current multimodal data object is greater than a first threshold, the current multimodal data object will be stored in a high-performance storage medium. If the storage value of the current multimodal data object is less than or equal to the first threshold, the current multimodal data object is stored in a low-cost storage medium.

5. The method according to claim 1, characterized in that, The multimodal query statement is parsed to determine the execution order of different types of indexes based on the selectivity of its structured query conditions, and a corresponding query execution plan is generated, including: The multimodal query statement is parsed to identify structured query conditions and semantic similarity query conditions. Based on historical or index statistics, the selectivity of the structured query conditions is calculated using a preset cost model. If the selectivity of the structured query condition is less than the second threshold, the execution order is determined to be to first perform a structured index scan and then perform a vector index retrieval. If the selectivity of the structured query condition is greater than or equal to the second threshold, the execution order is determined to be to first perform vector index retrieval and then perform structured index scanning. Based on the execution order, a query execution plan is generated that includes vector index retrieval operators and structured index scan operators.

6. The method according to claim 4, characterized in that, According to the query execution plan, vector index retrieval and structured index scanning are performed in parallel across all levels of storage media. The retrieval results are then jointly filtered and fused based on object identifiers to obtain multimodal data results, including: The query execution plan and index association mapping table are distributed to the corresponding data nodes to trigger vector index retrieval and structured index scanning in parallel. In each data node, semantic similarity retrieval is performed based on the vector index, and structured condition filtering is performed based on the structured index to obtain the corresponding intermediate retrieval results. All intermediate search results are aggregated according to object identifiers, and at least one of the following processes—duplicate removal, intersection, and union—is performed on the search results from different indexes based on the object identifiers to complete the joint filtering; The results of the multimodal data objects that have completed the joint screening are fused and sorted to form a multimodal data result containing multimodal data objects that meet the query conditions of the multimodal query statement.

7. The method according to any one of claims 1 to 6, characterized in that, The return to the user includes: According to the preset result organization rules, the multimodal data results are returned to the user in a combination of structured information and multimedia information. The structured information includes the object identifier, time attribute and status information of the data object, and the multimedia information includes the image access link or audio playback address corresponding to the data object.

8. An indexing device for multimodal data hybrid storage, characterized in that, include: The preprocessing module is used to preprocess the received multimodal raw dataset to obtain the metadata and semantic embedding vector of each raw data, and to encapsulate each raw data and its metadata and semantic embedding vector into a unified multimodal data object, wherein each multimodal data object is assigned a globally unique object identifier. The construction module is used to build vector indexes and structured indexes for each multimodal data object according to its modality type. It associates the vector indexes and structured indexes with the object identifier through an index association mapping mechanism, and stores each multimodal data object in different levels of storage media according to access frequency and storage value. The parsing module is used to parse the multimodal query statement sent by the user, which contains structured query conditions and semantic similarity query conditions, so as to determine the execution order of different types of indexes based on the selectivity of the structured query conditions and generate the corresponding query execution plan. The indexing module is used to perform vector index retrieval and structured index scanning in parallel across all levels of storage media according to the query execution plan, perform joint filtering and result fusion on the retrieval results based on object identifiers, obtain multimodal data results, and return them to the user. The multimodal data results contain all multimodal data objects that satisfy the query conditions of the multimodal query statement.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images