Index generation method and query method based on multimodal database

Abstract

The present invention discloses an index generation method and a query method based on a multimodal database, the method comprising: acquiring multimodal data and query data for constructing a database; performing feature extraction on each modality data in the multimodal data to obtain a data vector under each modality; generating a query vector based on the query data; and generating a graph index for the multimodal data based on the data vector under each modality and the query vector.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to an index generation method based on a multimodal database and a query method based on a multimodal database. Background Art

[0002] In modern database systems, the efficiency of indexing and retrieval directly affects the speed of data access and the overall performance of the system. With the continuous expansion of data scale, especially in the context of the increasing popularity of multimodal data (such as text, images, and videos), traditional indexing methods face huge challenges. Existing index structures often cannot effectively handle cross-modal queries, resulting in extended query response time and waste of resources. Current indexing technologies mainly include tree-based structures and hash tables. These methods perform well when processing single-modal data, but when faced with multimodal data, especially when the data is unevenly distributed or there are Out-of-Distribution (OOD) queries, the performance degrades significantly. Although existing approximate nearest neighbor search methods can improve retrieval speed, they often ignore the characteristics of query distribution, resulting in inefficiency in index construction and query processes. Summary of the invention

[0003] In order to solve the existing technical problems, the embodiments of the present application provide an index generation method based on a multimodal database and a query method based on a multimodal database, which can process data of different modalities and effectively improve the accuracy and response speed of the query.

[0004] In a first aspect, a method for generating an index based on a multimodal database is provided, comprising: acquiring multimodal data and query data for constructing a database; performing feature extraction on each modality data in the multimodal data to obtain a data vector under each modality; generating a query vector based on the query data; and generating a graph index for the multimodal data based on the data vector under each modality and the query vector, the graph index comprising data nodes corresponding to the data vectors, query nodes corresponding to the query vectors, and a connection relationship between the data nodes and the query nodes.

[0005] In a second aspect, a query method based on a multimodal database is provided, comprising: obtaining a query request containing at least one modal data and configured search parameters; converting the query request into the same feature space as the data vector in the graph index described in any one of the first aspect of the present application to obtain a feature vector to be queried; based on the search parameters and the feature vector to be queried, searching the graph index for a target data vector that matches the feature vector to be queried, and displaying the target data vector.

[0006] The present application converts multimodal data into data vectors and query data into query vectors, so that the data vectors and query vectors are located in the same high-dimensional feature space, and can capture and express complex information from different modalities in the same high-dimensional feature space; constructs a connection relationship between data nodes and query nodes in a graph index, and establishes a similarity association between query nodes and data nodes in a graph index, which makes it easier to find target nodes during the search process, improves search efficiency, and because the data nodes and query nodes in the graph index are located in the same high-dimensional feature space, it can process data of different modalities, which can effectively improve the accuracy and response speed of queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is an application environment diagram of an index generation method based on a multimodal database in an embodiment;

[0008] Figure 2 is a flowchart of an index generation method based on a multimodal database in one embodiment;

[0009] Figure 3 is a flow chart of a query method based on a multimodal database in one embodiment;

[0010] Figure 4 is a schematic diagram of an index generation device based on a multimodal database in an embodiment;

[0011] Figure 5 is a schematic diagram of a query device based on a multimodal database in an embodiment;

[0012] Figure 6 Schematic diagram of a terminal device in an embodiment. DETAILED DESCRIPTION

[0013] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0015] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it should be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0016] See also Figure 1 In one embodiment, the application environment diagram of the index generation method based on a multimodal database and the query method based on a multimodal database includes multiple terminal devices 100 and computing devices 200, wherein the terminal device 100 can be connected to the computing device 200 for communication, and the index generation method based on a multimodal database provided in the embodiment of the present application can be applied to the computing device 200, and the computing device 200 obtains multimodal data, processes these multimodal data, and obtains a graph index suitable for multimodal data retrieval. The multimodal data includes but is not limited to text data, image data, video data, sound data, sensor data, location data, and the like. After the graph index is established, the terminal device 100 can receive a query request based on the user interface input. After the computing device 200 obtains the query request, it searches for data matching the query request based on the established graph index and displays it.

[0017] See also Figure 2 , is a flowchart of an index generation method based on a multimodal database provided in an embodiment of the present application. The index generation method based on a multimodal database is applied to a computing device, and the index generation method based on a multimodal database includes the following steps:

[0018] S11. Acquire multimodal data and query data for building a database.

[0019] In this embodiment, multimodal data includes but is not limited to at least one of the following: text data, image data, video data, sound data, sensor data, location data, etc. These different types of data can be stored in combination in the same database to support richer data analysis and applications, wherein text data includes but is not limited to various forms of text, such as articles, comments, emails, etc.; image data includes but is not limited to photos, charts, maps, etc.; video data includes but is not limited to video files, streaming media, etc.; sound data includes but is not limited to audio files, voice messages, etc.; sensor data includes but is not limited to: data collected by various sensors such as temperature, humidity, and light; location data includes but is not limited to GPS data, geographic location information, etc. For data in any mode, it can include one or more files, for example, image data can include many images.

[0020] The query data includes but is not limited to the query files input by the user and the historical query data. The query files include multiple query requirements, and the historical query data also includes multiple historical query requirements. For example, queries with high query frequency and high query complexity can be selected as the query data based on the historical query data. In this way, common query requirements are established as nodes in the graph index, which can increase the efficiency of subsequent queries.

[0021] S12. Perform feature extraction on each modal data in the multimodal data to obtain a data vector under each modality.

[0022] In this embodiment, feature extraction is performed for each data file in each mode to obtain a feature vector in each mode, that is, a data vector. Each data file corresponds to a data vector, so that many data vectors can be obtained. The data vector indicates the characteristics of each data file. For example, for an audio file, the data vector includes at least one of the following: audio content, the time when the audio was created, the publisher of the audio, etc. For example, a text file includes at least one of the following: text content, the time when the text was created, the publisher of the text, etc. The feature extraction method includes but is not limited to an image feature extraction method, a text recognition method, an audio recognition method, etc.

[0023] S13. Generate a query vector based on the query data.

[0024] In this embodiment, the query data includes multiple query requirements, each query requirement is split into multiple query statements, and feature extraction is performed on each query statement to obtain a query vector corresponding to each query statement. Therefore, the query vector indicates the characteristics of the query requirement. It can be understood that the query data can also be multimodal data. For example, a query request can include multiple types of data, such as pictures and texts at the same time.

[0025] S14. Generate a graph index for multimodal data based on the data vector and query vector under each modality.

[0026] In this embodiment, the graph index includes data nodes corresponding to data vectors, query nodes corresponding to query vectors, and connection relationships between data nodes and query nodes. One data vector corresponds to one data node, and one query vector corresponds to one query node. The connection relationship between data nodes and query nodes represents the degree of association between the data nodes and the query nodes.

[0027] The data vector and the query vector belong to the same feature space, which is a high-dimensional feature space. The multimodal high-dimensional feature space refers to constructing a feature space that can integrate these different modal information when processing data containing multiple modalities (such as text, audio, and visual information). The dimension of this feature space is usually very high because it needs to capture and express complex information from different modalities. Data of different modalities are mapped to the same feature space through feature mapping technology, so that data of different modalities can be compared and connected under the same standard. When constructing the edges of the graph index, select those nodes that are close in the feature space, that is, nodes whose feature similarity meets the similarity conditions as connection objects to ensure that these connections reflect the true similarity relationship.

[0028] In the above embodiment, multimodal data is converted into data vectors, and query data is converted into query vectors, so that the data vectors and query vectors are located in the same high-dimensional feature space. In the same high-dimensional feature space, complex information from different modalities can be captured and expressed; a connection relationship between data nodes and query nodes is constructed in the graph index, and a similarity correlation between query nodes and data nodes is established in the graph index, which makes it easier to find the target node during the search process, improves the search efficiency, and because the data nodes and query nodes in the graph index are located in the same high-dimensional feature space, it is able to process data of different modalities, which can effectively improve the accuracy and response speed of the query.

[0029] In some embodiments, generating a graph index for multimodal data based on the data vector in each modality and the query vector includes:

[0030] Initializing the graph structure, wherein each of the query vectors is initialized as a query node in the graph structure, and each of the data vectors is initialized as a data node in the graph structure;

[0031] Based on the query node and the data node, an adjacency relationship between the query vector and the data vector is generated to obtain a connection relationship between each node in the initial graph index.

[0032] In this embodiment, the adjacency relationship between the data node and the query node includes a first connection edge and a second connection edge, wherein the first connection edge is directed from a query node to multiple adjacent data nodes, wherein the distance between the adjacent data nodes and the query node is relatively close, and the direction of the second connection edge is directed from the data node to the adjacent query node, wherein the second connection edge is directed from a data node to the adjacent query node, wherein the distance between the adjacent query node and the data node is the shortest, and the direction of the second connection edge is directed from the data node to the adjacent query node. The connection relationship between the data node and the query node represents the similarity correlation between the data node and the query node. The two nodes establish an edge in the graph index, indicating that the similarity of the features of the two nodes is relatively high. The graph index allows bidirectional connections between nodes, and the path from one node to another node can be carried out in two directions. This structure can improve the overall connectivity of the graph, making it easier to find the target node during the search process, and will improve the search efficiency. The bidirectional graph can better handle isolated nodes by providing reverse connections, and the bidirectional graph can improve the robustness of the system. If there is a problem with the connection in one direction, the connection in the other direction can still provide a valid path, thereby enhancing the stability of the system.

[0033] Optionally, the generating, based on the query node and the data node, a connection relationship between the nodes representing the adjacency relationship between the query vector and the data vector to obtain the connection relationship between the nodes in the graph index includes:

[0034] For the query node, calculate a first distance value between the query node and each data node to obtain multiple first distance values, sort the first distance values ​​from small to large, select a first preset number of first target distances from the multiple first distance values, use the data nodes corresponding to the first target distances as neighbor nodes of the query node, and generate a connection relationship between the neighbor nodes pointing from the query node to the query node;

[0035] For the data node, the second distance value between the data node and each query node is calculated to obtain multiple second distance values, and the query node corresponding to the minimum distance is selected from the multiple second distance values ​​as the neighbor node of the data node, and a connection relationship is generated from the data node to the data nodes.

[0036] In this embodiment, for a query node, the neighbor node of the query node is the adjacent data node, and the adjacent data node can be one or more. For a data node, the neighbor node of the data node is the adjacent query node, and the adjacent query node can be one or more. For each query node, the distance between the query node and each data node is calculated, such as the Euclidean distance or the cosine similarity, to obtain multiple first distance values, the first distance values ​​are sorted from small to large, and the first target distance of the first preset number N is selected from the multiple first distance values, and the N data nodes with the smallest distance are selected as the neighbor nodes of the query node, and then the edges are connected to its N nearest adjacent data nodes b1, b2, ..., bN, and the directionality of these edges is recorded, that is, from the query node to the adjacent node. Then, the out-degree of the query node is set to ensure that the out-degree of each query node is set to a larger value, so as to expand the coverage of the adjacent data points and fully model the adjacent relationship between the query node and the data node.

[0037] For each data node b, determine the query node q that is closest to the data node. In an optional implementation, allow the data node to establish an edge with its nearest query node, ensure that each data node can only be connected to one query node, and record the directionality of these edges, that is, from the data node to the query node. This bidirectional connection helps to better understand the relationship between the query node and the data node during the retrieval process.

[0038] In the above embodiment, a bilateral adjacency relationship between query nodes and data nodes is established, and bidirectional connections between nodes are allowed, which can improve the overall connectivity of the graph, making it easier to find the target node during the search process, and improve the search efficiency. The graph index structure using the bilateral adjacency relationship between query nodes and data nodes can better adapt to the characteristics of cross-modal data and can more effectively process queries between different modalities.

[0039] In some embodiments, generating a graph index for multimodal data based on the data vector in each modality and the query vector further includes:

[0040] Optimizing the initial graph index to obtain the graph index;

[0041] The optimizing the initial graph index to obtain the graph index comprises:

[0042] Searching for nodes to be optimized from the initial graph index, wherein the nodes to be optimized include at least one of the following: isolated nodes, nodes in a separated subgraph;

[0043] When the node to be optimized is a query node, a third distance value between the node to be optimized and each data node is calculated to obtain a plurality of third distance values, a second preset number of third target distances are screened from the plurality of third distance values, data nodes corresponding to the third target distances are used as supplementary neighbor nodes of the node to be optimized, and a connection relationship pointing from the node to be optimized to the supplementary neighbor nodes of the node to be optimized is generated;

[0044] When the node to be optimized is a data node, a fourth distance value between the node to be optimized and each query node is calculated to obtain multiple fourth distance values, a third preset number of fourth target distances are screened from the multiple fourth distance values, the query nodes corresponding to the fourth target distances are used as supplementary neighbor nodes of the node to be optimized, and a connection relationship from the node to be optimized to the supplementary neighbor nodes of the node to be optimized is generated.

[0045] In this embodiment, after obtaining the initial graph index, some data nodes may not have a direct connection with other query nodes, resulting in these nodes being unable to be effectively searched or accessed through the graph index structure. Such points are isolated nodes, and therefore, isolated nodes are nodes that are not connected to other nodes. A separated subgraph is an independent subgraph existing in the graph index. The nodes in the subgraph are connected within the subgraph and are not connected to other nodes in the graph index. For example, there are 40 nodes in the graph index, of which 3 nodes, A, B, and C, are interconnected and have no edge connections with the other 37 nodes. This may cause these isolated points or separated subgraphs to be unable to be effectively searched or accessed through the graph index structure during the search process.

[0046] In an optional implementation, the adjacency relationship of each node in the initial graph index is stored in an adjacency relationship file, and the adjacency relationship file can be at least one of: an adjacency list and an adjacency matrix. By traversing the adjacency relationship of each node recorded in the adjacency relationship file, isolated nodes and separated subgraphs can be found. The supplementary neighbor nodes of the node to be optimized are the approximate nodes of the node to be optimized. Approximate nodes refer to other nodes that are closer to the target node in the graph index. These nodes can serve as supplementary neighbors. An additional degree budget is set for each node to be optimized, allowing it to connect to a certain number of approximate nodes. For the node to be optimized as a query node, starting from the query node, the neighbor nodes of each node can be accessed along the edge of the initial graph index, and nodes similar to the query node can be gradually found. After finding a set of candidate nodes, the system will select the closest node as the approximate node based on the distance or similarity measure. When the node to be optimized is a data node, the processing is similar.

[0047] In the above embodiment, after the initial graph index is established, the initial graph index is also optimized to find out the isolated nodes and separated subgraphs in the initial graph index, and to find supplementary neighbor points for the isolated nodes and separated subgraphs, thereby improving the integrity of the connections in the graph index and facilitating the subsequent improvement of the efficiency of multimodal data retrieval.

[0048] In some embodiments, the method further comprises: in the process of optimizing the initial graph index, dynamically obtaining an index value for measuring the scale of the graph index, the index value comprising one of the following: the number of nodes, the degree of the graph;

[0049] When the indicator value is greater than or equal to a preset indicator value, the second preset number and the third preset number are reduced.

[0050] In this embodiment, the degree of the graph represents the number of all first connecting edges and all second connecting edges in the graph index. Therefore, the number of supplementary neighbor nodes of the node to be optimized and the number of neighbor nodes of each node are dynamically adjusted according to the degree of the graph, so as to dynamically adjust the scale of the graph index to avoid the scale of the graph index being too large, which affects the retrieval efficiency. For example, when the number of current nodes is greater than the preset number of nodes, and / or the degree of the current graph is greater than the preset degree, the number of supplementary neighbor nodes of the node to be optimized and the number of neighbor nodes of each node are reduced.

[0051] In the above embodiment, an index value for measuring the scale of the graph index is dynamically obtained, and the scale of the graph index is dynamically adjusted according to the current index value to avoid the scale of the graph index being too large, thereby affecting the retrieval efficiency.

[0052] In some embodiments, the method further comprises:

[0053] For any of the query nodes, taking the query node as the starting point and the target data node as the end point, traverse the connection relationship between the nodes in the graph index, and search for all paths from the query node to each target data node;

[0054] From all the paths from the query node to each target data node, the shortest path and alternative paths from the query node to each target data node are selected and recorded; based on the alternative paths, a connection relationship between the query node and each target data node is generated and recorded, wherein the path length indicated by the alternative path is greater than the length of the shortest path.

[0055] In this embodiment, the target data node can be any data node among the data nodes, or it can be a data node determined based on historical queries, such as a data node with a higher query frequency, etc. The shortest path can be one or more. The length of the path can be measured by the total number of nodes visited by the path. For the query node, an alternative path is added, and based on the alternative path, the connection relationship between the query node and each target data node is generated and recorded, that is, each node in the alternative path is connected by an edge. In the process of enhancing connectivity, it is ensured that multiple alternative paths are provided for the query node. This can be achieved by introducing additional edges in the graph to increase redundant connections between nodes. During the search process, these alternative paths are used to improve the flexibility and efficiency of the search. Ensure that the search algorithm can select the optimal path among multiple paths.

[0056] In the above embodiment, alternative paths are added between the query node and the target data node, and the nodes in the alternative paths are connected by edges, and these alternative paths are used to improve the flexibility and efficiency of the search, ensuring that the search algorithm can select the optimal path from multiple paths.

[0057] In some embodiments, the method further comprises:

[0058] Get the node to be changed;

[0059] According to the change operation indicated by the node to be changed, the node to be changed is added or deleted from the graph index to obtain a changed node;

[0060] Regenerate the graph index for multimodal data based on the changed nodes.

[0061] In this embodiment, when constructing the graph index structure, it is necessary to select a suitable data structure to store the bipartite graph, such as using a dictionary (hash table) to store nodes and their connected neighbor nodes. Ensure that the storage structure of the graph index can support fast query and update operations. The nodes to be changed include at least one of the following: newly added nodes, nodes to be deleted. For example, when a newly added data node is added, the connection between the newly added data node and the query node is recalculated, and the graph index is updated. After the graph index is updated, the updated graph index is optimized again, and the connectivity of the graph index is analyzed to ensure that there are no isolated query nodes or data nodes.

[0062] In the above embodiment, the nodes in the graph index can be dynamically adjusted, and the connection relationships in the graph index can be dynamically updated, so that the data in the multimodal database can be dynamically updated.

[0063] In some embodiments, Figure 3 As shown, a flowchart of a query method based on a multimodal database is shown, and the flowchart includes:

[0064] S31. Obtain a query request including at least one modality data and configured search parameters.

[0065] In this embodiment, when receiving query requests input by users, these query requests may include at least one of the following data: text, image or other modal data. The search parameters include a preset number of search layers and a preset number of candidate nodes. The preset number of search layers is the number of layers to search from the query starting point, and the preset number of candidate nodes is the number of candidate nodes retained in each layer of search.

[0066] S32. Convert the query request into the same feature space as the data vector in the graph index to obtain the feature vector to be queried.

[0067] In this embodiment, the input query is first preprocessed, including feature extraction and vectorization, and the query request is converted into a high-dimensional feature space of multimodal data so that the feature vector to be queried can be compared and calculated with the data vector in the same space.

[0068] S33. Based on the search parameters and the feature vector to be queried, search the graph index for a target data vector that matches the feature vector to be queried, and display the target data vector.

[0069] Optionally, searching the graph index for a target data vector that matches the feature vector to be queried based on the search parameter and the feature vector to be queried includes:

[0070] In the graph index, searching for a query node that is closest to the feature vector to be queried as a traversal query start node, and determining the query start node as a current search node;

[0071] In the graph index, in the next layer of the current search node, according to the adjacency relationship of the current search node, the neighbor nodes of the current search node are searched, and the current most similar neighbor nodes are determined among the neighbor nodes of the current search node, and according to the preset number of candidate nodes, the candidate nodes corresponding to the current most similar neighbor nodes are recorded, the current search node is updated with the most similar neighbor nodes, and the search in the graph index is continued;

[0072] During the search process, a current length index of the current search path is obtained. When the current length index indicates that the current search path is invalid, the current search node is returned to the next layer, and a target candidate node is selected from the candidate nodes corresponding to the current most similar neighbor node as the current search node, and the search is continued in the graph index until the search termination condition is met, and the target search path is obtained;

[0073] Based on the target search path, the target data vector is obtained.

[0074] In this embodiment, during the search process, the validity of the current path is evaluated in real time. When the current length index value is greater than the preset length, or the neighbor nodes of the current node are less than the preset number threshold, the search direction can be adjusted, and the target candidate node can be reselected in the next layer of the current search node as the current search node. Selecting a target candidate node from the candidate nodes corresponding to the current most similar neighbor node as the current search node includes at least one of the following: taking the candidate node with the most neighbor nodes among the candidate nodes corresponding to the current most similar neighbor node as the target candidate node; taking the candidate node closest to the target node as the target candidate node; taking the node with the highest access frequency among the candidate nodes corresponding to the current most similar neighbor node as the target candidate node. The target node is the data node indicated by the query request. The candidate node with the most neighbor nodes among the candidate nodes corresponding to the current most similar neighbor node is taken as the target candidate node, that is, connected to multiple other nodes. These nodes can provide more alternative paths, thereby improving the flexibility and efficiency of the search. Selecting a node that is closer to the target node as a well-connected node can reduce the search time. The distance can be evaluated by calculating the similarity between nodes or other metrics. Nodes with higher access frequency may also be considered good nodes because they show better performance in historical queries.

[0075] During the search, all nodes visited by the search path are recorded to avoid repeated visits and improve search efficiency.

[0076] Optionally, obtaining the target data vector based on the target search path includes:

[0077] According to the target search path, obtaining a target data vector node in the target search path;

[0078] The similarity between the target data vector node and the feature vector to be queried is calculated, multiple similarities are sorted from large to small, and the target data vector nodes that are ranked in the front fourth preset number are selected as the target data vector.

[0079] In the above embodiment, the search starts from the query start node and selects the closest neighbor node for expansion. In each step, the node most similar to the query vector is selected from the neighbor nodes of the current search node. In each layer, multiple candidate nodes are retained and these candidate nodes are continued to be expanded in the next layer. In this way, diversity can be maintained during the search process, avoiding falling into a local optimum, thereby improving the search efficiency.

[0080] On the other hand, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the index generation method based on a multimodal database described in any embodiment of the present application.

[0081] Among them, in the computer program product, an optional implementation form of the program module architecture of the computer program that implements each step of the index generation method based on a multimodal database may be an index generation device based on a multimodal database.

[0082] See also Figure 4 An embodiment of the present application provides an index generation device based on a multimodal database, including: an acquisition module 41, used to acquire multimodal data and query data for constructing a database; a conversion module 42, used to extract features of each modality data in the multimodal data to obtain a data vector under each modality; the conversion module 42 is also used to generate a query vector based on the query data; a generation module 43 is used to generate a graph index for the multimodal data based on the data vector under each modality and the query vector, the graph index including data nodes corresponding to the data vector, query nodes corresponding to the query vector, and a connection relationship between the data nodes and the query nodes.

[0083] Optionally, the generating module 43 is also used for

[0084] Initializing the graph structure, wherein each of the query vectors is initialized as a query node in the graph structure, and each of the data vectors is initialized as a data node in the graph structure;

[0085] Based on the query node and the data node, an adjacency relationship between the query vector and the data vector is generated to obtain a connection relationship between each node in the initial graph index.

[0086] Optionally, the generating module 43 is further used for:

[0087] For the query node, calculate a first distance value between the query node and each data node to obtain multiple first distance values, sort the first distance values ​​from small to large, select a first preset number of first target distances from the multiple first distance values, use the data nodes corresponding to the first target distances as neighbor nodes of the query node, and generate a connection relationship between the neighbor nodes pointing from the query node to the query node;

[0088] For the data node, the second distance value between the data node and each query node is calculated to obtain multiple second distance values, and the query node corresponding to the minimum distance is selected from the multiple second distance values ​​as the neighbor node of the data node, and a connection relationship is generated from the data node to the data nodes.

[0089] Optionally, the generating module 43 is further used for:

[0090] Optimizing the initial graph index to obtain the graph index;

[0091] The optimizing the initial graph index to obtain the graph index comprises:

[0092] Searching for nodes to be optimized from the initial graph index, wherein the nodes to be optimized include at least one of the following: isolated nodes, nodes in a separated subgraph;

[0093] When the node to be optimized is a query node, a third distance value between the node to be optimized and each data node is calculated to obtain a plurality of third distance values, a second preset number of third target distances are screened from the plurality of third distance values, data nodes corresponding to the third target distances are used as supplementary neighbor nodes of the node to be optimized, and a connection relationship pointing from the node to be optimized to the supplementary neighbor nodes of the node to be optimized is generated;

[0094] When the node to be optimized is a data node, a fourth distance value between the node to be optimized and each query node is calculated to obtain multiple fourth distance values, a third preset number of fourth target distances are screened from the multiple fourth distance values, the query nodes corresponding to the fourth target distances are used as supplementary neighbor nodes of the node to be optimized, and a connection relationship from the node to be optimized to the supplementary neighbor nodes of the node to be optimized is generated.

[0095] Optionally, the generating module 43 is further used for:

[0096] In the process of optimizing the initial graph index, dynamically obtaining an index value for measuring the scale of the graph index, the index value comprising one of the following: the number of nodes, the degree of the graph;

[0097] When the indicator value is greater than or equal to a preset indicator value, the second preset number and the third preset number are reduced.

[0098] Optionally, the generating module 43 is further used for:

[0099] For any of the query nodes, taking the query node as the starting point and the target data node as the end point, traverse the connection relationship between the nodes in the graph index, and search for all paths from the query node to each target data node;

[0100] From all the paths from the query node to each target data node, the shortest path and alternative paths from the query node to each target data node are selected and recorded; based on the alternative paths, a connection relationship between the query node and each target data node is generated and recorded, wherein the path length indicated by the alternative path is greater than the length of the shortest path.

[0101] Optionally, the generating module 43 is further used for:

[0102] Get the node to be changed;

[0103] According to the change operation indicated by the node to be changed, the node to be changed is added or deleted from the graph index to obtain a changed node;

[0104] Regenerate the graph index for multimodal data based on the changed nodes.

[0105] On the other hand, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the query method based on a multimodal database described in any embodiment of the present application.

[0106] Among them, in the computer program product, an optional implementation form of the program module architecture of the computer program that implements each step of the gas detection method can be a query device based on a multimodal database.

[0107] See also Figure 5An embodiment of the present application provides a query device based on a multimodal database, including: a data acquisition module 51, used to obtain a query request containing at least one modal data and a configured search parameter; a data conversion module 52, used to convert the query request into a feature space that is the same as the data vector in the graph index described by the query based on the multimodal database provided in any embodiment of the present application, to obtain a feature vector to be queried; a query module 53, used to search for a target data vector matching the feature vector to be queried in the graph index based on the search parameter and the feature vector to be queried, and display the target data vector.

[0108] Optionally, wherein the search parameters include a preset number of search layers and a preset number of candidate nodes, the query module 53 is further used to:

[0109] In the graph index, searching for a query node that is closest to the feature vector to be queried as a traversal query start node, and determining the query start node as a current search node;

[0110] In the graph index, in the next layer of the current search node, according to the adjacency relationship of the current search node, the neighbor nodes of the current search node are searched, and the current most similar neighbor nodes are determined among the neighbor nodes of the current search node, and according to the preset number of candidate nodes, the candidate nodes corresponding to the current most similar neighbor nodes are recorded, the current search node is updated with the most similar neighbor nodes, and the search in the graph index is continued;

[0111] During the search process, a current length index of the current search path is obtained. When the current length index indicates that the current search path is invalid, the current search node is returned to the next layer, and a target candidate node is selected from the candidate nodes corresponding to the current most similar neighbor node as the current search node, and the search is continued in the graph index until the search termination condition is met, and the target search path is obtained;

[0112] Based on the target search path, the target data vector is obtained.

[0113] Optionally, the query module 53 is further used for:

[0114] According to the target search path, obtaining a target data vector node in the target search path;

[0115] The similarity between the target data vector node and the feature vector to be queried is calculated, and according to the similarity between the target data vector node and the feature vector to be queried, a fourth preset number of target data vector nodes are selected as the target data vector.

[0116] It can be understood by those skilled in the art that Figure 4The structure of the index generation device based on the multimodal database does not constitute a limitation on the index generation device based on the multimodal database. Figure 5 The structure of the query device based on a multimodal database does not constitute a limitation on the query device based on a multimodal database, and the various modules can be implemented in whole or in part by software, hardware, and a combination thereof. The above modules can be embedded in or independent of the controller in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the controller can call and execute the operations corresponding to the above modules. In other embodiments, the index generation device based on a multimodal database may include more or fewer modules than shown in the figure.

[0117] See also Figure 6 In another aspect of the embodiment of the present application, a computing device 200 is provided, including a memory 3011 and a processor 3012. The memory 3011 stores a computer program. When the computer program is executed by the processor, the processor 3012 executes the steps of the index generation method based on the multimodal database provided in any of the above embodiments of the present application. The terminal device 100 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch) or similar devices, a robotic device.

[0118] The index generation devices based on the multimodal database include but are not limited to: embedded devices and terminal devices.

[0119] The processor 3012 is the control center, which uses various interfaces and lines to connect various data of the entire computer device, and executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 3011, and calling data stored in the memory 3011. Optionally, the processor 3012 may include one or more processing cores; preferably, the processor 3012 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 3012.

[0120] The memory 3011 can be used to store software programs and modules. The processor 3012 executes various functional applications and data processing by running the software programs and modules stored in the memory 3011. The memory 3011 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 3011 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 3011 may also include a memory controller to provide the processor 3012 with access to the memory 3011.

[0121] On the other hand, an embodiment of the present application further provides a storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the index generation method based on a multimodal database provided in any of the above embodiments of the present application.

[0122] Those of ordinary skill in the art can understand that the implementation of all or data flows in the methods provided in the above embodiments can be completed by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the process of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0123] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating an index based on a multimodal database, characterized in that: include: Obtain multimodal data and query data for building a database; Extracting features of each modality of the multimodal data to obtain a data vector for each modality; Based on the query data, generate a query vector; Based on the data vectors under each mode and the query vector, a graph index for multimodal data is generated, wherein the graph index includes a data node corresponding to the data vector, a query node corresponding to the query vector, and a connection relationship between the data node and the query node, wherein the adjacency relationship between the data node and the query node includes a first connection edge and a second connection edge, wherein the first connection edge is directed from a query node to multiple adjacent data nodes, and the direction of the second connection edge is from the data node to multiple adjacent query nodes; The generating of the graph index for the multimodal data based on the data vector under each modality and the query vector includes: obtaining an initial graph index, optimizing the initial graph index, and obtaining the graph index; The optimizing the initial graph index to obtain the graph index comprises: Searching for nodes to be optimized from the initial graph index, wherein the nodes to be optimized include at least one of the following: isolated nodes, nodes in a separated subgraph; When the node to be optimized is a query node, a third distance value between the node to be optimized and each data node is calculated to obtain a plurality of third distance values, a second preset number of third target distances are screened from the plurality of third distance values, data nodes corresponding to the third target distances are used as supplementary neighbor nodes of the node to be optimized, and a connection relationship pointing from the node to be optimized to the supplementary neighbor nodes of the node to be optimized is generated; When the node to be optimized is a data node, a fourth distance value between the node to be optimized and each query node is calculated to obtain multiple fourth distance values, a third preset number of fourth target distances are screened from the multiple fourth distance values, the query nodes corresponding to the fourth target distances are used as supplementary neighbor nodes of the node to be optimized, and a connection relationship from the node to be optimized to the supplementary neighbor nodes of the node to be optimized is generated.

2. The index generation method based on a multimodal database according to claim 1, characterized in that: The obtaining of the initial graph index comprises: Initializing the graph structure, wherein each of the query vectors is initialized as a query node in the graph structure, and each of the data vectors is initialized as a data node in the graph structure; Based on the query node and the data node, an adjacency relationship between the query vector and the data vector is generated to obtain a connection relationship between each node in the initial graph index.

3. The index generation method based on a multimodal database according to claim 2, characterized in that: The generating, based on the query node and the data node, a node for representing the adjacency relationship between the query vector and the data vector, and obtaining the connection relationship between the nodes in the graph index comprises: For the query node, calculate a first distance value between the query node and each data node to obtain multiple first distance values, sort the first distance values ​​from small to large, select a first preset number of first target distances from the multiple first distance values, use the data nodes corresponding to the first target distances as neighbor nodes of the query node, and generate a connection relationship between the neighbor nodes pointing from the query node to the query node; For the data node, the second distance value between the data node and each query node is calculated to obtain multiple second distance values, and the query node corresponding to the minimum distance is selected from the multiple second distance values ​​as the neighbor node of the data node, and a connection relationship between the neighbor nodes pointing from the data node to the data node is generated.

4. The index generation method based on a multimodal database according to claim 1, characterized in that: The method further comprises: In the process of optimizing the initial graph index, dynamically obtaining an index value for measuring the scale of the graph index, the index value comprising one of the following: the number of nodes, the degree of the graph; When the indicator value is greater than or equal to a preset indicator value, the second preset number and the third preset number are reduced.

5. The index generation method based on a multimodal database according to claim 1, characterized in that: The method further comprises: For any of the query nodes, taking the query node as the starting point and the target data node as the end point, traverse the connection relationship between the nodes in the graph index, and search for all paths from the query node to each target data node; From all the paths from the query node to each target data node, the shortest path and alternative paths from the query node to each target data node are selected and recorded; based on the alternative paths, a connection relationship between the query node and each target data node is generated and recorded, wherein the path length indicated by the alternative path is greater than the length of the shortest path.

6. The index generation method based on a multimodal database according to claim 1, characterized in that: The method further comprises: Get the node to be changed; According to the change operation indicated by the node to be changed, the node to be changed is added or deleted from the graph index to obtain a changed node; Regenerate the graph index for multimodal data based on the changed nodes.

7. A query method based on a multimodal database, characterized in that: The method comprises: Obtaining a query request including at least one modality data and configured search parameters; Converting the query request into the same feature space as the data vector in the graph index as claimed in any one of claims 1 to 6 to obtain a feature vector to be queried; Based on the search parameter and the feature vector to be queried, a target data vector matching the feature vector to be queried is searched in the graph index, and the target data vector is displayed.

8. The query method based on a multimodal database according to claim 7, characterized in that: The search parameters include a preset number of search layers and a preset number of candidate nodes, and searching the graph index for a target data vector matching the feature vector to be queried based on the search parameters and the feature vector to be queried includes: In the graph index, searching for a query node that is closest to the feature vector to be queried as a traversal query start node, and determining the query start node as a current search node; In the graph index, in the next layer of the current search node, according to the adjacency relationship of the current search node, the neighbor nodes of the current search node are searched, and the current most similar neighbor nodes are determined among the neighbor nodes of the current search node, and according to the preset number of candidate nodes, the candidate nodes corresponding to the current most similar neighbor nodes are recorded, the current search node is updated with the most similar neighbor nodes, and the search in the graph index is continued; During the search process, a current length index of the current search path is obtained. When the current length index indicates that the current search path is invalid, the current search node is returned to the next layer, and a target candidate node is selected from the candidate nodes corresponding to the current most similar neighbor node as the current search node, and the search is continued in the graph index until the search termination condition is met, and the target search path is obtained; Based on the target search path, the target data vector is obtained.

9. The query method based on a multimodal database according to claim 8, characterized in that: The obtaining the target data vector based on the target search path comprises: According to the target search path, obtaining a target data vector node in the target search path; The similarity between the target data vector node and the feature vector to be queried is calculated, multiple similarities are sorted from large to small, and the target data vector nodes that are ranked in the front fourth preset number are selected as the target data vector.

Images