Query method based on multimodal database

Abstract

The present invention discloses a query method based on a multimodal database, the method comprising: obtaining data to be processed including multiple modalities; based on the data to be processed, converting the data to be processed into category data corresponding to the data to be processed based on a category model; obtaining a request to be queried, decomposing the request to be queried into multiple sub-queries; based on the multiple sub-queries, obtaining multiple sub-results, based on the category data, merging the multiple sub-results to obtain a merged result; converting the merged result into target query data in a target data format.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a query method based on a multimodal database. Background Art

[0002] In modern database systems, with the rapid development of big data and cloud computing technologies, the diversity and complexity of data are increasing. Currently, many application systems need to process different types of data models at the same time, such as relational data, document data, graph data, key-value data, etc. However, traditional database management systems usually only support a single data model, which makes it difficult to cope with the needs of modern applications for multi-model data storage and query. Summary of the invention

[0003] In order to solve the existing technical problems, the embodiment of the present application provides a query method based on a multimodal database, which can realize multimodal data query and realize redundant processing and optimization of cross-model queries to improve query efficiency and enhance system performance.

[0004] In a first aspect, a query method based on a multimodal database is provided, comprising: obtaining data to be processed including multiple modalities; based on the data to be processed, converting the data to be processed into category data corresponding to the data to be processed based on a category model; obtaining a request to be queried, and decomposing the request to be queried into multiple sub-queries; based on the multiple sub-queries, obtaining multiple sub-results, and based on the category data, merging the multiple sub-results to obtain a merged result; and converting the merged result into target query data in a target data format.

[0005] The present application converts the data to be processed under multiple modalities into the category data under the category model, so as to unify the data representation under various modalities, thereby ensuring the representation consistency of data in different modalities. When the data structures of different modalities are consistent, the accessibility between data in different modalities can be ensured. After converting the basic data of each database into the category data, the query request is obtained, and the query request is decomposed into multiple sub-queries according to the mapping of the pattern category in the category model. The decomposed sub-queries are translated into the query language of each database management system. At the same time, the results returned by each database can be uniformly converted into the category model for integration processing, and the data under different modalities are fused and processed to obtain a merged result, and the merged result is converted to the target query data under the target data format. Therefore, the present application can realize multi-modal data query, and can realize redundant processing and optimization of cross-model queries to improve query efficiency and enhance system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1is an application environment diagram of a query method based on a multimodal database in an embodiment;

[0007] Figure 2 is a flow chart of a query method based on a multimodal database in one embodiment;

[0008] Figure 3 is a flow chart of converting data to be processed into category data based on a category model in one embodiment;

[0009] Figure 4 is a schematic diagram of a query device based on a multimodal database in an embodiment;

[0010] Figure 5 Schematic diagram of a terminal device in an embodiment. DETAILED DESCRIPTION

[0011] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0013] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it should be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] See also Figure 1 In one embodiment, a query method based on a multimodal database and an application environment diagram of the query method based on a multimodal database include multiple terminal devices 100 and computing devices 200, wherein the terminal device 100 can be connected to the computing device 200 for communication, and the query method based on a multimodal database provided in the embodiment of the present application can be applied to the computing device 200, the computing device 200 obtains multimodal data, processes these multimodal data, obtains category data suitable for multimodal data retrieval, and represents the multimodal data as a collection of instance objects and morphisms in the category data. The multimodal data includes but is not limited to text data, image data, video data, sound data, sensor data, location data, etc. After the graph index is established, the terminal device 100 can receive a query request based on the user interface input. After the computing device 200 obtains the query request, it searches for data matching the query request based on the established category model and displays it.

[0015] The concept of multi-model database came into being. This type of database can support multiple data models at the same time and provide unified storage and management for different types of data. However, in actual applications, different multi-model databases have great differences in model combination methods, query languages, storage strategies, etc., which leads to the following problems:

[0016] 1. Lack of a unified query language: There is currently no recognized standard query language that can support queries on multi-model data at the same time. Different database systems use their own query languages ​​or make non-standard extensions to SQL, which increases the learning and usage costs for users.

[0017] 2. Inconsistent query result models: Query results from different systems usually rely on their original core models, which cannot meet the requirements of cross-model queries and unified representation of query results, resulting in diversity and inconsistency in query results.

[0018] 3. Cross-model redundancy and optimization difficulties: Although multi-model databases can naturally support cross-model redundancy, they lack an effective mechanism to select the optimal query strategy, making it difficult to optimize query performance.

[0019] 4. High query complexity: Queries in multi-model databases involve multiple different data models. Query execution not only needs to consider cross-model operations, but also needs to convert between different query languages, which increases the complexity of the system.

[0020] See also Figure 2 , is a flow chart of a query method based on a multimodal database provided in an embodiment of the present application. The query method based on a multimodal database is applied to a computing device, and the query method based on a multimodal database includes the following steps:

[0021] S11. Acquire data to be processed including multiple modalities.

[0022] In this embodiment, the data to be processed includes but is not limited to at least one of the following: text data, image data, video data, sound data, sensor data, location data, etc. These different types of data can be stored in combination in the same database to support richer data analysis and applications, wherein text data includes but is not limited to various forms of text, such as articles, comments, emails, etc.; image data includes but is not limited to photos, charts, maps, etc.; video data includes but is not limited to video files, streaming media, etc.; sound data includes but is not limited to audio files, voice messages, etc.; sensor data includes but is not limited to: data collected by various sensors such as temperature, humidity, and light; location data includes but is not limited to GPS data, geographic location information, etc. For data in any mode, one or more files may be included, for example, image data may include many images. The data to be processed may be data from one or more different types of databases.

[0023] S12. Based on the data to be processed, convert the data to be processed into category data corresponding to the data to be processed based on the category model.

[0024] In this embodiment, a category model is used to abstractly model data of different modalities and to unify data of different modalities. A category includes: an instance object set, a morphism set, and a synthesis operation acting on the morphism. One instance object is modeled as an object tuple, including an automatically assigned unique identifier, an optional user-defined name (such as the name of the entity type in the data model), an object attribute set, and an attribute group set that can uniquely identify the object, wherein the unique representation is used to represent the instance object, the user-defined name is similar to the entity type name in the general database model, and the attribute group set is similar to the candidate key set in the relational database; a morphism is modeled as a morphism tuple, including a label that can distinguish all morphisms, the domain of the morphism, the codomain of the morphism, the minimum number of occurrences of the morphism, and the maximum number of occurrences of the morphism; the result of the synthesis operation of two morphisms M1 and M2 is a new morphism M3, and the fields of M3: the label is the product of the labels of M2 and M1, the domain is the domain of M1, the codomain is the codomain of M2, and the upper and lower limits are the minimum value of the lower limit and the maximum value of the upper limit, which is similar to the operation of a composite function. The connection between instance objects is through morphism. Morphism can be understood as an arrow between two instance objects, indicating a certain relationship between domain and codomain.

[0025] The data to be processed is data in a general database, such as an Oracle database or a MySQL database. The data to be processed needs to be converted into a category model to obtain the category data corresponding to the data to be processed.

[0026] Abstract representation method of multimodal data based on category model: Using the concept of small categories in category theory, multimodal data is represented as a collection of objects and morphisms. This category model can cover all common data models and their combinations, providing a unified representation of multimodal data.

[0027] S13: Obtain a pending query request, and decompose the pending query request into multiple sub-queries.

[0028] In this embodiment, the query request is a user's request, and the basic data is usually distributed in different databases and data models. It is necessary to decompose the query request into multiple sub-queries according to the mapping of the mode category in the category model, and execute them in the corresponding database management system respectively.

[0029] S14. Based on the multiple sub-queries, multiple sub-results are obtained, and based on the category data, the multiple sub-results are merged to obtain a merged result.

[0030] In this embodiment, the decomposed subqueries are translated into the query language of each database management system. For example, part of the subqueries are translated into SQL, and the other part of the subqueries are translated into the query language of MongoDB to ensure semantic consistency. As a unified abstract layer for multimodal data, the category model can decompose and map query requests to different database models (such as SQL, MongoDB), and at the same time, it can uniformly convert the results returned by each database into the category model for integration processing to obtain a merged result. Since this category model structure can operate records from different data models in a unified manner, it can fuse and process data under different modalities. The method of mapping each sub-result into the category model is the same as the method of mapping the data to be processed into the category model.

[0031] S15. Convert the merged result into target query data in a target data format.

[0032] In this embodiment, the target data format indicates the format that the user needs to convert, and the category data results of the merged results are converted into the target data format to obtain the target query data, and finally the merged results are converted into the format required by the user (such as JSON, XML, etc.).

[0033] In the above embodiment, the data to be processed under multiple modalities are converted into category data under the category model, so that the data representation under various modalities can be unified, thereby ensuring the representation consistency of data in different modalities. When the data structures of different modalities are consistent, the accessibility between data in different modalities can be ensured; after the basic data of each database is converted into category data, the query request to be queried is obtained, and the query request is decomposed into multiple sub-queries according to the mapping of the pattern category in the category model, and the decomposed sub-queries are translated into the query language of each database management system. At the same time, the results returned by each database can be uniformly converted into the category model for integration processing, and the data under different modalities are fused and processed to obtain a merged result, and the merged result is converted to the target query data under the target data format. Therefore, the present application can realize multi-modal data query, and can realize redundant processing and optimization of cross-model queries to improve query efficiency and enhance system performance.

[0034] In some embodiments, Figure 3 As shown, Figure 3 1 is a flow chart of converting the data to be processed into category data based on the category model in one embodiment; S12 also includes:

[0035] S121, based on each record to be processed in the data to be processed, generate a directed tree structure corresponding to each record to be processed.

[0036] In this embodiment, the data to be processed may be data extracted from different modal databases, and the different modal databases include but are not limited to: relational databases, document databases, image databases, vector databases, etc. In the category model, each record to be processed is represented as a directed tree, wherein the tree structure includes multiple nodes and edges between two nodes, each node indicates an attribute in a type of data, and the edge indicates the nested relationship between the attributes corresponding to the two nodes. For example, a record to be processed is represented as a schedule relationship between a student and a teacher. There are three nodes in the directed tree, and these three nodes represent students, schedules, and teachers respectively. The edge between the student and the schedule represents the class that the student needs to take, and the edge between the teacher and the schedule represents the class that the teacher needs to take. The attributes corresponding to each node may be different according to the data of different modalities, including but not limited to fields, time, image data, etc. When there are many records to be processed, many directed trees can be obtained. For example, each record to be processed is similar to a row of data in a data table, and the attribute corresponding to each node is similar to the attribute value indicated by a data unit in a row of data in the data table, such as the name of a course, etc.

[0037] S122: For each type of data in the data to be processed, obtain a first access path corresponding to each type of data.

[0038] In this embodiment, the first access path is used to traverse the generated directed tree, thereby generating category data corresponding to the record to be processed. Each type of data corresponds to a first access path, for example, the relational database corresponds to the first access path A1, the image database corresponds to the first access path A2, the document database corresponds to the first access path A3, and so on. The role of the first access path is to provide a method for mapping the data in the general database model to the category representation during the conversion process from the model to the category, thereby ensuring the consistency and accessibility of the data. Through the first access path, all attributes at the same level in the tree can be quickly accessed. These attributes correspond to the same instance category object, thereby realizing structured representation and effective processing of the data. In order to make it easier to understand and process, for example, a JSON-like text representation is used to describe the first access path. As the core of the mapping, the first access path specifically describes the internal organization of the data category in a tree structure, reflecting the nested relationship between different attributes.

[0039] Taking a relational database as an example, in PostgreSQL, the primary key can be a set of attribute signatures that constitute the primary identifier. The first access path is represented by a tree structure, in which each node corresponds to an attribute of the data category, and the edge represents the nested relationship between the attributes. The root node of the tree is an auxiliary node, and its child nodes correspond to the top-level attributes of the data category. Each node is also the root of the first access subpath, describing the structure of the corresponding nested attributes. The name of the attribute may be static, anonymous, or dynamically derived, and the context information provides the specific location of the attribute in the data structure. The attribute value may be a simple value, an array, or a nested structure, the latter two of which are considered to be complex values ​​of the attribute. Data categories include, but are not limited to, data categories (Kind) in databases with different data modes such as tables, documents, and key values. Attributes refer to the attributes of these data categories. The first access path is a tree structure that describes the nested relationship between attributes of different data categories.

[0040] S123, based on the directed tree structure corresponding to each record to be processed and the first access path corresponding to each type of data, convert the record to be processed into the category data corresponding to the record to be processed, and obtain the instance object corresponding to each record to be processed.

[0041] Optionally, S123 may also include:

[0042] Traversing the directed tree structure corresponding to each to-be-processed record to obtain a plurality of to-be-processed nodes, storing the plurality of to-be-processed nodes and the first access path corresponding to each type of data in a stack, storing context information in the traversal process through the stack, and for each to-be-processed node, determining whether an attribute corresponding to each to-be-processed node exists in the current category data;

[0043] Obtain a first node to be processed whose attributes exist in the current scope data, obtain an instance object corresponding to the first processing node in the current scope data, and update the active domain of the instance object corresponding to the first processing node, and obtain a second node to be processed whose attributes do not exist in the current scope data, and create an instance object corresponding to the second processing node in the current scope data.

[0044] In this embodiment, the current category data represents the currently stored category data. The active domain is used to represent the valid data set of the instance object, which is the set of all attribute values ​​in the instance object. These attribute values ​​constitute a complete description of the instance object. For example, the user table of this traditional database is UserID | Name | Age | DeptID, 1 | Alice | 25 | 101; 2 | Bob | 30 | 102; if the user table is converted into a category, for the first record, there will be an instance object in the category instance: super identifier: including UserID=1, etc., active domain: {Name="Alice", Age=25, DeptID=101}, this instance object represents a specific user entity.

[0045] S124, based on the instance object corresponding to each record to be processed, determining the target category of each record to be processed in the category data, and updating the attribute set corresponding to the target category.

[0046] In this embodiment, once the stack is empty, it means that all the nodes to be processed have been successfully converted into the instance category. After checking the integrity, the updated instance category is returned, which contains all the data converted from the general database model.

[0047] Optionally, S124 also includes:

[0048] Acquire an associated object related to the instance object corresponding to each to-be-processed record from the current category data, and determine a target category of the instance object corresponding to each to-be-processed record in the category data according to the associated object and the category of the associated object;

[0049] Obtain a super identifier set corresponding to the target category, and update the attributes in the instance object corresponding to each super identifier in the super identifier set.

[0050] In this embodiment, a category indicates a modality, such as a document category, a vector category, a picture category, an audio category, and the like. For an instance object corresponding to a record to be processed, it is necessary to determine the category to which the created instance object belongs, and then construct the category data associated with the category. A root object or root morphism related to the instance object corresponding to the record to be processed can be obtained, wherein the root object is an instance object in the category, and for any other object in the category, there is a unique morphism from the root object to the instance object, and the root morphism refers to a morphism from the root object to another object. The target category of the instance object corresponding to the record to be processed in the category data is determined according to the category in which the root object or the root morphism is located. The super identifier set corresponding to the target category includes super identifiers corresponding to each instance object in the target category, and the super identifier can be used to identify the instance object. For the instance object being constructed, that is, the instance object corresponding to the record to be processed, the attribute modification is performed in the active domain of the instance object corresponding to each super identifier in the set, and the attribute modification includes: adding a new attribute, updating the value of the attribute, or deleting the attribute. For each attribute with sub-attributes, corresponding processing is performed according to the attribute type, which may involve creating a new attribute, updating the value of an existing attribute, or adding a new relationship. The attribute type includes simple attribute, complex attribute or array. Attributes with sub-attributes are complex attributes, and attributes without sub-attributes are simple attributes.

[0051] In the above embodiment, each record to be processed is first converted into a directed tree structure, and then the first access path is used to traverse the nodes in the directed tree to generate an instance object corresponding to each record to be processed, and the target category of the instance object corresponding to each record to be processed is determined, and the attributes in the target category are updated, so that a unified representation of data of different modalities can be achieved, multi-modal data query can be achieved, and redundant processing and optimization of cross-model queries can be achieved to improve query efficiency and enhance system performance.

[0052] In some embodiments, converting the merged result into target query data in a target data format comprises:

[0053] Acquire multiple mapping tuples from the target data format to the category model, each of the mapping tuples indicating a data category and a data structure and data content in the data category of the record, each of the mapping tuples including at least the following: a database identifier, a data category identifier, a root object associated with the data category identifier, a root morphism, a morphism signature set constituting a primary key, a reference set, and a second access path describing the data structure in the data category;

[0054] Based on the mapping tuple, the merged result is converted into target query data in a target data format.

[0055] In this embodiment, when the complex data structure of the general database model is converted into the category model, the complex data structure of the general database model is decomposed into multiple mapping tuples, each mapping tuple describes in detail a data category and the internal structure and content of its records. Specifically, each mapping tuple contains a database identifier (such as a connection string), a data category identifier, a root object associated with the data category identifier, a root morphism, a morphism signature set constituting a primary key, a reference set, and a second access path describing the data structure in the data category. The database identifier is used to uniquely identify the database, the data category identifier is used to uniquely identify a data category, and the second access path is used to traverse the data in the category model. The morphism signature set constituting the primary key is an ordered set of a set of morphism signatures used to uniquely identify instance objects in the category model, and each morphism signature describes a specific morphism feature. The morphism signature set constituting the primary key forms a unique identification mechanism for instance objects by combining multiple morphism signatures; the reference set is the core mechanism for managing the association relationship between different data categories in the category model, which stores the reference relationship between different data categories and their integrity constraint information.

[0056] The second access path is a tree structure, where each node represents an attribute and the edges represent the nested relationship between attributes. This structure enables records from data models of different modalities to be operated in a unified way.

[0057] In an optional embodiment, converting the merged result into target query data in a target data format based on the mapping tuple includes:

[0058] Based on the mapping element, acquiring a pattern category and an instance category associated with the merge result in the category data, wherein the pattern category represents a definition description of a data structure in the category data, and the instance category represents data of an instance object in the category data;

[0059] Based on the schema categories, creating a database definition statement;

[0060] Based on the instance scope, creating a data manipulation statement, and inserting the data manipulation statement into the target data format;

[0061] Add integrity constraint statements;

[0062] The target query data is generated based on the database definition statement, the data manipulation statement and the integrity constraint statement.

[0063] In this embodiment, the pattern category is a definition description of the category structure, which includes all the attributes of the category model and the relationship between the attributes, and defines how the data in the data database in the category model is stored and how they are related to each other. The pattern category is similar to the schema in the database. The instance category includes multiple instance objects, which are specific data instances of these structures and contain actual data values. This step provides a basis for conversion and ensures that all necessary information can be extracted from the category model.

[0064] Optionally, the creating a database definition statement based on the mode category includes:

[0065] Determine an attribute including sub-attributes in the pattern category as a complex attribute, store the sub-attributes corresponding to the complex attribute in a stack, process the sub-attributes corresponding to the complex attribute in the stack to obtain processed attribute data, and generate a database definition statement corresponding to the complex attribute in a wrapper based on the processed attribute data;

[0066] The attributes without sub-attributes in the pattern category are determined as simple attributes, and a database definition statement corresponding to the simple attribute is generated in the wrapper.

[0067] The creating a data manipulation statement based on the instance scope and inserting the data manipulation statement into the target data format comprises:

[0068] Obtain all object pairs in the data category with root morphisms in the instance category, and construct a create data manipulation statement for each pair of object pairs;

[0069] A root object of a data class without a root morphism in the instance category is obtained, the root object is used to obtain a super identifier set in the instance category, and a data manipulation statement is created based on the super identifier set.

[0070] In this embodiment, if a data category is completely independent and does not depend on other data categories, then the data category has no root morphism. On the contrary, if there is a structural association between the data categories, such as a parent-child relationship or a nested structure, then the two data categories will be represented in the category model through root morphisms.

[0071] Add integrity constraints (IC statements). Integrity constraints include identifiers and references to ensure data consistency and correctness. First, process the identifiers, collect the names of all related identifiers and add them to the wrapper. Then process the references, for each reference, collect the signature name pairs and create reference pairs, then add them to the wrapper, and finally return the created integrity constraint statement.

[0072] After the database definition statements (Data Definition Language, DDL), data manipulation statements (Data Manipulation Language, DML) and IC statements are created, these statements will be sent to the target database system for execution. DDL statements are executed first to create the data structure required by the data model. Subsequently, DML statements are executed to fill the data into the created structure. Finally, integrity constraints are added to ensure the correctness and integrity of the data. In an optional implementation, the converted target query data can also be verified to verify whether the conversion is successful.

[0073] In the above embodiment, the merged result is obtained by mapping each sub-result into a category model for merging, thereby realizing multi-modal data query and merging, and then converting the merged result into target query data in a target data format, which can realize multi-modal data query and redundant processing and optimization of cross-model queries, so as to improve query efficiency and enhance system performance.

[0074] In some embodiments, obtaining a plurality of sub-results based on the plurality of sub-queries, and merging the plurality of sub-results based on the category data to obtain a merged result includes:

[0075] Translate each subquery into the query language corresponding to each subquery;

[0076] Use the query language corresponding to each subquery to query and obtain the subresults corresponding to each subquery;

[0077] Based on the category data, convert each sub-result into each sub-instance object;

[0078] Each sub-instance object is merged to obtain a merged result.

[0079] In the above embodiment, the query request is first mapped to the category model, and then the query is decomposed into multiple sub-queries according to the storage distribution of multiple data types in the category model, and then each sub-query is executed in its corresponding database management system. Finally, the results of each sub-query are converted into a representation on the category model, merged and further processed to obtain the final query result, thereby realizing multi-modal data query, and can realize redundant processing and optimization of cross-model queries to improve query efficiency and enhance system performance.

[0080] In an optional embodiment, the multi-modal database unified query language (MUDSQL, Multi-Model UNVDB Query Language) can be used to achieve the fusion of multi-modal data. MUQSQL can execute queries on abstract category models. MUQSQL supports common query operations, such as selection, projection, connection, aggregation, filtering, etc., and is suitable for query requirements across multiple modal data models. When the query request is executed, the query request is first mapped to the category model through the MUQSQL query language, and then the query is decomposed into multiple sub-queries according to the storage distribution of multiple data types in the category model, and then each sub-query is executed in its corresponding database management system (DBMS). Finally, the results of each sub-query are converted into a representation on the category model, merged and further processed to obtain the final query result. By representing cross-model redundant data in the category model, providing multiple query decomposition schemes, and selecting the optimal query execution plan through the cost estimation model, the query efficiency is improved.

[0081] Unified query language MUQSQL,In order to effectively handle query operations across multi-model databases, the present invention designs a unified query language for multimodal databases, called MUQSQL. MUQSQL can execute queries on abstract category models, shield the differences in underlying data models, and provide a consistent query interface. MUQSQL enables users to quickly get started and execute complex cross-model queries.

[0082] MUQSQL's syntax structure, MUQSQL inherits the SQL core syntax structure and expands it to meet the query requirements of multi-model data. Its basic query structure includes the following parts:

[0083] WITH: represents a common table expression (CTE), which is used to split and reuse in complex queries.

[0084] SELECT: Specifies the query result mode and the variables returned.

[0085] FROM: Specifies the data source of the query, which can be a table, view, subquery, etc.

[0086] JOIN: supports join operations, including inner join, outer join, and self join.

[0087] WHERE: Defines the query conditions and supports complex Boolean expressions.

[0088] GROUP BY: Specifies grouping variables for aggregation operations.

[0089] HAVING: Filters the aggregation results.

[0090] ORDER BY: Specifies how the results are sorted.

[0091] LIMIT: Limit the number of results returned.

[0092] OFFSET: Specifies the offset of the result, used for paging.

[0093] In order to better understand the comparison between the multimodal database based on the category model and the general query, the following example is used for illustration.

[0094] Example 1: Customer product recommendations based on image tags

[0095] An image database (such as a file system or object storage, supporting metadata queries) where product images are stored, and each image has some labels (such as color, category, etc.) stored in a document database. You want to recommend products based on customer preferences (such as the color the customer likes).

[0096] WITH CustomerPreferences AS (

[0097] Query a document database for customer preferences (e.g., the customer's favorite color)

[0098] SELECT customer_doc->>'name' AS name, customer_doc->>'preferred_color' AS preferred_color, id AS customer_id

[0099] FROM customers_document_store

[0100] ),

[0101] ProductImages AS (

[0102] Query the image database for images of products and their labels (e.g., color) from their metadata

[0103] SELECT image_metadata->>'product_id' AS product_id, image_metadata->>'color' AS color, image_path

[0104] FROM image_storage_system

[0105] WHERE image_type = 'product'

[0106] ),

[0107] RecommendedProducts AS (

[0108] Recommend products based on customer preference colors

[0109] SELECT CustomerPreferences.name, ProductImages.product_id,ProductImages.image_path

[0110] FROM CustomerPreferences

[0111] JOIN ProductImages ON CustomerPreferences.preferred_color =ProductImages.color )

[0113] Returns the recommended products for the customer and their image paths

[0114] SELECT name, product_id, image_path

[0115] FROM RecommendedProducts;

[0116] CustomerPreferences means extracting customer preference information from the document database, such as the customer's favorite color.

[0117] ProductImages represents a query for product images and their labels (colors) from an image storage system.

[0118] JOIN means combining customer preferences with product image labels to recommend products that match customer preferences.

[0119] SELECT represents a path that returns the customer's recommended products and their corresponding images.

[0120] Example 2: Customer service analysis based on audio content

[0121] The audio files of customer service calls are stored in the object storage, and the metadata information of the audio (such as customer ID, call duration, sentiment analysis results, etc.) is stored in the document database. The query will extract the customer's sentiment analysis results from the audio data and analyze them in combination with the customer's purchase history.

[0122] WITH AudioMetadata AS (

[0123] Query the metadata of the audio from the document database, including the customer ID and sentiment analysis results

[0124] SELECT audio_doc->>'customer_id' AS customer_id, audio_doc->>'sentiment' AS sentiment, audio_doc->>'duration' AS duration, audio_file_path

[0125] FROM audio_metadata_store

[0126] ),

[0127] CustomerPurchases AS (

[0128] Query customer purchase records from a relational database

[0129] SELECT customer_id, SUM(price) AS total_spent

[0130] FROM orders

[0131] JOIN order_items ON orders.id = order_items.order_id

[0132] GROUP BY customer_id )

[0134] Analyze based on audio sentiment analysis results and customer purchase records

[0135] SELECT AudioMetadata.customer_id, AudioMetadata.sentiment,AudioMetadata.duration, CustomerPurchases.total_spent, AudioMetadata.audio_file_path

[0136] FROM AudioMetadata

[0137] JOIN CustomerPurchases ON AudioMetadata.customer_id =CustomerPurchases.customer_id

[0138] WHERE AudioMetadata.sentiment = 'negative';

[0139] AudioMetadata: extracts audio metadata of customer service calls from the document database, including customer ID, sentiment analysis results, and audio file path.

[0140] CustomerPurchases: Query customer purchase records from a relational database.

[0141] JOIN: Combine the sentiment analysis results of the audio with the customer's purchase history to analyze customers with negative sentiment.

[0142] WHERE: Filter out customers whose sentiment analysis result is "negative".

[0143] Example 3: Ad delivery strategy based on video metadata

[0144] Video ads are stored in the video storage system, and the metadata of the video (such as duration, theme, audience, etc.) is stored in the document database. We hope to recommend new ad videos based on the customer's interests and the videos the customer has watched.

[0145] WITH CustomerInterests AS (

[0146] Query the customer's interest tags (e.g., sports, technology, etc.) from the document database

[0147] SELECT customer_doc->>'name' AS name, customer_doc->>'interests'AS interests, id AS customer_id

[0148] FROM customers_document_store

[0149] ),

[0150] VideoMetadata AS (

[0151] Query video information from the metadata of the video storage system, including video subject and target audience

[0152] SELECT video_metadata->>'video_id' AS video_id, video_metadata->>'theme' AS theme, video_metadata->>'audience' AS audience, video_file_path

[0153] FROM video_storage_system

[0154] ),

[0155] RecommendedVideos AS (

[0156] Recommend videos based on customer interests

[0157] SELECT CustomerInterests.name, VideoMetadata.video_id,VideoMetadata.theme, VideoMetadata.video_file_path

[0158] FROM CustomerInterests

[0159] JOIN VideoMetadata ON CustomerInterests.interests @>ARRAY[VideoMetadata.theme] )

[0161] Returns the video recommended to each customer and its file path

[0162] SELECT name, video_id, theme, video_file_path

[0163] FROM RecommendedVideos;

[0164] CustomerInterests: extracts customer interest tags from the document database.

[0165] VideoMetadata: Extracts video metadata (subject, audience, etc.) from video storage systems.

[0166] JOIN: Recommend videos that match the customer's interests based on the customer's interest tags.

[0167] SELECT: Returns the client's recommended videos and their corresponding file paths.

[0168] Example 4: Cross-modal query: multimodal analysis combining images, audio, and video

[0169] There is a complex scenario involving multimodal data such as images, audio, and video. In this scenario, we want to analyze the customer's shopping behavior and use the customer's shopping images, customer service call audio, and watched video ads to generate recommendations.

[0170] WITH CustomerImages AS (

[0171] Query the image database for shopping images uploaded by customers (for example, shopping receipts or product photos)

[0172] SELECT image_metadata->>'customer_id' AS customer_id, image_metadata->>'product_type' AS product_type, image_path

[0173] FROM image_storage_system

[0174] WHERE image_type = 'shopping'

[0175] ),

[0176] AudioAnalysis AS (

[0177] Querying sentiment analysis of customer service calls from audio metadata

[0178] SELECT audio_doc->>'customer_id' AS customer_id, audio_doc->>'sentiment' AS sentiment, audio_file_path

[0179] FROM audio_metadata_store

[0180] ),

[0181] VideoInteractions AS (

[0182] Query the ad videos that the customer has watched from the video metadata

[0183] SELECT video_metadata->>'customer_id' AS customer_id, video_metadata->>'video_id' AS video_id, video_metadata->>'theme' AS theme, video_file_path

[0184] FROM video_storage_system )

[0186] Comprehensively analyze customers' shopping pictures, customer service call audio, and advertising video viewing records to generate personalized recommendations

[0187] SELECT ci.customer_id, ci.image_path, aa.sentiment, aa.audio_file_path, vi.video_file_path, vi.theme

[0188] FROM CustomerImages ci

[0189] JOIN AudioAnalysis aa ON ci.customer_id = aa.customer_id

[0190] JOIN VideoInteractions vi ON ci.customer_id = vi.customer_id

[0191] WHERE aa.sentiment = 'positive' AND vi.theme = 'fashion';

[0192] explain:

[0193] CustomerImages: Query the image database for shopping images uploaded by customers (such as shopping receipts or product photos).

[0194] AudioAnalysis: Extract sentiment analysis results of customer service calls from audio metadata.

[0195] VideoInteractions: Query the video storage system for the advertising video records that the customer has watched.

[0196] JOIN: Combine customers’ shopping pictures, call audios, and videos for comprehensive analysis.

[0197] WHERE: Filter out customers whose sentiment analysis is positive and who have viewed fashion-themed ads.

[0198] In examples 1-3, only the association between a single type of data (image, audio, video) and user data is processed. In the fourth example, a comprehensive analysis of the three types of data, image, audio, and video, is performed, and the fusion capability of multimodal data is performed through the MUQSQL query language. Using the MUQSQL query language is an optional implementation method, and other query languages ​​suitable for multimodality can also be used to implement multimodal data queries.

[0199] In addition, it can be seen from the above examples that feature extraction and data conversion functions are added to the SELECT clause (such as extract_features extracts features from images), cross-model data sources and similarity-based connection operations (such as connections based on image similarity) are supported in the FROM and JOIN clauses, and multimedia comparison and fuzzy matching capabilities are expanded in the WHERE clause (such as image_similarity calculates image similarity).

[0200] On the other hand, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the query method based on a multimodal database described in any embodiment of the present application.

[0201] Among them, in the computer program product, an optional implementation form of the program module architecture of the computer program that implements each step of the query method based on the multimodal database can be a query device based on the multimodal database.

[0202] See also Figure 4 An embodiment of the present application provides a query device based on a multimodal database, including: an acquisition module 41, used to acquire data to be processed including multiple modalities; a conversion module 42, used to convert the data to be processed into category data corresponding to the data to be processed based on a category model based on the data to be processed; the acquisition module 41 is also used to acquire a request to be queried, and decompose the request to be queried into multiple sub-queries; a merging module 43, used to obtain multiple sub-results based on multiple sub-queries, and merge the multiple sub-results based on the category data to obtain a merged result; the conversion module 42 is also used to convert the merged result into target query data in a target data format.

[0203] Optionally, the conversion module 42 is further used for:

[0204] Based on each record to be processed in the data to be processed, generating a directed tree structure corresponding to each record to be processed;

[0205] For each type of data in the data to be processed, obtaining a first access path corresponding to each type of data;

[0206] Based on the directed tree structure corresponding to each record to be processed and the first access path corresponding to each type of data, the record to be processed is converted into the category data corresponding to the record to be processed, and an instance object corresponding to each record to be processed is obtained;

[0207] Based on the instance object corresponding to each record to be processed, the target category of each record to be processed in the category data is determined, and the attribute set corresponding to the target category is updated.

[0208] Optionally, the tree structure includes multiple nodes and edges between two nodes, each node indicates an attribute in a type of data, and the edge represents a nested relationship between the attributes corresponding to the two nodes.

[0209] Optionally, the conversion module 42 is further used for:

[0210] Traversing the directed tree structure corresponding to each to-be-processed record to obtain a plurality of to-be-processed nodes, storing the plurality of to-be-processed nodes and the first access path corresponding to each type of data in a stack, storing context information in the traversal process through the stack, and for each to-be-processed node, determining whether an attribute corresponding to each to-be-processed node exists in the current category data;

[0211] Obtain a first node to be processed whose attributes exist in the current scope data, obtain an instance object corresponding to the first processing node in the current scope data, and update the active domain of the instance object corresponding to the first processing node, and obtain a second node to be processed whose attributes do not exist in the current scope data, and create an instance object corresponding to the second processing node in the current scope data.

[0212] Optionally, the conversion module 42 is further used for:

[0213] Acquire an associated object related to the instance object corresponding to each to-be-processed record from the current category data, and determine a target category of the instance object corresponding to each to-be-processed record in the category data according to the associated object and the category of the associated object;

[0214] Obtain a super identifier set corresponding to the target category, and update the attributes in the instance object corresponding to each super identifier in the super identifier set.

[0215] Optionally, the conversion module 42 is further used for:

[0216] Acquire multiple mapping tuples from the target data format to the category model, each of the mapping tuples indicating a data category and a data structure and data content in the data category of the record, each of the mapping tuples including at least the following: a database identifier, a data category identifier, a root object associated with the data category identifier, a root morphism, a morphism signature set constituting a primary key, a reference set, and a second access path describing the data structure in the data category;

[0217] Based on the mapping tuple, the merged result is converted into target query data in a target data format.

[0218] Optionally, the conversion module 42 is further used for:

[0219] Based on the mapping element, acquiring a pattern category and an instance category associated with the merge result in the category data, wherein the pattern category represents a definition description of a data structure in the category data, and the instance category represents data of an instance object in the category data;

[0220] Based on the schema categories, creating a database definition statement;

[0221] Based on the instance scope, creating a data manipulation statement, and inserting the data manipulation statement into the target data format;

[0222] Add integrity constraint statements;

[0223] The target query data is generated based on the database definition statement, the data manipulation statement and the integrity constraint statement.

[0224] Optionally, the conversion module 42 is further used for:

[0225] Determine an attribute including sub-attributes in the pattern category as a complex attribute, store the sub-attributes corresponding to the complex attribute in a stack, process the sub-attributes corresponding to the complex attribute in the stack to obtain processed attribute data, and generate a database definition statement corresponding to the complex attribute in a wrapper based on the processed attribute data;

[0226] The attributes without sub-attributes in the pattern category are determined as simple attributes, and a database definition statement corresponding to the simple attribute is generated in the wrapper.

[0227] Optionally, the conversion module 42 is further used for:

[0228] Obtain all object pairs in the data category with root morphisms in the instance category, and construct a create data manipulation statement for each pair of object pairs;

[0229] A root object of a data class without a root morphism in the instance category is obtained, the root object is used to obtain a super identifier set in the instance category, and a data manipulation statement is created based on the super identifier set.

[0230] Optionally, the merging module 43 is further used for:

[0231] Translate each subquery into the query language corresponding to each subquery;

[0232] Use the query language corresponding to each subquery to query and obtain the subresults corresponding to each subquery;

[0233] Based on the category data, convert each sub-result into each sub-instance object;

[0234] Each sub-instance object is merged to obtain a merged result.

[0235] It can be understood by those skilled in the art that Figure 4 The structure of the query device based on the multimodal database does not constitute a limitation on the query device based on the multimodal database. Figure 4 The structure of the query device based on a multimodal database does not constitute a limitation on the query device based on a multimodal database, and the various modules can be implemented in whole or in part by software, hardware, and a combination thereof. The above modules can be embedded in or independent of the controller in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the controller can call and execute the operations corresponding to the above modules. In other embodiments, the query device based on a multimodal database may include more or fewer modules than shown in the figure.

[0236] See also Figure 5 In another aspect of the embodiment of the present application, a computing device 200 is provided, including a memory 3011 and a processor 3012. The memory 3011 stores a computer program. When the computer program is executed by the processor, the processor 3012 executes the steps of the query method based on the multimodal database provided in any of the above embodiments of the present application. The terminal device 100 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch) or similar devices, a robotic device.

[0237] The query devices based on the multimodal database include but are not limited to: embedded devices and terminal devices.

[0238] The processor 3012 is the control center, which uses various interfaces and lines to connect various data of the entire computer device, and executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 3011, and calling data stored in the memory 3011. Optionally, the processor 3012 may include one or more processing cores; preferably, the processor 3012 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 3012.

[0239] The memory 3011 can be used to store software programs and modules. The processor 3012 executes various functional applications and data processing by running the software programs and modules stored in the memory 3011. The memory 3011 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 3011 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 3011 may also include a memory controller to provide the processor 3012 with access to the memory 3011.

[0240] On the other hand, an embodiment of the present application further provides a storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the query method based on a multimodal database provided in any of the above embodiments of the present application.

[0241] Those of ordinary skill in the art can understand that the implementation of all or data flows in the methods provided in the above embodiments can be completed by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the process of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0242] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A query method based on a multimodal database, characterized in that: include: Acquire data to be processed including multiple modalities; Based on the data to be processed, the data to be processed is converted into category data corresponding to the data to be processed based on the category model, wherein the data to be processed is converted into category data corresponding to the data to be processed based on the category model, including: based on each record to be processed in the data to be processed, generating a directed tree structure corresponding to each record to be processed; for each type of data in the data to be processed, obtaining a first access path corresponding to each type of data; based on the directed tree structure corresponding to each record to be processed and the first access path corresponding to each type of data, converting the record to be processed into the category data corresponding to the record to be processed, and obtaining an instance object corresponding to each record to be processed; based on each The target category of each record to be processed in the category data is determined based on the instance object corresponding to each record to be processed, and the attribute set corresponding to the target category is updated; wherein the target category of each record to be processed in the category data is determined based on the instance object corresponding to each record to be processed, and the attribute set corresponding to the target category is updated, including: obtaining the associated object related to the instance object corresponding to each record to be processed from the current category data, and determining the target category of the instance object corresponding to each record to be processed in the category data according to the category of the associated object; obtaining the super identifier set corresponding to the target category, and updating the attribute in the instance object corresponding to each super identifier in the super identifier set; Obtaining a request to be queried, and decomposing the request to be queried into multiple sub-queries; Based on the multiple sub-queries, multiple sub-results are obtained, and based on the category data, the multiple sub-results are merged to obtain a merged result; The merged result is converted into target query data in a target data format.

2. The query method based on a multimodal database according to claim 1, characterized in that: The tree structure includes multiple nodes and edges between two nodes, each node indicates an attribute in a type of data, and the edge represents the nested relationship between the attributes corresponding to the two nodes.

3. The query method based on a multimodal database according to claim 1, characterized in that: The process of converting the records to be processed into category data based on the directed tree structure corresponding to each record to be processed and the first access path corresponding to each type of data, and obtaining the instance object corresponding to each record to be processed includes: Traversing the directed tree structure corresponding to each to-be-processed record to obtain a plurality of to-be-processed nodes, storing the plurality of to-be-processed nodes and the first access path corresponding to each type of data in a stack, storing context information in the traversal process through the stack, and for each to-be-processed node, determining whether an attribute corresponding to each to-be-processed node exists in the current category data; Obtain a first node to be processed whose attributes exist in the current category data, obtain an instance object corresponding to the first node to be processed in the current category data, and update the active domain of the instance object corresponding to the first node to be processed, and obtain a second node to be processed whose attributes do not exist in the current category data, and create an instance object corresponding to the second node to be processed in the current category data.

4. The query method based on a multimodal database according to claim 1, characterized in that: The step of converting the merged result into target query data in a target data format comprises: Acquire multiple mapping tuples from the target data format to the category model, each of the mapping tuples indicating a data category and a data structure and data content in the data category of the record, each of the mapping tuples including at least the following: a database identifier, a data category identifier, a root object associated with the data category identifier, a root morphism, a morphism signature set constituting a primary key, a reference set, and a second access path describing the data structure in the data category; Based on the mapping tuple, the merged result is converted into target query data in a target data format.

5. The query method based on a multimodal database according to claim 4, characterized in that: The converting the merged result into target query data in a target data format based on the mapping tuple comprises: Based on the mapping tuple, acquiring a pattern category and an instance category associated with the merge result in the category data, wherein the pattern category represents a definition description of a data structure in the category data, and the instance category represents data of an instance object in the category data; Based on the schema categories, creating a database definition statement; Based on the instance scope, creating a data manipulation statement, and inserting the data manipulation statement into the target data format; Add integrity constraint statements; The target query data is generated based on the database definition statement, the data manipulation statement and the integrity constraint statement.

6. The query method based on a multimodal database according to claim 5, characterized in that: The creating of a database definition statement based on the mode category includes: Determine an attribute including sub-attributes in the pattern category as a complex attribute, store the sub-attributes corresponding to the complex attribute in a stack, process the sub-attributes corresponding to the complex attribute in the stack to obtain processed attribute data, and generate a database definition statement corresponding to the complex attribute in a wrapper based on the processed attribute data; The attributes without sub-attributes in the pattern category are determined as simple attributes, and a database definition statement corresponding to the simple attribute is generated in the wrapper.

7. The query method based on a multimodal database according to claim 5, characterized in that: The creating a data manipulation statement based on the instance scope and inserting the data manipulation statement into the target data format comprises: Obtain all object pairs in the data category with root morphisms in the instance category, and construct a create data manipulation statement for each pair of object pairs; A root object of a data class without a root morphism in the instance category is obtained, the root object is used to obtain a super identifier set in the instance category, and a data manipulation statement is created based on the super identifier set.

8. The query method based on a multimodal database according to claim 1, characterized in that: The obtaining of multiple sub-results based on the multiple sub-queries and the merging of the multiple sub-results based on the category data to obtain a merged result includes: Translate each subquery into the query language corresponding to each subquery; Use the query language corresponding to each subquery to query and obtain the subresults corresponding to each subquery; Based on the category data, convert each sub-result into each sub-instance object; Each sub-instance object is merged to obtain a merged result.

Images