Data query method and related device
By performing intent recognition and schema mapping on natural language queries, selecting appropriate database adapters for querying, and integrating results at the data management layer, the problem of low query efficiency in existing technologies is solved, achieving efficient and accurate data querying.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QINGDAO JUSHANGHUI NETWORK TECH CO LTD
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-08
AI Technical Summary
Existing data query methods are inefficient and ineffective in handling different types of query data when faced with diverse query needs.
By receiving natural query statements, the system identifies the intent and maps it to query types and conditions suitable for database operations, selects the appropriate database adapter for querying, and integrates the results at the data management layer.
It enables precise access to different types of databases, improves the overall efficiency of data querying, reduces resource consumption, and enhances the accuracy and consistency of query results.
Smart Images
Figure CN121996682A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data query method and related apparatus. Background Technology
[0002] With the continuous development of information systems, the scale of data continues to grow, and data sources and storage methods are becoming increasingly diverse. Users' data query needs are also showing a trend towards higher frequency and greater complexity. In practical applications, users typically initiate query requests using natural language or business statements. The system needs to parse, transform, and execute corresponding data query operations to return results that meet the query conditions. In existing data query methods, different types of query data require different query statements and processing flows, leading to inefficiencies when facing diverse query needs. Therefore, there is an urgent need for a method to improve data query efficiency. Summary of the Invention
[0003] In view of the above problems, this application provides a data query method and related apparatus to improve data query efficiency. The specific solution is as follows:
[0004] The first aspect of this application provides a data query method, comprising: receiving a query statement sent by a querying user, and performing intent recognition on the query statement to obtain the type of data to be queried and the query conditions for the data to be queried;
[0005] The schema mapping operation is used to map the type of data to be queried and the query conditions of the data to be queried, so as to obtain the query type and query conditions suitable for database operations.
[0006] Select a query method based on the query type applicable to the database operation, and select a database adapter based on the query method. Send the query conditions applicable to the database operation to the selected database adapter, and obtain the query results obtained by the database adapter in the corresponding database according to the query conditions applicable to the database operation.
[0007] Input the query results into the data management layer for data integration, obtain the integrated data results, and output the integrated data results.
[0008] Optionally, intent recognition is performed on the natural language query to obtain the type of data to be queried and the query conditions for the data to be queried, including:
[0009] Based on LLM, intent recognition is performed on natural language queries to obtain recognition results. LLM is a pre-built large language model.
[0010] If the recognition result only contains structured conditions, then the query type of the data to be queried is determined to be structured data, and the query conditions of the data to be queried are determined based on the recognition result.
[0011] If the recognition result only contains semantic search conditions, then the query type of the data to be queried is determined to be unstructured data, and the query conditions of the data to be queried are determined based on the recognition result.
[0012] If the recognition result contains both structured conditions and semantic search conditions, the query type of the data to be queried is determined to be mixed data, and the query conditions of the data to be queried are determined based on the recognition result.
[0013] Optionally, the query conditions for the data to be queried can be determined based on the recognition results, including:
[0014] If the query type of the data to be queried is determined to be structured data, the target entity and attribute fields used to represent structured data are extracted from the recognition results, and the query conditions of the data to be queried are determined based on the target entity and attribute fields.
[0015] If the query type of the data to be queried is determined to be unstructured data, then the text content used to represent unstructured data is extracted from the recognition results, and the query conditions of the data to be queried are determined based on the text content.
[0016] If the query type of the data to be queried is determined to be mixed data, the target entity, attribute field and text content used to represent structured data and unstructured data are extracted from the recognition results respectively, and the query conditions of the data to be queried are determined based on the target entity, attribute field and text content.
[0017] Optionally, a query method is selected based on the query type applicable to the database operation, and a database adapter is selected based on the query method. The query conditions applicable to the database operation are sent to the selected database adapter, and the query results obtained by the database adapter in the corresponding database based on the query conditions applicable to the database operation are obtained, including:
[0018] If the query type applicable to database operations is unstructured data, then select the vector query method, and select the Milvus adapter and MySQL adapter based on the vector query method;
[0019] The query conditions applicable to database operations are vectorized using a preset embedding model to obtain query vectors;
[0020] The Milvus adapter is invoked to perform similarity retrieval on the query vector in the vector database, resulting in a collection of vector data.
[0021] The vector data in the retrieved set are sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity are obtained. The similarity is the similarity between the query vector and the vector data in the vector database.
[0022] The MySQL adapter is invoked to query the storage address of the unstructured data corresponding to the vector IDs of the first K vector data in the structured database, and the unstructured data is obtained from the storage address.
[0023] Optionally, a query method is selected based on the query type applicable to the database operation, and a database adapter is selected based on the query method. The query conditions applicable to the database operation are sent to the selected database adapter, and the query results obtained by the database adapter in the corresponding database based on the query conditions applicable to the database operation are obtained, including:
[0024] If the query type applicable to the database operation is mixed data, then select vector query method and data query method, and select Milvus adapter and MySQL adapter based on vector query method and data query method;
[0025] The query conditions related to unstructured data are vectorized using a pre-defined embedding model to obtain query vectors, and the structured query conditions related to structured data are extracted.
[0026] Call the MySQL adapter to query structured data in a structured database based on structured query conditions;
[0027] The Milvus adapter is invoked to perform similarity retrieval on the query vector in the vector database, resulting in a collection of vector data.
[0028] The vector data in the retrieved set are sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity are obtained. The similarity is the similarity between the query vector and the vector data in the vector database.
[0029] The MySQL adapter is invoked to query the storage address of the unstructured data corresponding to the vector IDs of the first K vector data in the structured database, and the unstructured data is obtained from the storage address.
[0030] Optionally, the query results can be input into the data management layer for data integration, and the integrated data results can be output, including:
[0031] The pre-defined result aggregator based on the data management layer performs aggregator operations on the received query results to obtain the aggregator result.
[0032] LLM is used to process the integrated results into natural language, and the natural language is obtained and output.
[0033] A second aspect of this application provides a data query device, comprising:
[0034] The intent recognition module is used to receive the natural language query statement sent by the querying user, and to perform intent recognition on the natural language query statement to obtain the type of data to be queried and the query conditions of the data to be queried.
[0035] The data mapping module is used to map the type of data to be queried and the query conditions of the data to be queried using schema mapping operations, so as to obtain the query type and query conditions suitable for database operations.
[0036] The query module is used to select a query method based on the query type applicable to the database operation, select a database adapter based on the query method, send the query conditions applicable to the database operation to the selected database adapter, and obtain the query results obtained by the database adapter in the corresponding database according to the query conditions applicable to the database operation.
[0037] The data integration module is used to input query results into the data management layer for data integration, obtain the data integration results, and output the data integration results.
[0038] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the data query method described in the first aspect or any implementation thereof.
[0039] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:
[0040] The memory is used to store computer programs;
[0041] The processor is used to execute the computer program so that the electronic device can implement the data query method of the first aspect or any implementation thereof.
[0042] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the data query method described in the first aspect or any implementation thereof.
[0043] By employing the above technical solution, compared with existing technologies, the data query method provided in this application, through intent recognition of natural language queries, parses the query requirement into the type of data to be queried and the query conditions for that data. Furthermore, it transforms these into query types and conditions suitable for database operations through schema mapping operations, enabling standardized processing of query requests. Secondly, by selecting the corresponding query method based on the query type and further matching the database adapter, precise access to data in different types of databases is achieved, thus avoiding the complexity of requiring users to use different query logic and query statements for different query data. Finally, by uniformly inputting the query results into the data management layer for integrated processing, unified data management and output are achieved, improving the overall efficiency of data querying and reducing resource consumption. Attached Figure Description
[0044] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0045] Figure 1 A flowchart illustrating a data query method provided in this application;
[0046] Figure 2 A detailed flowchart illustrating a data query method provided in this application;
[0047] Figure 3 A schematic diagram illustrating database selection for a data query method provided in this application;
[0048] Figure 4 This application provides an unstructured data query intent for a data query method.
[0049] Figure 5 A file storage diagram illustrating a data query method provided in this application;
[0050] Figure 6 The mixed data query intent of the data query method provided in this application;
[0051] Figure 7 A schematic diagram of the structure of a data query device provided in this application;
[0052] Figure 8 A schematic block diagram of an electronic device provided in this application. Detailed Implementation
[0053] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.
[0054] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.
[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.
[0056] This application provides a data query method, such as Figure 1 The method may include the following steps:
[0057] S101: Receive the query statement sent by the query user, and perform intent recognition on the query statement to obtain the type of data to be queried and the query conditions of the data to be queried.
[0058] In this embodiment, the system first receives a natural language query statement sent by the user. This query statement can be a natural language expression submitted by the user through text input, speech transcription, or other human-computer interaction methods, used to express their query needs. Examples include "query orders for user ID 123," "find pictures containing cats," and "find landscape photos uploaded by users who have purchased mobile phones." Subsequently, intent recognition processing is performed on the query statement. Intent recognition is used to parse the user's query intent from the query statement and determine the type of data to be queried and the query conditions for that data. The type of data to be queried characterizes the category of data the user wishes to retrieve, and the query conditions describe specific constraints or semantic content related to the query target.
[0059] The types of data to be queried can include structured data, unstructured data, and mixed data that includes both structured and unstructured data. Structured data refers to information stored in a fixed format or schema, which can usually be retrieved and processed quickly. The most typical example of this type of data is information in database tables, such as customer lists and sales records. Unstructured data lacks a predefined schema or structure, is diverse in form, and is difficult to process directly. Common unstructured data includes emails, social media posts, images, and audio files. The query conditions can include those for both structured and unstructured data. For example, when the natural language query is "query order information for user ID 123," the system can extract "user" as the target entity and "user ID" and "order information" as attribute fields from the recognition results. "User ID" and "order information" can then be used as query conditions for structured data. For unstructured data, such as when the natural language query is "find pictures containing cats," the system extracts "contains cats" and "pictures" from the recognition results as query conditions for unstructured data. Through the above intent recognition processing, the user's query requirements expressed in natural language are converted into a query semantic representation that the system can understand, thereby reducing the query threshold for users, improving the accuracy and consistency of query request parsing, and enhancing the intelligence level and query efficiency of the overall data query process.
[0060] S102: Use schema mapping operations to map the type of data to be queried and the query conditions of the data to be queried, so as to obtain the query type and query conditions suitable for database operations.
[0061] Optionally, in this embodiment, a schema mapping operation can be used to map the type of data to be queried and the query conditions for the data to be queried, so as to convert the query information based on natural language semantics into query types and query conditions suitable for database operations. Specifically, schema mapping rules or schema mapping tables can be pre-constructed to describe the correspondence between data types at the natural language level and data type identifiers that can be recognized at the database level, as well as the mapping relationship between natural language query conditions and database query syntax. For example, when the type of data to be queried is structured data, and the query condition is "orders with user ID 123", the system uses a schema mapping operation to map the data type to the corresponding structured data type identifier in the database, and maps the query condition to query conditions suitable for database operations, such as a query expression containing target entities and attribute fields. For example, "user ID" is mapped to the attribute field user_id, "123" is mapped to the attribute value in the corresponding attribute field, and "order" is mapped to the "Order" target entity table. Then, a query expression suitable for database operations is generated: "Filter records from the Order table where user_id equals 123". When the data to be queried is unstructured data, and the query condition is "images containing cats," the system uses schema mapping operations to map the data type to an unstructured data type identifier recognizable in the database, and maps the query condition to semantic query conditions for subsequent similarity retrieval. For example, mapping "image" to "photo" and "cat" to "cat" results in the semantic query condition "find photos containing cats in the vector database," where the recognizable type identifiers in the database are pre-defined identifiers. When the data to be queried is mixed data, such as "find landscape photos uploaded by users who have purchased mobile phones," the system performs schema mapping operations on both the structured and semantic parts to generate structured and semantic query conditions suitable for database operations. Through these schema mapping operations, natural language query requirements with different expressions can be uniformly converted into query types and query conditions that the database can directly process, providing standardized input for subsequent query method selection and database adapter calls.
[0062] S103: Select a query method based on the query type applicable to the database operation, select a database adapter based on the query method, send the query conditions applicable to the database operation to the selected database adapter, and obtain the query results obtained by the database adapter in the corresponding database according to the query conditions applicable to the database operation.
[0063] Optionally, in this embodiment, a corresponding query method is selected based on the query type applicable to the database operation, and a database adapter is further selected based on the query method. The query conditions applicable to the database operation are sent to the selected database adapter to execute the query operation in the corresponding database and obtain the query results. The query method is used to characterize the retrieval mechanism used when the database executes the query. Different query methods correspond to different types of data access strategies. The database adapter can be used to shield the differences between the underlying databases and realize unified access to different databases.
[0064] For example, when the query type applicable to database operations is structured data, the system selects a data query method and, based on this method, selects a database adapter corresponding to the structured database, such as a MySQL adapter. The query conditions applicable to database operations are sent to the MySQL adapter in the form of a standard query statement. The MySQL adapter then executes the query in the corresponding database and returns the query results. When the query type applicable to database operations is unstructured data, the system selects a vector query method and, based on this method, selects a database adapter corresponding to a vector database, such as a Milvus adapter. The query conditions applicable to database operations are converted into query vectors that can be used for similarity retrieval and sent to the Milvus adapter to obtain data results that match the semantics of the query conditions. When the query type applicable to database operations is mixed data, the system simultaneously selects both a data query method and a vector query method, and calls the corresponding database adapters respectively to execute the query operations on the structured query conditions and semantic query conditions in parallel or sequentially to obtain the corresponding query results. In this embodiment, the system can adaptively select the query method and database adapter according to the query type, thereby avoiding strong coupling between the query logic and the specific database implementation and improving the scalability and execution efficiency of the query process.
[0065] S104: Input the query results into the data management layer for data integration, obtain the data integration results, and output the data integration results.
[0066] Optionally, in this embodiment, the query results obtained from the database adapter can be input into the data management layer for data integration processing. The data management layer is used to uniformly manage and process query results from different query methods and different databases. Specifically, the data management layer performs integration operations such as format normalization, deduplication, association, and sorting on the received query results to generate data integration results that meet the output requirements. For example, when the query results come from a single database adapter, the data management layer performs field unification and format conversion on the query results to form a data integration result with a consistent structure. When the query results come from both data query methods and vector query methods, the data management layer performs association processing on the structured query results and the query results obtained from semantic retrieval. In this embodiment, the data management layer can also process the query results based on a preset result integrator and convert the integrated results into an output format that is easy for users to understand. Finally, the data integration results can be output to the query user. The output can be structured data results, file or image links, or query results presented in natural language. By centrally inputting the query results into the data management layer for unified integration and output, data obtained from different query methods can be managed and presented under the same processing framework, improving the integrity of the overall data query process and the user experience.
[0067] In one embodiment, intent recognition is performed on the query natural language statement based on LLM to obtain the recognition result. LLM is a pre-built large language model.
[0068] If the recognition result only contains structured conditions, then the query type of the data to be queried is determined to be structured data, and the query conditions of the data to be queried are determined based on the recognition result.
[0069] If the recognition result only contains semantic search conditions, then the query type of the data to be queried is determined to be unstructured data, and the query conditions of the data to be queried are determined based on the recognition result.
[0070] If the recognition result contains both structured conditions and semantic search conditions, the query type of the data to be queried is determined to be mixed data, and the query conditions of the data to be queried are determined based on the recognition result.
[0071] Specifically, such as Figure 2In one embodiment, a natural language query obtained from the application layer can be input into an LLM (Large Language Model) to perform intent recognition operations on the query, yielding a recognition result. The application layer can be a web application, mobile application, or API client. The LLM is a pre-built large language model, and the natural language interface is the interface between the application layer and the LLM. The large language model, trained on a large amount of natural language text, possesses the ability to perform semantic understanding, entity extraction, and condition recognition on natural language queries. For example, a received natural language query can be input into the LLM, which outputs a recognition result containing the query intent, entity information, and condition type.
[0072] In this embodiment, if the identification result only contains structured conditions, which are conditional information that can be clearly mapped to database fields, attributes, or relational constraints, they typically include target entities, attribute fields, and specific numerical values, ranges, or enumeration constraints. For example, for the natural language query "query orders with user ID 123", the identification result can extract the target entity as "orders", the attribute field as "user ID", and the condition as "equal to 123". This type of condition can be directly converted into a structured query condition that can be executed by the database. If the identification result only contains the above information, it can be determined that the query type of the data to be queried is structured data, and the query conditions of the data to be queried can be determined based on the identification result, such as extracting the target entity and its corresponding attribute conditions from the identification result. If the identification result only contains semantic search conditions, such as text information used to describe content features or semantic features, it can be determined that the query type of the data to be queried is unstructured data, and the query conditions of the data to be queried can be determined based on the identification result. Specifically, semantic search conditions refer to conditional information mainly used to describe the semantic features of data content. They usually exist in the form of natural language text and are difficult to directly map to specific database fields, but need to be retrieved through semantic similarity calculation. For example, for natural language queries such as "find pictures containing cats" or "find pictures similar to a certain picture," the conditions extracted from the recognition results are mainly textual or semantic information used to describe content features; these conditions are semantic search conditions. If the recognition results contain both structured conditions and semantic search conditions—for example, both explicit entity or attribute constraints and semantic descriptive information—then the query type of the data to be queried can be determined as mixed data, and the query conditions for the queried data can be comprehensively determined based on the recognition results.
[0073] From the above, it can be concluded that this application improves the accuracy of query intent recognition and the adaptability to complex query requests by introducing an intent recognition method based on a large language model and determining the query type based on the recognition results. This is conducive to improving the intelligence level and processing efficiency of the overall data query process.
[0074] Optionally, such as Figure 2 As shown, in another embodiment, the user's natural language query can be sent to the agent layer (data management agent) through the application layer. Subsequently, the user's natural language query is input to the Large Language Model (LLM) via a natural language interface. The LLM performs intent recognition on the query to obtain the recognition result. The query router outputs the corresponding query request to the data processing layer based on the recognition result, and selects different processing methods in the data processing layer according to different query results. For example, when the recognition result corresponds to unstructured data, since it involves data content or file content, a vectorization processor and embedding model are called to perform vectorization processing on the relevant data; when the recognition result corresponds to structured data, the SQL generator in the data processing layer is directly called to generate the corresponding SQL statement; when the recognition result involves file queries, the file manager is called to obtain the storage address of the corresponding file. Subsequently, the database adaptation layer calls the corresponding database adapter based on the data processed by the data processing layer. The database adapter includes a MySQL adapter, a Milvus adapter, and an S3 adapter, which are used to access MySQL structured data, Milvus vector data, and S3 file storage data in the data layer, respectively. After each database adapter completes the query and obtains the corresponding query results, the query results are input into the result integrator for integration processing to obtain the integrated result. The integrated result is then sent to the natural language interface, which outputs the integrated result in natural language form.
[0075] In one embodiment, if the query type of the data to be queried is determined to be structured data, the target entity and attribute fields used to characterize the structured data are extracted from the identification results, and the query conditions of the data to be queried are determined based on the target entity and attribute fields.
[0076] If the query type of the data to be queried is determined to be unstructured data, then the text content used to represent unstructured data is extracted from the recognition results, and the query conditions of the data to be queried are determined based on the text content.
[0077] If the query type of the data to be queried is determined to be mixed data, the target entity, attribute field and text content used to represent structured data and unstructured data are extracted from the recognition results respectively, and the query conditions of the data to be queried are determined based on the target entity, attribute field and text content.
[0078] Specifically, such as Figure 2In one embodiment, after identifying the intent of the natural language query and determining the query type of the data to be queried, the query router can further construct query conditions for the queried data based on the identification results. Specifically, if the query type of the data to be queried is determined to be structured data, the system extracts the target entity and attribute fields used to represent the structured data from the identification results, and determines the query conditions for the queried data based on the target entity and attribute fields. For example, when the natural language query is "query order information for user ID 123", "user" can be extracted as the target entity, and "user ID" and "order information" can be extracted as attribute fields from the identification results, and the corresponding structured query conditions can be constructed based on the target entity and attribute fields.
[0079] If the system determines that the data to be queried is unstructured data, it extracts text content from the recognition results to characterize the unstructured data and determines the query conditions based on the text content. For example, when the natural language query is "find pictures containing cats", the system extracts "contains cats" and "pictures" as text content from the recognition results and generates query conditions for semantic retrieval based on this text content.
[0080] If the query type of the data to be queried is determined to be mixed data, the system can extract target entities and attribute fields representing structured data, and text content representing unstructured data, from the recognition results. The query conditions are then determined based on these target entities, attribute fields, and text content. For example, when the natural language query is "find landscape photos uploaded by users who have purchased mobile phones," the system extracts "users" and "purchase records" from the recognition results as target entities and attribute fields relevant to the structured query, while simultaneously extracting "landscape photos" as text content for unstructured data. These combined information are then used to construct the mixed query conditions.
[0081] From the above, we can conclude that by constructing matching query conditions for different query types, the query conditions can fully and accurately reflect the user's query intent, thereby improving the accuracy of query results and the ability to handle complex query needs.
[0082] In one embodiment, if the query type applicable to the database operation is unstructured data, then the vector query method is selected, and the Milvus adapter and MySQL adapter are selected based on the vector query method.
[0083] The query conditions applicable to database operations are vectorized using a preset embedding model to obtain query vectors;
[0084] The Milvus adapter is invoked to perform similarity retrieval on the query vector in the vector database, resulting in a collection of vector data.
[0085] The vector data in the retrieved set are sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity are obtained. The similarity is the similarity between the query vector and the vector data in the vector database.
[0086] The MySQL adapter is invoked to query the storage address of the unstructured data corresponding to the vector IDs of the first K vector data in the structured database, and the unstructured data is obtained from the storage address.
[0087] Specifically, such as Figure 2 As shown, when the query type applicable to database operations is unstructured data, a vector query method can be selected, and based on this method, a Milvus adapter and a MySQL adapter can be chosen. The Milvus adapter is used to interact with the vector database, and the MySQL adapter is used to interact with the structured database to achieve the retrieval and location of unstructured data. Specifically, a preset embedding model can be used to vectorize the query conditions applicable to database operations, generating query vectors. The embedding model can be a text embedding model or a multimodal embedding model, used to convert the query conditions into vector representations. Subsequently, the Milvus adapter can be called to perform a similarity retrieval operation on the query vector in the vector database, obtaining a set of vector data similar to the query vector. Here, similarity is used to characterize the semantic similarity between the query vector and the vector data in the vector database. After obtaining the vector data set, the vector data in the set is sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity are selected. After obtaining the vector IDs of the top K vector data, the system calls the MySQL adapter to query the storage address of the unstructured data corresponding to the vector ID in the structured database, and obtains the corresponding unstructured data based on the storage address, where K is greater than or equal to 1. For example, unstructured data can be image files, audio files, or video files, and the storage address can be a file system path or an object storage access address.
[0088] Optional, such as Figure 3As shown, in one embodiment, different storage methods can be selected through the adaptation layer, including relational database adapters, vector database adapters, and object storage adapters, which can be combined in various ways. For example, PostgreSQL + pgvector + S3 is a database combination suitable for teams using the PostgreSQL technology stack. MySQL + Qdrant + MinIO is suitable for private deployment scenarios, and the MySQL + Milvus + S3 adapter is the recommended adapter due to its high stability. From the above, it can be concluded that by using vector queries and combining Milvus and MySQL adapters for collaborative queries, efficient retrieval and accurate positioning of unstructured data can be achieved while ensuring semantic matching accuracy, thereby improving the accuracy and response efficiency of unstructured data queries.
[0089] Optional, such as Figure 4 As shown, in another embodiment, when a user inputs the natural language query "find all documents about artificial intelligence", the LLM (Limited Language Model) is first used to perform intent recognition, i.e., analyze the query intent, identifying it as a semantic search, i.e., a query type of unstructured data. Then, the query text "artificial intelligence" is converted into a vector through an embedding model and returned to the agent. A similarity search is performed in the Milvus vector database to obtain a set of vector data similar to the query vector. The similarity is used to characterize the semantic similarity between the query vector and the vector data in the vector database. After obtaining the vector data set, the vector data in the set is sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity are selected. After obtaining the vector IDs of the top K vector data, the system calls the MySQL adapter to query the storage address of the unstructured data corresponding to the vector ID in the structured database. Based on the storage address, the corresponding unstructured data, such as a document URL, is obtained from S3 and sent to the agent. The agent then generates a natural language response, such as "found 15 related documents", and outputs the documents. Here, K is greater than or equal to 1. For example, unstructured data can be image files, audio files, or video files, and the storage address can be a file system path or an object storage access address.
[0090] In one embodiment, if the query type applicable to the database operation is mixed data, then a vector query method and a data query method are selected, and a Milvus adapter and a MySQL adapter are selected based on the vector query method and the data query method.
[0091] The query conditions related to unstructured data are vectorized using a pre-defined embedding model to obtain query vectors, and the structured query conditions related to structured data are extracted.
[0092] Call the MySQL adapter to query structured data in a structured database based on structured query conditions;
[0093] The Milvus adapter is invoked to perform similarity retrieval on the query vector in the vector database, resulting in a collection of vector data.
[0094] The vector data in the retrieved set are sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity are obtained. The similarity is the similarity between the query vector and the vector data in the vector database.
[0095] The MySQL adapter is invoked to query the storage address of the unstructured data corresponding to the vector IDs of the first K vector data in the structured database, and the unstructured data is obtained from the storage address.
[0096] Specifically, when the query type applicable to database operations is mixed data, both vector query and data query methods can be selected simultaneously. Based on these methods, Milvus and MySQL adapters are selected respectively. First, the query conditions can be split to distinguish between those related to unstructured data and those related to structured data. Then, the system uses a preset embedding model to vectorize the query conditions related to unstructured data, obtaining query vectors for semantic retrieval, while simultaneously extracting structured query conditions related to structured data. For example, when the natural language query is "find landscape photos uploaded by users who have purchased mobile phones," the system can identify "users who have purchased mobile phones" as a structured query condition, and "landscape photos" as a semantic query condition related to unstructured data. After obtaining the structured query conditions, the MySQL adapter can be called to query structured data in the structured database based on the structured query conditions, thereby obtaining a data set that meets the structured conditions. Simultaneously, the Milvus adapter can be called to perform a similarity retrieval operation in the vector database based on the query vectors, obtaining a vector data set semantically similar to the query vectors.
[0097] After obtaining the vector data set, the vector data in the set can be sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity can be selected. Similarity is used to characterize the semantic matching degree between the query vector and the vector data in the vector database. Subsequently, the system calls the MySQL adapter to query the storage address of the unstructured data corresponding to the vector IDs of the top K vector data in the structured database, and retrieves the corresponding unstructured data based on the storage address.
[0098] In one embodiment, such as Figure 5As shown, for storing unstructured data, after a user uploads a file, the uploaded file type can be identified (as shown in the diagram), where the type can be text, image, audio, or video. Then, an embedding model is called to process the data into vectors. Specifically, for text data, a text embedding model is called to return a text vector; for image data, an image embedding model is called to return an image vector; for audio data, an audio embedding model is called to return an audio vector; and for video data, keyframes are extracted, image embedding is performed on the keyframes, and a video vector is returned. After vectorization, three storage operations can be executed in parallel: uploading the original file to S3 object storage and returning the file URL; storing the vector data in a Milvus vector database and returning the vector ID; and storing the storage metadata, such as structured data including the S3 URL, vector ID, and user ID, in a MySQL relational database.
[0099] Optional, such as Figure 6 As shown, in another embodiment, when the user inputs a natural language query, "Query photos containing landscapes uploaded by users registered in 2024," this query contains both structured and unstructured conditions (image content). LLM analysis identifies it as a mixed query, which can be split into two subqueries executed in parallel: the subquery for structured data retrieves a list of user IDs registered in 2024 from the structured database MySQL by calling the MySQL adapter and returns it to the agent; the subquery for unstructured data retrieves image vectors related to "landscape" from the unstructured database by calling the Milvus adapter and returns a list of vector IDs and their similarity. Then, metadata, such as metadata containing S3URLs, is associated with the vector IDs and user IDs and sent to the agent to generate a response, such as "Found 23 landscape photos that meet the criteria."
[0100] In one embodiment, the received query results are integrated based on a preset result integrator in the data management layer to obtain an integrated result.
[0101] LLM is used to process the integrated results into natural language, and the natural language is obtained and output.
[0102] Specifically, the process involves obtaining the query results returned by the database adapter and inputting them into the data management layer. A pre-defined result integrator within the data management layer then integrates these results to produce a unified result. This result integrator performs unified processing on data from different query methods or databases. Its integration operations include data deduplication, data association, data sorting, and structured encapsulation of results to create a consistent and semantically clear integrated result. For example, when the query results include both structured data obtained through the MySQL adapter and unstructured data retrieved through the Milvus adapter and located by the MySQL adapter, the result integrator can associate entity identifiers in the structured data with corresponding content in the unstructured data based on pre-defined association rules, generating a unified data result set for subsequent processing. After integrating the query results, LLM (Local Language Management) can be used to perform natural language processing on the integrated results, converting them into a natural language expression that conforms to user understanding habits, and outputting the generated natural language content as the query result. For example, when the integrated results contain multiple data records that meet the query conditions, LLM can summarize, generalize, or explain the integrated results, generating natural language output such as "Relevant data that meets your conditions has been found, including...".
[0103] As can be concluded from the above, this application, by unifying and integrating query results in the data management layer and combining LLM with natural language processing of the integrated results, can present complex and multi-source query results to query users in an intuitive and easy-to-understand way, thereby reducing the user's understanding cost and improving the readability and interactive experience of the query results.
[0104] The above describes a data query method provided by an embodiment of this application. The following will describe the data query apparatus that performs the above description.
[0105] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application. Figure 7 As shown, the data query device includes:
[0106] The intent recognition module 301 is used to receive the query statement sent by the querying user, and to perform intent recognition on the query statement to obtain the type of data to be queried and the query conditions of the data to be queried.
[0107] The data mapping module 302 is used to map the type of data to be queried and the query conditions of the data to be queried using schema mapping operations, so as to obtain the query type and query conditions suitable for database operations.
[0108] The query module 303 is used to select a query method based on the query type applicable to the database operation, select a database adapter based on the query method, send the query conditions applicable to the database operation to the selected database adapter, and obtain the query results obtained by the database adapter in the corresponding database according to the query conditions applicable to the database operation.
[0109] The data integration module 304 is used to input the query results into the data management layer for data integration, obtain the data integration results, and output the data integration results.
[0110] In one embodiment, the intent recognition module 301 is specifically used to: perform intent recognition on the query natural language statement to obtain the type of data to be queried and the query conditions for the data to be queried, including:
[0111] Based on LLM, intent recognition is performed on natural language queries to obtain recognition results. LLM is a pre-built large language model.
[0112] If the recognition result only contains structured conditions, then the query type of the data to be queried is determined to be structured data, and the query conditions of the data to be queried are determined based on the recognition result.
[0113] If the recognition result only contains semantic search conditions, then the query type of the data to be queried is determined to be unstructured data, and the query conditions of the data to be queried are determined based on the recognition result.
[0114] If the recognition result contains both structured conditions and semantic search conditions, the query type of the data to be queried is determined to be mixed data, and the query conditions of the data to be queried are determined based on the recognition result.
[0115] In one embodiment, the intent recognition module 301 is specifically used to: determine the query conditions for the queried data based on the recognition result, including:
[0116] If the query type of the data to be queried is determined to be structured data, the target entity and attribute fields used to represent structured data are extracted from the recognition results, and the query conditions of the data to be queried are determined based on the target entity and attribute fields.
[0117] If the query type of the data to be queried is determined to be unstructured data, then the text content used to represent unstructured data is extracted from the recognition results, and the query conditions of the data to be queried are determined based on the text content.
[0118] If the query type of the data to be queried is determined to be mixed data, the target entity, attribute field and text content used to represent structured data and unstructured data are extracted from the recognition results respectively, and the query conditions of the data to be queried are determined based on the target entity, attribute field and text content.
[0119] In one embodiment, the query module 303 is specifically configured to: select a query method based on the query type applicable to the database operation, select a database adapter based on the query method, send the query conditions applicable to the database operation to the selected database adapter, and obtain the query results obtained by the database adapter in the corresponding database according to the query conditions applicable to the database operation, including:
[0120] If the query type applicable to database operations is unstructured data, then select the vector query method, and select the Milvus adapter and MySQL adapter based on the vector query method;
[0121] The query conditions applicable to database operations are vectorized using a preset embedding model to obtain query vectors;
[0122] The Milvus adapter is invoked to perform similarity retrieval on the query vector in the vector database, resulting in a collection of vector data.
[0123] The vector data in the retrieved set are sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity are obtained. The similarity is the similarity between the query vector and the vector data in the vector database.
[0124] The MySQL adapter is invoked to query the storage address of the unstructured data corresponding to the vector IDs of the first K vector data in the structured database, and the unstructured data is obtained from the storage address.
[0125] In one embodiment, the query module 303 is specifically configured to: select a query method based on the query type applicable to the database operation, select a database adapter based on the query method, send the query conditions applicable to the database operation to the selected database adapter, and obtain the query results obtained by the database adapter in the corresponding database according to the query conditions applicable to the database operation, including:
[0126] If the query type applicable to the database operation is mixed data, then select vector query method and data query method, and select Milvus adapter and MySQL adapter based on vector query method and data query method;
[0127] The query conditions related to unstructured data are vectorized using a pre-defined embedding model to obtain query vectors, and the structured query conditions related to structured data are extracted.
[0128] Call the MySQL adapter to query structured data in a structured database based on structured query conditions;
[0129] The Milvus adapter is invoked to perform similarity retrieval on the query vector in the vector database, resulting in a collection of vector data.
[0130] The vector data in the retrieved set are sorted from high to low similarity, and the vector IDs of the top K vector data with the highest similarity are obtained. The similarity is the similarity between the query vector and the vector data in the vector database.
[0131] The MySQL adapter is invoked to query the storage address of the unstructured data corresponding to the vector IDs of the first K vector data in the structured database, and the unstructured data is obtained from the storage address.
[0132] In one embodiment, the data integration module 304 is specifically used to: input the query results into the data management layer for data integration, obtain the data integration result, and output the data integration result, including:
[0133] The pre-defined result aggregator based on the data management layer performs aggregator operations on the received query results to obtain the aggregator result.
[0134] LLM is used to process the integrated results into natural language, and the natural language is obtained and output.
[0135] This application also provides an electronic device in its embodiments. (See reference...) Figure 8 The diagram illustrates a structural schematic of an electronic device suitable for implementing the data query method in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0136] like Figure 8 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0137] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0138] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the data query methods provided in this application.
[0139] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the data query methods provided in this application.
[0140] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0142] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.
[0143] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
Claims
1. A data query method, characterized in that, include: Receive the query statement sent by the query user, and perform intent recognition on the query statement to obtain the type of data to be queried and the query conditions for the data to be queried; The schema mapping operation is used to map the type of data to be queried and the query conditions of the data to be queried, so as to obtain the query type and query conditions suitable for database operations. Based on the query type applicable to the database operation, a query method is selected, and based on the query method, a database adapter is selected. The query conditions applicable to the database operation are sent to the selected database adapter, and the query results obtained by the database adapter in the corresponding database according to the query conditions applicable to the database operation are obtained. The query results are input into the data management layer for data integration, and the integrated data results are output.
2. The data query method according to claim 1, characterized in that, The process of performing intent recognition on the natural language query to obtain the type of data to be queried and the query conditions for the data to be queried includes: The intent recognition operation is performed on the natural language query based on LLM to obtain the recognition result. The LLM is a pre-built large language model. If the recognition result contains only structured conditions, then the query type of the data to be queried is determined to be structured data, and the query conditions of the data to be queried are determined based on the recognition result; If the recognition result only contains semantic search conditions, then the query type of the data to be queried is determined to be unstructured data, and the query conditions of the data to be queried are determined based on the recognition result; If the recognition result contains both structured conditions and semantic search conditions, then the query type of the data to be queried is determined to be mixed data, and the query conditions of the data to be queried are determined based on the recognition result.
3. The data query method according to claim 2, characterized in that, The process of determining the query conditions for the data based on the identification results includes: If the query type of the data to be queried is determined to be structured data, then the target entity and attribute fields used to characterize the structured data are extracted from the identification result, and the query conditions of the data to be queried are determined based on the target entity and attribute fields. If it is determined that the query type of the data to be queried is unstructured data, then the text content used to characterize the unstructured data is extracted from the recognition result, and the query conditions of the data to be queried are determined based on the text content. If the query type of the data to be queried is determined to be mixed data, then the target entities, attribute fields and text content used to represent structured data and unstructured data are extracted from the recognition results respectively, and the query conditions of the data to be queried are determined based on the target entities, attribute fields and text content.
4. The data query method according to claim 1, characterized in that, The process of selecting a query method based on the query type applicable to the database operation, selecting a database adapter based on the query method, sending the query conditions applicable to the database operation to the selected database adapter, and obtaining the query results obtained by the database adapter from querying the corresponding database according to the query conditions applicable to the database operation includes: If the query type applicable to database operations is unstructured data, then the vector query method is selected, and the Milvus adapter and MySQL adapter are selected based on the vector query method. The query conditions applicable to database operations are vectorized using a preset embedding model to obtain query vectors; The Milvus adapter is invoked to perform a similarity search on the query vector in the vector database, resulting in a collection of vector data. The vector data in the retrieved set are sorted from high to low according to the similarity, and the vector IDs of the top K vector data with the highest similarity are obtained. The similarity is the similarity between the query vector and the vector data in the vector database. The MySQL adapter is invoked to query the storage address of the unstructured data corresponding to the vector IDs of the first K vector data in the structured database, and the unstructured data is obtained from the storage address.
5. The data query method according to claim 1, characterized in that, The process of selecting a query method based on the query type applicable to the database operation, selecting a database adapter based on the query method, sending the query conditions applicable to the database operation to the selected database adapter, and obtaining the query results obtained by the database adapter from querying the corresponding database according to the query conditions applicable to the database operation includes: If the query type applicable to database operations is mixed data, then select vector query method and data query method, and select Milvus adapter and MySQL adapter based on the vector query method and data query method; The query conditions related to unstructured data are vectorized using a pre-defined embedding model to obtain query vectors, and the structured query conditions related to structured data are extracted. Call the MySQL adapter to query structured data in a structured database based on structured query conditions; The Milvus adapter is invoked to perform a similarity search on the query vector in the vector database, resulting in a collection of vector data. The vector data in the retrieved set are sorted from high to low according to the similarity, and the vector IDs of the top K vector data with the highest similarity are obtained. The similarity is the similarity between the query vector and the vector data in the vector database. The MySQL adapter is invoked to query the storage address of the unstructured data corresponding to the vector IDs of the first K vector data in the structured database, and the unstructured data is obtained from the storage address.
6. The data query method according to claim 1, characterized in that, The process of inputting the query results into the data management layer for data integration, obtaining the data integration result, and outputting the data integration result includes: The data management layer's preset result integrator performs an integration operation on the received query results to obtain the integrated result; The integrated result is processed using LLM to obtain natural language and output it.
7. A data query device, characterized in that, include: The intent recognition module is used to receive the query statement sent by the querying user, and to perform intent recognition on the query statement to obtain the type of data to be queried and the query conditions of the data to be queried. The data mapping module is used to map the type of data to be queried and the query conditions of the data to be queried using schema mapping operations, so as to obtain the query type and query conditions suitable for database operations. The query module is used to select a query method based on the query type applicable to the database operation, select a database adapter based on the query method, send the query conditions applicable to the database operation to the selected database adapter, and obtain the query results obtained by the database adapter in the corresponding database according to the query conditions applicable to the database operation. The data integration module is used to input the query results into the data management layer for data integration, obtain the data integration result, and output the data integration result.
8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the data query method as described in any one of claims 1 to 6.
9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the data query method as described in any one of claims 1 to 6.
10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the data query method as described in any one of claims 1 to 6.