Data query method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a data query method and device, equipment and a storage medium. The method comprises the steps that in response to a data storage request, an index construction service deployed in the cloud environment is utilized, and at least one vector index corresponding to at least one data block of data is generated; storing the at least one data block and the at least one vector index to a data storage service in a cloud environment, wherein each data block at least comprises at least part of content of the data and the corresponding vector index; and in response to the query request, querying data matched with the query request from the data storage service by using a query service deployed in the cloud environment, the query service and the index construction service being allocated with different resources in the cloud environment. In this way, management of system resources can be optimized in a storage and calculation separation mode.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to a method, apparatus, device, and computer-readable storage medium for data query. Background Art

[0002] With the development of computer technology, the scale of unstructured data (for example, images, videos, texts, etc.) continues to grow. The demand for analysis of unstructured data is also growing. In some scenarios, users often need to query unstructured data. In the computer field, unstructured data can be represented in the form of vectors. Therefore, vector queries are widely used in query scenarios of unstructured data such as image queries, text semantic queries, video queries, and multimodal queries. Through vector queries, users can obtain one or more vector query results that are most similar to the specified vector. In recent years, vector queries have attracted more and more attention as an important query enhancement method. However, there are significant differences between the storage and processing of vector data and the storage and processing of traditional scalar data. Therefore, it is necessary to provide a query solution that is compatible with scalar queries while also enabling efficient and fast vector queries. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for data query is provided. The method comprises: in response to a data storage request, using an index building service deployed in a cloud environment, generating at least one vector index corresponding to at least one data block of the data; storing at least one data block and at least one vector index in a data storage service in the cloud environment, wherein each data block includes at least part of the content of the data and a corresponding vector index; and in response to a query request, using a query service deployed in the cloud environment, querying data matching the query request from the data storage service, wherein the query service and the index building service are allocated with different resources in the cloud environment.

[0004] In a second aspect of the present disclosure, a device for data query is provided. The device includes: a generation module configured to: in response to a data storage request, generate at least one vector index corresponding to at least one data block of the data using an index building service deployed in a cloud environment; a storage module configured to: store at least one data block and at least one vector index to a data storage service in the cloud environment, wherein each data block includes at least part of the content of the data and a corresponding vector index; and a query module configured to: in response to a query request, query the data storage service for data matching the query request using a query service deployed in the cloud environment, wherein the query service and the index building service are allocated with different resources in the cloud environment.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments according to the present disclosure may be implemented;

[0011] Figure 2 shows a database framework according to some embodiments of the present disclosure;

[0012] Figure 3 A flowchart showing an example process of data query according to some embodiments of the present disclosure;

[0013] Figure 4 A flowchart showing an example process of data query according to some embodiments of the present disclosure;

[0014] Figure 5 A flowchart showing an example process of data query according to some embodiments of the present disclosure;

[0015] Figure 6 A schematic structural block diagram showing an example apparatus for data query according to some embodiments of the present disclosure; and

[0016] Figure 7A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0020] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0021] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0022] The schemes in this specification and the embodiments, if involving the processing of personal information, will be processed on the premise of having a legal basis (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. The user's refusal to process personal information other than the necessary information required for basic functions will not affect the user's use of basic functions. As an optional but non-restrictive implementation method, in response to receiving the user's active request, a method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation method of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation method of the present disclosure.

[0023] With the development of computer technology, the scale of unstructured data (for example, images, videos, texts, etc.) continues to grow. The demand for analysis of unstructured data is also growing. In some scenarios, users often need to query unstructured data. In the computer field, unstructured data can be represented in the form of vectors. Therefore, vector queries are widely used in query scenarios of unstructured data such as image queries, text semantic queries, video queries, and multimodal queries. Through vector queries, users can obtain one or more vector query results that are most similar to the specified vector. In recent years, vector queries have attracted more and more attention as an important query enhancement method. However, there are significant differences between the storage and processing of vector data and the storage and processing of traditional scalar data. Therefore, as discussed below, when implementing vector retrieval, at least the following aspects need to be considered.

[0024] Vector query needs to find the most similar results in a large number of vectors, and similarity calculation has a high demand for computing resources. Therefore, it is usually necessary to construct the vector data to be queried into a certain structure organization (i.e., vector index) according to the similarity relationship before querying, so as to reduce the number of similarity calculations required during actual query, which is sometimes called approximate nearest neighbor calculation. Approximate nearest neighbor calculation effectively improves the efficiency of query at the cost of reducing a certain degree of accuracy. In operation, the performance of approximate nearest neighbor calculation depends on the index of a specific structure, i.e., vector index. However, the construction of vector index usually takes a long time, much longer than the time of data insertion. In addition, the construction of vector index usually consumes more computing resources and memory resources. Therefore, when implementing the vector query function, it is necessary to reasonably configure the computing resources and / or memory resources used to construct the vector index so that the vector index can be created in time and will not affect the progress of other tasks due to occupying too many resources.

[0025] Vector indexes usually contain the following information: the original or compressed data of the vector data and some additional structure-related data (for example, additional information used to implement fast vector query, screening, and filtering, such as the association relationship and similarity relationship between vectors). Since vector indexes contain more information, vector indexes usually occupy a relatively large memory space. In addition, since the data structure of the vector index is large, the time to load the vector index each time will also be relatively long. For scenarios with low query latency and / or high query rates per second, in order to ensure efficient execution of queries, when executing vector queries, it is usually necessary to read all vector indexes into memory (also called in-memory computing), that is, it is usually necessary to store vector indexes in memory for a long time. Therefore, when implementing vector retrieval, it is necessary to consider the use of in-memory computing to support efficient vector queries, and also to consider the reasonable allocation of memory resources.

[0026] In addition, in the process of executing vector query, users usually need to query a lot of other attribute information of similar vectors. In this case, it is usually necessary to use some scalar filtering conditions to obtain more expected result screening, which is also called scalar-vector combination query or simply combined query. As an example, in an e-commerce scenario, a user enters a picture including a dress for search and specifies the color as red. In this specific embodiment, the user specifies the scalar filtering condition 'red' to achieve a more accurate query. Therefore, when implementing the vector query function, it is also necessary to consider how to execute the scalar filtering statement so as not to introduce too much resource overhead while achieving accurate query.

[0027] In order to make the database support vector retrieval, in some solutions, the database is designed as a dedicated vector database centered on vector data, also known as a dedicated vector database solution. In this solution, the database takes vector-specific data management and computing operations as a baseline, and expands to support other related functions on this basis. The advantage of the dedicated vector database solution is that it can be deeply customized for vector query scenarios. However, since the dedicated vector database solution does not support relatively complex queries, it cannot cope with the diverse query needs of users. In other solutions, the database is designed to use an existing database that supports scalar queries as a baseline, and vector queries are attached to the existing database as an extended function, also known as a database extension implementation solution. In the database extension implementation solution, vector queries are implemented as a specific operator / function. The advantage of the database extension implementation solution is that vector queries can make full use of all the functions of the currently implemented database, such as flexible and rich database query statements. However, in the database extension implementation solution, since the vector index is implemented as an extension of the index type and the vector query is implemented as a specific operator / function, it is difficult to achieve high computing performance. It can be seen that both the traditional dedicated vector database solution and the database extension implementation solution are in urgent need of further improvement.

[0028] The embodiment of the present disclosure proposes a data query scheme. In the scheme, in response to a data storage request, at least one vector index corresponding to at least one data block of the data is generated by using an index building service deployed in a cloud environment; at least one data block and at least one vector index are stored in a data storage service in the cloud environment, wherein each data block includes at least part of the content of the data and a corresponding vector index; and in response to a query request, the query service deployed in the cloud environment is used to query data matching the query request from the data storage service, wherein the query service and the index building service are allocated with different resources in the cloud environment.

[0029] In this way, according to the embodiments of the present disclosure, all functions of the currently implemented database can be fully utilized. More importantly, a system framework of storage and computing separation can be implemented, that is, the query service and the index construction service are allocated different resources in the cloud environment, thereby ensuring that the vector index can be constructed in time and that the index construction task does not occupy too many system resources.

[0030] Various example implementations of the solution are described in detail below in conjunction with the accompanying drawings.

[0031] Example Environment

[0032] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1As shown, the example environment 100 may include a terminal device 125. In the example environment 100, an application 120 is installed in the terminal device 125. A user 140 may interact with the application 120 via the terminal device 125 and / or an attached device of the terminal device 125.

[0033] The application 120 can provide the user 140 with an integration of multiple applications or components. These applications can be used as application modules in the application 120. In some embodiments, the application 120 can be downloaded and installed as an application on the terminal device 125. In some embodiments, the application 120 can also be accessed in other ways, such as through a web page.

[0034] Application 120 may be any appropriate type of application capable of providing media content, examples of which may include, but are not limited to: local life services, social applications, audio / video applications, media item playback applications, broadcast applications, etc. The embodiments of the present disclosure are not limited in this respect.

[0035] exist Figure 1 In the environment 100, if the application 120 is in an active state, the terminal device 125 can present an interactive page through the application 120. The interactive page can be any appropriate type of page, which can support the user 140 to input any appropriate type of data and present media items of any media item type to the user 140. The interactive interface can include various interfaces that the application 120 can provide, such as a parameter setting interface, a loading interface, a special effect display interface, and the like.

[0036] The terminal device 125 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDA), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 125 can also support any type of interface for the user (such as "wearable" circuits, etc.).

[0037] The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 130 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0038] A communication connection may be established between the server 130 and the terminal device 125. The communication connection may be established in a wired manner or a wireless manner. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In the embodiments of the present disclosure, the server 130 and the terminal device 125 may implement signaling interaction through a mutual communication connection.

[0039] For ease of discussion, the terminal device 125 and / or the server are sometimes collectively referred to as the electronic device 110. For ease of discussion only, in some example embodiments of the present disclosure, the example process is described with the electronic device 110 as the execution subject. It should be understood that in the present disclosure, the electronic device 110 performing action A may include at least one of the following: the terminal device 125 independently performs action A, the server independently performs action A, or the terminal device 125 and the server cooperate to perform action A.

[0040] It should be understood that the structure and function of each element in the environment 100 are described for exemplary purposes only, and do not imply any limitation on the scope of the present disclosure. Some exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0041] Example Interaction

[0042] Next, we will combine Figure 2-4 To describe the data query scheme of each embodiment of the present disclosure. Figure 1 To describe Figure 2-4 , for example, by using electronic device 110, application 125, user 140, server 130 and terminal device 125. In operation, user 140 can trigger data query through interface 150. Terminal device 125 can obtain data query results through communication with server 130, and the query results are displayed in interface 150.

[0043] See also Figure 2The database framework 200 according to some embodiments of the present disclosure is shown. The database framework 200 includes a storage service. Through the storage service, the data in the database is stored in the database in the form of data blocks, and each data block may include corresponding data and indexes. Figure 2 In the embodiment, the data block 230 may include at least one of the following: original data 231, a scalar index 232 or a vector index 233.

[0044] The database framework 200 further includes a query service and an index building service, wherein the query service may further include a scalar query service and a vector query service. Figure 2 In the embodiment, the index building service is implemented as a data writing worker node 220, and the query service is implemented as a query worker node 221-1 for processing scalar queries and a vector retrieval query worker node 221-2 for processing vector queries.

[0045] The database framework 200 further includes a classification and coordination service. In some embodiments, the classification and coordination service can distinguish the query request of the user, for example, determine whether the query request of the user is a scalar data query or a vector data query. In addition, the classification and coordination service can also coordinate the resources of various types of working nodes, for example, scheduling the resources of the vector index construction through the vector index construction scheduler.

[0046] In operation, in response to a data storage request, the electronic device 110 generates at least one vector index corresponding to at least one data block of the data using an index building service deployed in a cloud environment. Figure 2 In the example of , in response to a data storage request, the electronic device 110 can use the vector index construction service of the data writing working node 220 to build a vector index. The electronic device 110 can store at least one data block and at least one vector index to a data storage service in the cloud environment, where each data block includes at least part of the data content and a corresponding vector index. Figure 2 In the embodiment, the vector index constructed by the vector index construction service of the data writing work node 220 can be stored as the vector index 233 of the data block 230. In this way, when the data is stored in the database, the database can complete the indexing of the data to facilitate subsequent queries.

[0047] In response to the query request, the electronic device 110 uses the query service deployed in the cloud environment to retrieve data matching the query request from the data storage service. Figure 2 In the embodiment of the present invention, the database framework 200 includes query services, such as (scalar search) query work node 221-1 and vector search query work node 222-2. More importantly, in Figure 2In the embodiment, the query service and the index building service are allocated different resources in the cloud environment. Specifically, the data writing work node 220 and the query service ((scalar retrieval) query work node 221-1 and the vector retrieval query work node 222-2) are configured with different cloud resources. In this way, it can be ensured that the vector data is indexed in a timely manner without affecting the execution of the data query. In short, the above embodiment can realize the architecture of the storage and computing separation framework. By utilizing the resource isolation mechanism and the flexible resource scheduling strategy, the vector index building task can be separated into the writing node, that is, the index building task with a long duration and high resource occupancy is separated from the vector retrieval query. Furthermore, the task scheduling mechanism can be used to schedule the index building tasks to support both synchronous and asynchronous construction methods.

[0048] As briefly discussed above, in-memory computing can improve the efficiency of data queries. According to some embodiments of the present disclosure, the electronic device 110 may preload at least one vector index stored in the data storage service into a first cache area corresponding to the query service. In this case, the electronic device 110 determines whether data matching the query request is stored in the data storage service based on a matching relationship between the query request and at least one vector index in the first cache area, and in response to determining whether data matching the query request is stored in the data storage service, obtains data corresponding to the matching vector index from the data storage service based on the matching vector index. Figure 2 In the embodiment of the present invention, the vector index cache 231 stores preloaded vector indexes. When performing vector retrieval, the vector retrieval computing service 232 can use the preloaded indexes in the vector index cache 231 to perform data query. In this way, the processing time of data query can be shortened.

[0049] In addition to vector indexes being preloaded, according to some embodiments of the present disclosure, scalar indexes can also be preloaded into the memory. Specifically, in some embodiments, each data block also includes a scalar index corresponding to at least part of the content of the data. In this case, the electronic device 110 preloads at least one scalar index stored in the data storage service into the second cache area corresponding to the query service. When performing data retrieval, the electronic device 110 determines whether data matching the query request is stored in the data storage service based on a matching relationship between the query request and at least one scalar index in the first cache area. In response to determining whether data matching the query request is stored in the data storage service, based on the matching vector index, the electronic device 110 obtains the data corresponding to the matching scalar index from the data storage service. See further. Figure 2In the example embodiment, the scalar index of the data block is stored in the scalar attribute cache and read service 232. In this case, the vector retrieval query worker node 221-2 can use the scalar attribute cache and read service 232 to implement a scalar-vector combined query. It should be understood that the (scalar retrieval) query worker node 221-1 can also be loaded with scalar data to shorten the query time of the scalar data.

[0050] By preloading, pre-built vector indexes and / or scalar indexes can be preloaded to the corresponding vector retrieval query nodes based on the topological structure of the data blocks, thereby reducing the import overhead of the query nodes. In other words, according to the above-mentioned embodiments of the present disclosure, vector index cache and scalar column cache mechanisms can be added to ensure that when there is sufficient memory, vector retrieval related calculations can perform high-performance in-memory calculations. At the same time, when there is insufficient memory, a swap-in and call-out mechanism based on the least recently used (Least Recently Used, LRU) strategy is also provided, that is, through an efficient memory management mechanism, continuous import of indexes and efficient swap-in and swap-out are supported.

[0051] As briefly discussed above, the classification and coordination service can classify and pre-process the user's query request, for example, via the query parser 205 and the optimizer 210. In some embodiments, the query service includes a vector query service based on a vector index and a scalar query service based on a scalar index, that is, vector data queries and scalar data queries can be processed separately.

[0052] Next, we will further combine Figure 3 and 4 To further describe the specific processing of the data request. In some embodiments, by parsing the query request, the electronic device 110 can determine whether the query request corresponds to a vector index-based query or a scalar index-based query. Additionally, in some embodiments, if the electronic device 1101 determines that the parsing result of the query request indicates that the query request includes an operator specific to a vector query, the electronic device 110 determines that the query request corresponds to a vector index-based query.

[0053] Next, in response to determining that the query request corresponds to a query based on a vector index, the query request is sent to a vector query service for processing. Accordingly, in response to determining that the query request corresponds to a query based on a scalar index, the electronic device 110 sends the query request to a scalar query service for processing. Figure 3 In an embodiment of the invention, at block 310, the query parser may parse the query request. At block 330, it is determined whether a vector search calculation is included. If it is determined to be included, vector query processing is performed, such as executing an embedded vector search calculation mode. Otherwise, scalar query processing is performed, such as entering a general execution link. Further reference is made to Figure 4 In the embodiment, when the data query is a vector search (block 410 ), the user's query may be sent to the vector query node 420 . On the contrary, if the data query request is a scalar data search, the user's query request is sent to the scalar query node 430 .

[0054] As discussed above, the electronic device 110 may pre-process the query request, for example, via an optimizer. In some embodiments, the electronic device 110 may extract at least one operator specific to vector query from the query request, and based on the at least one operator, convert the query request into a remote procedure call request for the vector query service. Next, the electronic device 110 may receive a call response to the remote procedure call request from the vector query service, the call response including data matching the query request, and generate a query response to the query request based on the call response. Figure 3 In an example embodiment, in box 310, the query parser can parse the query request of the user. Based on the parsing result of the query parser, in box 320, the optimizer can extract at least one operator specific to the vector query from the query request, and based on the at least one operator, convert the query request into a remote procedure call request for the vector query service. In operation, the optimizer can determine the actual execution path, that is, whether to execute the vector query or the scalar query. In operation, for vector retrieval related queries, specific operators in the query (for example, vector retrieval metric function, query column, query vector, candidate query result parameter, filter statement, etc.) are extracted. The optimizer can generate a remote call request and directly send it to the corresponding query node for query, and the query results are collected and merged uniformly. In this way, the query request has been preprocessed into a standard query statement before being sent to the query node, and the query node can perform the query operation without additional processing (for example, determining the local query plan), thereby shortening the processing time of the query request.

[0055] According to some embodiments of the present disclosure, the computational logic of data query may be further optimized. Example embodiments are further discussed as follows.

[0056] In some embodiments, the electronic device 110 selects data matching the query request from multiple candidate databases based on a matching relationship between the query request and at least one vector index in the first cache area, and based on a matching relationship between at least one cached scalar index and the query request. In traditional data query schemes, in a combined query scenario, a scalar query is usually performed first, and then a vector query is performed in the result of the scalar query. According to an embodiment of the present disclosure, possible candidate results are first obtained through vector indexes. In this way, the advantages of vector queries can be fully utilized and the accuracy of query results is improved.

[0057] According to some embodiments of the present disclosure, multiple query logics may be preconfigured for scenarios where the query request includes a filter statement (i.e., a combined query scenario). When executing a data query, the optimal query logic is selected from the multiple query logics to execute the query according to specific query requirements.

[0058] In some embodiments, the plurality of retrieval strategies include a first retrieval strategy configured to filter data blocks stored by the data storage service based on a filter statement, and retrieve data matching the query request based on an index corresponding to the filtered data blocks.

[0059] Alternatively or additionally, in some embodiments, the multiple retrieval strategies include a second retrieval strategy, which is configured to perform filtering on data blocks stored by the data storage service based on a filter statement, and when the number of filtered data blocks is lower than a threshold, determine the data that matches the query request based on the similarity between the query request and the data content included in the filtered data blocks.

[0060] Alternatively or additionally, in some embodiments, the multiple retrieval strategies include a third retrieval strategy, which is configured to select multiple candidate data blocks from data blocks stored by the data storage service based on a matching relationship between the query request and at least one vector index, and filter the multiple candidate data blocks based on a filter statement to obtain data that matches the query request.

[0061] exist Figure 4 In the example embodiment of FIG. 4 , the filtering and vector search execution strategy 450 may select an appropriate strategy from among a plurality of strategies. According to the selected target search strategy, the electronic device 110 may retrieve data matching the query request from the data storage service.

[0062] In this way, through an efficient query calculation mechanism, high-performance vector query calculation and rapid extraction of relevant attribute information can be achieved.

[0063] Through the above-mentioned embodiments of the present disclosure, a hybrid computing architecture that takes into account both conventional analytical queries and high-performance vector retrieval can be implemented. Based on this hybrid computing architecture, both scalar data queries and vector data queries can make full use of existing database technologies. In some embodiments, the management of system resources can be optimized by separating storage and computing. In some embodiments, in-memory computing can be fully utilized to shorten the processing time of data queries. In some embodiments, query logic can be optimized to improve query performance.

[0064] Example Process

[0065] Figure 5 1 is a flowchart of an example process 500 of data query according to some embodiments of the present disclosure. The process 500 may be implemented at the terminal device 125 (or at the server 130, or at the terminal device 125 and the server 130). Figure 1 The process 500 is described below.

[0066] In block 510 , in response to a data storage request, the electronic device generates at least one vector index corresponding to at least one data block of the data using an index building service deployed in a cloud environment.

[0067] In box 520, the electronic device stores at least one data block and at least one vector index to a data storage service in a cloud environment, wherein each data block includes at least a portion of the content of the data and a corresponding vector index.

[0068] In block 530 , in response to the query request, the electronic device uses a query service deployed in the cloud environment to query data matching the query request from the data storage service, wherein the query service and the index building service are allocated different resources in the cloud environment.

[0069] In some embodiments, the electronic device may preload at least one vector index stored in the data storage service into a first cache area corresponding to the query service. Further, the electronic device queries the data storage service for data matching the query request by performing the following operations: determining whether the data storage service stores data matching the query request based on a matching relationship between the query request and at least one vector index in the first cache area; in response to determining whether the data storage service stores data matching the query request, obtaining data corresponding to the matching vector index from the data storage service based on the matching vector index.

[0070] In some embodiments, each data block may also include a scalar index corresponding to at least part of the content of the data. In this case, the electronic device may preload at least one scalar index stored in the data storage service into the second cache area corresponding to the query service, and query the data matching the query request from the data storage service by: determining whether the data matching the query request is stored in the data storage service based on the matching relationship between the query request and at least one scalar index in the first cache area; in response to determining whether the data matching the query request is stored in the data storage service, obtaining the data corresponding to the matching scalar index from the data storage service based on the matching vector index.

[0071] In some embodiments, each data block also includes a scalar index corresponding to at least part of the content of the data, and the query service includes a vector query service based on the vector index and a scalar query service based on the scalar index. In this case, the electronic device can determine whether the query request corresponds to a query based on the vector index or a query based on the scalar index by parsing the query request. In response to determining that the query request corresponds to a query based on the vector index, the electronic device sends the query request to the vector query service for processing. In response to determining that the query request corresponds to a query based on the scalar index, the electronic device sends the query request to the scalar query service for processing.

[0072] In some embodiments, the electronic device may determine whether the query request corresponds to a vector index-based query or a scalar index-based query by the following operations: in response to a parsing result of the query request indicating that the query request includes an operator specific to vector queries, determining that the query request corresponds to a vector index-based query.

[0073] In some embodiments, the electronic device may send a query request to a vector query service by: extracting at least one operator specific to the vector query from the query request; and based on the at least one operator, converting the query request into a remote procedure call request for the vector query service. Additionally, the electronic device receives a call response to the remote procedure call request from the vector query service, the call response including data matching the query request. The electronic device may generate a query response to the query request based on the response.

[0074] In some embodiments, the electronic device selects data matching the query request from multiple candidate data blocks related to the query request from a data storage service based on a matching relationship between the query request and at least one vector index in a first cache area, and selects data matching the query request from multiple candidate databases based on a matching relationship between at least one cached scalar index and the query request.

[0075] In some embodiments, in response to determining that the query request includes a filter statement, the electronic device selects a target retrieval strategy from a plurality of retrieval strategies, the plurality of retrieval strategies including at least one of the following: a first retrieval strategy, the first retrieval strategy is configured to filter the data blocks stored by the data storage service based on the filter statement, and retrieve the data matching the query request based on the index corresponding to the filtered data blocks; a second retrieval strategy, the second retrieval strategy is configured to filter the data blocks stored by the data storage service based on the filter statement, and when the number of filtered data blocks is lower than a threshold, determine the data matching the query request based on the similarity between the query request and the data content included in the filtered data blocks; and a third retrieval strategy, the third retrieval strategy is configured to select a plurality of candidate data blocks from the data blocks stored by the data storage service based on the matching relationship between the query request and at least one vector index, and filter the plurality of candidate data blocks based on the filter statement to obtain the data matching the query request. The electronic device retrieves the data matching the query request from the data storage service according to the selected target retrieval strategy.

[0076] Example devices and equipment

[0077] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 A schematic structural block diagram of an example apparatus 600 for data query according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the terminal device 125 (or in the server 130, or in the terminal device 125 and the server 130). Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0078] like Figure 6 As shown, the device 600 includes: a generation module 610, configured to: in response to a storage request for data, generate at least one vector index corresponding to at least one data block of data by using an index building service deployed in a cloud environment; a storage module 620, configured to: store at least one data block and at least one vector index in a data storage service in the cloud environment, wherein each data block includes at least part of the content of the data and a corresponding vector index; and a query module 630, configured to: in response to a query request, query the data storage service for data matching the query request by using a query service deployed in the cloud environment, wherein the query service and the index building service are allocated different resources in the cloud environment.

[0079] In some embodiments, the apparatus 600 further includes a first preloading module configured to: preload at least one vector index stored in the data storage service into a first cache area corresponding to the query service. The query module 630 is further configured to: determine whether data matching the query request is stored in the data storage service based on a matching relationship between the query request and at least one vector index in the first cache area; and in response to determining whether data matching the query request is stored in the data storage service, obtain data corresponding to the matching vector index from the data storage service based on the matching vector index.

[0080] In some embodiments, each data block also includes a scalar index corresponding to at least part of the content of the data. The device 600 also includes a second preloading module, which is configured to: preload at least one scalar index stored in the data storage service to a second cache area corresponding to the query service. The query module 630 is also configured to: determine whether data matching the query request is stored in the data storage service based on a matching relationship between the query request and at least one scalar index in the first cache area; in response to determining whether data matching the query request is stored in the data storage service, based on the matching vector index, obtain data corresponding to the matching scalar index from the data storage service.

[0081] In some embodiments, each data block also includes a scalar index corresponding to at least part of the content of the data, and the query service includes a vector query service based on the vector index and a scalar query service based on the scalar index. The device 600 also includes a parsing module configured to: determine whether the query request corresponds to a query based on the vector index or a query based on the scalar index by parsing the query request; in response to determining that the query request corresponds to a query based on the vector index, send the query request to the vector query service for processing; and in response to determining that the query request corresponds to a query based on the scalar index, send the query request to the scalar query service for processing.

[0082] In some embodiments, the parsing module is further configured to: in response to a parsing result of the query request indicating that the query request includes an operator specific to a vector query, determine that the query request corresponds to a vector index-based query.

[0083] In some embodiments, the parsing module is further configured to: extract at least one operator specific to the vector query from the query request; and based on the at least one operator, convert the query request into a remote procedure call request for the vector query service. The apparatus 600 also includes a response generation module, configured to: receive a call response to the remote procedure call request from the vector query service, the call response including data matching the query request; and generate a query response to the query request based on the response.

[0084] In some embodiments, the query module 630 is further configured to: based on a matching relationship between the query request and at least one vector index in the first cache area, select multiple candidate data blocks related to the query request from the data storage service; and based on a matching relationship between at least one cached scalar index and the query request, select data matching the query request from multiple candidate databases.

[0085] In some embodiments, the query module 630 is further configured to: in response to determining that the query request includes a filter statement, select a target retrieval strategy from a plurality of retrieval strategies, the plurality of retrieval strategies including at least one of the following: a first retrieval strategy, the first retrieval strategy being configured to filter the data blocks stored by the data storage service based on the filter statement, and retrieve the data matching the query request based on the index corresponding to the filtered data blocks; a second retrieval strategy, the second retrieval strategy being configured to filter the data blocks stored by the data storage service based on the filter statement, and when the number of filtered data blocks is lower than a threshold, determine the data matching the query request based on the similarity between the query request and the data content included in the filtered data blocks; and a third retrieval strategy, the third retrieval strategy being configured to select a plurality of candidate data blocks from the data blocks stored by the data storage service based on the matching relationship between the query request and at least one vector index, and filter the plurality of candidate data blocks based on the filter statement to obtain the data matching the query request; and retrieve the data matching the query request from the data storage service according to the selected target retrieval strategy.

[0086] like Figure 7 As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 770, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0087] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0088] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0089] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0090] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0091] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0092] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0094] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0095] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0096] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A data query method, comprising: In response to a storage request for data, generating at least one vector index corresponding to at least one data block of the data using an index building service deployed in a cloud environment; storing the at least one data block and the at least one vector index in a data storage service in the cloud environment, wherein each data block includes at least a portion of the data and a corresponding vector index; as well as In response to a query request, data matching the query request is retrieved from the data storage service using a query service deployed in the cloud environment, wherein the query service and the index building service are allocated different resources in the cloud environment.

2. The method according to claim 1, further comprising: Preloading at least one vector index stored in the data storage service into a first cache area corresponding to the query service; Retrieving data matching the query request from the data storage service includes: Determining whether data matching the query request is stored in the data storage service based on a matching relationship between the query request and at least one vector index in the first cache area; as well as In response to determining whether the data storage service stores data matching the query request, based on the matching vector index, the data corresponding to the matching vector index is acquired from the data storage service.

3. The method according to claim 1, wherein each data block further comprises a scalar index corresponding to at least part of the content of the data, the method further comprising: Preloading at least one scalar index stored in the data storage service into a second cache area corresponding to the query service; and Retrieving data matching the query request from the data storage service includes: Determine whether data matching the query request is stored in the data storage service based on a matching relationship between the query request and at least one scalar index in the first cache area; In response to determining whether the data storage service stores data matching the query request, based on the matching vector index, the data corresponding to the matching scalar index is obtained from the data storage service.

4. The method according to claim 1, wherein each data block further comprises a scalar index corresponding to at least part of the content of the data, and the query service comprises a vector query service based on a vector index and a scalar query service based on a scalar index, the method further comprising: Determining, by parsing the query request, whether the query request corresponds to a query based on a vector index or a query based on a scalar index; In response to determining that the query request corresponds to a vector index-based query, sending the query request to the vector query service for processing; as well as In response to determining that the query request corresponds to a scalar index-based query, the query request is sent to the scalar query service for processing.

5. The method according to claim 4, wherein determining whether the query request corresponds to a query based on a vector index or a query based on a scalar index comprises: In response to a parsing result of the query request indicating that the query request includes an operator specific to a vector query, it is determined that the query request corresponds to a vector index-based query.

6. The method according to claim 4, wherein sending the query request to the vector query service comprises: extracting at least one operator specific to vector query from the query request; as well as Based on the at least one operator, converting the query request into a remote procedure call request for the vector query service, and The method further comprises: receiving a call response to the remote procedure call request from the vector query service, the call response including data matching the query request; as well as A query response to the query request is generated based on the call response.

7. The method according to claim 1, wherein retrieving data matching the query request from the data storage service comprises: Based on a matching relationship between the query request and at least one vector index in the first cache area, selecting a plurality of candidate data blocks related to the query request from the data storage service; as well as Based on a matching relationship between at least one cached scalar index and the query request, data matching the query request is selected from the multiple candidate databases.

8. The method according to claim 1, wherein retrieving data matching the query request from the data storage service comprises: In response to determining that the query request includes a filter statement, a target search strategy is selected from the multiple search strategies, wherein the multiple search strategies include at least one of the following: a first retrieval strategy, wherein the first retrieval strategy is configured to filter the data blocks stored by the data storage service based on the filter statement, and retrieve data matching the query request based on indexes corresponding to the filtered data blocks; a second retrieval strategy, wherein the second retrieval strategy is configured to filter the data blocks stored by the data storage service based on the filter statement, and determine data matching the query request based on a similarity between the query request and data content included in the filtered data blocks when the number of filtered data blocks is lower than a threshold, and a third retrieval strategy, wherein the third retrieval strategy is configured to select a plurality of candidate data blocks from the data blocks stored by the data storage service based on a matching relationship between the query request and at least one vector index, and filter the plurality of candidate data blocks based on the filter statement to obtain data matching the query request; as well as Data matching the query request is retrieved from the data storage service according to the selected target retrieval strategy.

9. A data query device, comprising: A generating module is configured to: in response to a storage request for data, generate at least one vector index corresponding to at least one data block of the data by using an index building service deployed in a cloud environment; A storage module, configured to: store the at least one data block and the at least one vector index in a data storage service in the cloud environment, wherein each data block includes at least a portion of the data and a corresponding vector index; as well as The query module is configured to: in response to a query request, query the data matching the query request from the data storage service using the query service deployed in the cloud environment, wherein the query service and the index building service are allocated different resources in the cloud environment.

10. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 8.

Images